I want to use sparse matrices to store a "map" of occupied/free space detected by a laser range finder. I think sparse matrices are a good fit to this problem because there is much more free space in the environment (denoted with 0s), than there are obstacles (denoted with 1s, or a float between 0 and 1 for probability).
After acquiring this map, as part of a matching algorithm I want to apply different spatial transformations to it, such as, but not necessarily limited to, rotations and translations.
As a proof of concept for the transformations part, I have this small SciPy sparse matrix defined in coordinates format (coo_matrix) as follows:
import numpy as np
from scipy import sparse
row = np.array([2])
col = np.array([2])
data = np.array([1])
shape = (4, 4)
m = sparse.coo_matrix((data, (row, col)), shape)
m.toarray()
array([[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 0]])
I want to, for example, rotate this matrix 90 degrees counter-clockwise around its center, to get this new matrix:
array([[0, 0, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 0],
[0, 0, 0, 0]])
What I did was:
Get the coordinates for the non-zero point:
point = [m.col[0], m.row[0], 1]
[2, 2, 1]
Calculate the transformation matrix corresponding to a) putting the center at (1.5, 1.5) and b) rotating 90 degrees counter-clockwise:
rot = np.array([[0, -1, 3],[1, 0, 0],[0, 0, 1]])
array([[ 0, -1, 3],
[ 1, 0, 0],
[ 0, 0, 1]])
Apply the transformation by multiplying the point at 1) with the matrix at 2):
rot.dot(point)
array([1, 2, 1])
Which effectively has rotated point at (2,2) to (1,2) (or to (1,2,1) in homogeneous coordinates which is the same), with center on (1.5, 1.5).
Now, this feels rather "manual". Is there a shorter way of applying this transformation? I know there is some quaternion-based rotation supported by scipy but I'm more interested in a general approach that can include not only rotations but translations and basically any transformation (scipy only supports rotations for now anyways).
Is there a more direct way of transforming this matrix without having to manually a) build each non-zero point from coordinates and b) applying the transform to each point?
Note: I'm using sparse matrices for convenience and storage, but if as an intermediate step it had to be converted to another format, transformed, and converted back to sparse, that's ok too.
Thanks!
Related
First of all, I want to summarize how I arrived at this particular problem. I wanted to create a song recommender using collaborative filtering method. But the problem is that I have a very large dataset at hand, 1m rows x 2.2m columns. If my understanding is correct, I needed to create a sparse matrix in order to move forward with my idea, since I do not know of anything that can hold a matrix with the size of 1m x 2.2m.* Hence, sparse matrix.
Now, since this matrix will only contain 1s or 0s in the cells, I've somehow mapped out which cells should have 1 if I were to create a hypothetical monstrous matrix. The information I have looks like this;
rows
locations
row1
[56110, 78999, 1508886, 2090010]
row2
[1123, 976554]
...
...
row1000000
[334555, 2200100]
The problem is that I don't know how to create a sparse matrix using this information. I've checked many sources but couldn't find any viable solution. If you could help me, I would very much appreciate it. Also, if you have any notes on collaborative filtering methods that utilize sparse matrices I would also be very grateful.
There are several ways you could do this. Here is one that creates a csr_matrix, since the data that you show is close to this format. (That docstring has a terse explanation of the csr_matrix attributes data, indices and indptr.) Whether or not this is the best method (for some definition of "best") depends on the actual "raw" form of your data (among other things).
I assume you can put the data that you show in the locations column into a list of lists, called locations. It is important that there is an entry in locations for each row, even if the list is empty. I also assume that the values given in locations are 0-based indices that correspond to the column of the matrix. Here's an example, for an array that has shape (5, 8).
In [23]: locations = [[2, 3], [], [1, 3, 5], [0, 1, 7], [7]]
To form indptr, we compute the cumulative sum of the lengths of the lists, and prepend a 0:
In [28]: lengths = np.array([len(t) for t in locations])
In [29]: lengths
Out[29]: array([2, 0, 3, 3, 1])
In [30]: indptr = np.concatenate(([0], lengths.cumsum()))
In [31]: indptr
Out[31]: array([0, 2, 2, 5, 8, 9])
indices is just the flattened version of locations. Note that sum() in the following is the Python builtin sum() function, not np.sum. That function call concatenates all the lists in locations.
In [32]: indices = sum(locations, start=[])
In [33]: indices
Out[33]: [2, 3, 1, 3, 5, 0, 1, 7, 7]
The data for the array is an array of 1s that is the same length as indices:
In [38]: data = np.ones_like(indices)
We now have all the pieces we need to create a SciPy csr_matrix:
In [39]: from scipy.sparse import csr_matrix
In [40]: A = csr_matrix((data, indices, indptr))
In [41]: A
Out[41]:
<5x8 sparse matrix of type '<class 'numpy.int64'>'
with 9 stored elements in Compressed Sparse Row format>
In [42]: A.toarray()
Out[42]:
array([[0, 0, 1, 1, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 1, 0, 1, 0, 0],
[1, 1, 0, 0, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 1]])
I'm trying to perform a rigid + scale transformation on a 3D volume with pytorch, but I can't seem to understand how the theta required for torch.nn.functional.affine_grid works.
I have a transformation matrix of size (1,4,4) generated by multiplying the matrices Translation * Scale * Rotation. If I use this matrix in, for example, scipy.ndimage.affine_transform, it works with no issues. However, the same matrix (cropped to size (1,3,4)) fails completely with torch.nn.functional.affine_grid.
I have managed to understand how the translation works (range -1 to 1) and I have confirmed that the Translation matrix works by simply normalizing the values to that range. As for the other two, I am lost.
I tried using a basic Scaling matrix alone (below) as a most basic comparison but the results in pytorch are different than that of scipy
Scaling =
[[0.75, 0, 0, 0],
[[0, 0.75, 0, 0],
[[0, 0, 0.75, 0],
[[0, 0, 0, 1]]
How can I convert the (1,4,4) affine matrix to work the same with torch.nn.functional.affine_grid? Alternatively, is there a way to generate the correct matrix based on the transformation parameters (shift, euler angles, scaling)?
To anyone that comes across a similar issue in the future, the problem with scipy vs pytorch affine transforms is that scipy applies the transforms around (0, 0, 0) while pytorch applies it around the middle of the image/volume.
For example, let's take the parameters:
euler_angles = [ea0, ea1, ea2]
translation = [tr0, tr1, tr2]
scale = [sc0, sc1, sc2]
and create the following transformation matrices:
# Rotation matrix
R_x(ea0, ea1, ea2) = np.array([[1, 0, 0, 0],
[0, math.cos(ea0), -math.sin(ea0), 0],
[0, math.sin(ea0), math.cos(ea0), 0],
[0, 0, 0, 1]])
R_y(ea0, ea1, ea2) = np.array([[math.cos(ea1), 0, math.sin(ea1), 0],
[0, 1, 0, 0],
[-math.sin(ea1), 0, math.cos(ea1)], 0],
[0, 0, 0, 1]])
R_z(ea0, ea1, ea2) = np.array([[math.cos(ea2), -math.sin(ea2), 0, 0],
[math.sin(ea2), math.cos(ea2), 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]])
R = R_x.dot(R_y).dot(R_z)
# Translation matrix
T(tr0, tr1, tr2) = np.array([[1, 0, 0, -tr0],
[0, 1, 0, -tr1],
[0, 0, 1, -tr2],
[0, 0, 0, 1]])
# Scaling matrix
S(sc0, sc1, sc2) = np.array([[1/sc0, 0, 0, 0],
[0, 1/sc1, 0, 0],
[0, 0, 1/sc2, 0],
[0, 0, 0, 1]])
If you have a volume of size (100, 100, 100), the scipy transform around the centre of the volume requires moving the centre of the volume to (0, 0, 0) first, and then moving it back to (50, 50, 50) after S, T, and R have been applied. Defining:
T_zero = np.array([[1, 0, 0, 50],
[0, 1, 0, 50],
[0, 0, 1, 50],
[0, 0, 0, 1]])
T_centre = np.array([[1, 0, 0, -50],
[0, 1, 0, -50],
[0, 0, 1, -50],
[0, 0, 0, 1]])
The scipy transform around the centre is then:
transform_scipy_centre = T_zero.dot(T).dot(S).dot(R).T_centre
In pytorch, there are some slight differences to the parameters.
The translation is defined between -1 and 1. Their order is also different. Using the same (100, 100, 100) volume as an example, the translation parameters in pytorch are given by:
# Note the order difference
translation_pytorch = =[tr0_p, tr1_p, tr2_p] = [tr0/50, tr2/50, tr1/50]
T_p = T(tr0_p, tr1_p, tr2_p)
The scale parameters are in a different order:
scale_pytorch = [sc0_p, sc1_p, sc2_p] = [sc2, sc0, sc1]
S_p = S(sc0_p, sc1_p, sc2_p)
The euler angles are the biggest difference. To get the equivalent transform, first the parameters are negative and in a different order:
# Note the order difference
euler_angles_pytorch = [ea0_p, ea1_p, ea2_p] = [-ea0, -ea2, -ea1]
R_x_p = R_x(ea0_p, ea1_p, ea2_p)
R_y_p = R_y(ea0_p, ea1_p, ea2_p)
R_z_p = R_z(ea0_p, ea1_p, ea2_p)
The order in which the rotation matrix is calculated is also different:
# Note the order difference
R_p = R_x_p.dot(R_z_p).dot(R_y_p)
With all these considerations, the scipy transform with:
transform_scipy_centre = T_zero.dot(T).dot(S).dot(R).T_centre
is equivalent to the pytorch transform with:
transform_pytorch = T_p.dot(S_p).dot(R_p)
I hope this helps!
For my current project I need to be able to calculate the rank of 64*64 matrices with entries from GF(2). I was wondering if anyone had a good solution.
I've been using pyfinite for this, but it is rather slow since it's a pure python implementation. I've also tried to cythonise the code I've been using, but have had issues due to relying on pyfinite.
My next idea would be to write my own class in cython, but that seems a bit overkill for what I need.
I need the following functionality
matrix = GF2Matrix(size=64) # creating a 64*64 matrix
matrix.setRow(i, [1,0,1....,1]) # set row using list
matrix += matrix2 # addition of matrices
rank(matrix) # then computing the rank
Thanks for any ideas.
One way to efficiently represent a matrix over GF(2) is to store the rows as integers, interpreting each integer as a bit-string. So for example, the 4-by-4 matrix
[0 1 1 0]
[1 0 1 1]
[0 0 1 0]
[1 0 0 1]
(which has rank 3) could be represented as a list [6, 13, 4, 9] of integers. Here I'm thinking of the first column as corresponding to the least significant bit of the integer, and the last to the most significant bit, but the reverse convention would also work.
With this representation, row operations can be performed efficiently using Python's bitwise integer operations: ^ for addition, & for multiplication.
Then you can compute the rank using a standard Gaussian elimination approach.
Here's some reasonably efficient code. Given a collection rows of nonnegative integers representing a matrix as above, we repeatedly remove the last row in the list, and then use that row to eliminate all 1 entries from the column corresponding to its least significant bit. If the row is zero then it has no least significant bit and doesn't contribute to the rank, so we simply discard it and move on.
def gf2_rank(rows):
"""
Find rank of a matrix over GF2.
The rows of the matrix are given as nonnegative integers, thought
of as bit-strings.
This function modifies the input list. Use gf2_rank(rows.copy())
instead of gf2_rank(rows) to avoid modifying rows.
"""
rank = 0
while rows:
pivot_row = rows.pop()
if pivot_row:
rank += 1
lsb = pivot_row & -pivot_row
for index, row in enumerate(rows):
if row & lsb:
rows[index] = row ^ pivot_row
return rank
Let's run some timings for random 64-by-64 matrices over GF2. random_matrices is a function to create a collection of random 64-by-64 matrices:
import random
def random_matrix():
return [random.getrandbits(64) for row in range(64)]
def random_matrices(count):
return [random_matrix() for _ in range(count)]
and here's the timing code:
import timeit
count = 1000
number = 10
timer = timeit.Timer(
setup="ms = random_matrices({})".format(count),
stmt="[gf2_rank(m.copy()) for m in ms]",
globals=globals())
print(min(timer.repeat(number=number)) / count / number)
The result printed on my machine (2.7 GHz Intel Core i7, macOS 10.14.5, Python 3.7) is 0.0001984686384, so that's a touch under 200µs for a single rank computation.
200µs is quite respectable for a pure Python rank computation, but in case this isn't fast enough, we can follow your suggestion to use Cython. Here's a Cython function that takes a 1d NumPy array of dtype np.uint64, again thinking of each element of the array as a row of your 64-by-64 matrix over GF2, and returns the rank of that matrix.
# cython: language_level=3, boundscheck=False
from libc.stdint cimport uint64_t, int64_t
def gf2_rank(uint64_t[:] rows):
"""
Find rank of a matrix over GF2.
The matrix can have no more than 64 columns, and is represented
as a 1d NumPy array of dtype `np.uint64`. As before, each integer
in the array is thought of as a bit-string to give a row of the
matrix over GF2.
This function modifies the input array.
"""
cdef size_t i, j, nrows, rank
cdef uint64_t pivot_row, row, lsb
nrows = rows.shape[0]
rank = 0
for i in range(nrows):
pivot_row = rows[i]
if pivot_row:
rank += 1
lsb = pivot_row & -pivot_row
for j in range(i + 1, nrows):
row = rows[j]
if row & lsb:
rows[j] = row ^ pivot_row
return rank
Running equivalent timings for 64-by-64 matrices, now represented as NumPy arrays of dtype np.uint64 and shape (64,), I get an average rank-computation time of 7.56µs, over 25 times faster than the pure Python version.
I wrote a Python package galois that extends NumPy arrays over Galois fields. Linear algebra on Galois field matrices is one of the intended use cases. It is written in Python but JIT compiled using Numba for speed. It is quite fast and most linear algebra routines are also compiled. (One exception, as of 08/11/2021 the row reduction routine hasn't been JIT compiled, but that could be added.)
Here is an example using the galois library to do what you are describing.
Create a GF(2) array class and create an explicit array and a random array.
In [1]: import numpy as np
In [2]: import galois
In [3]: GF = galois.GF(2)
In [4]: A = GF([[0, 0, 1, 0], [0, 1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0]]); A
Out[4]:
GF([[0, 0, 1, 0],
[0, 1, 1, 1],
[1, 0, 1, 0],
[1, 0, 1, 0]], order=2)
In [5]: B = GF.Random((4,4)); B
Out[5]:
GF([[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 0, 0],
[0, 0, 1, 0]], order=2)
You can update an entire row (as you requested) like this.
In [6]: B[0,:] = [1,0,0,0]; B
Out[6]:
GF([[1, 0, 0, 0],
[1, 1, 1, 0],
[1, 1, 0, 0],
[0, 0, 1, 0]], order=2)
Matrix arithmetic works with normal binary operators. Here is matrix addition and matrix multiplication.
In [7]: A + B
Out[7]:
GF([[1, 0, 1, 0],
[1, 0, 0, 1],
[0, 1, 1, 0],
[1, 0, 0, 0]], order=2)
In [8]: A # B
Out[8]:
GF([[1, 1, 0, 0],
[0, 0, 0, 0],
[0, 1, 0, 0],
[0, 1, 0, 0]], order=2)
There is an added method to the NumPy arrays called row_reduce() which performs Gaussian elimination on the matrix. You can also call the standard NumPy linear algebra functions on a Galois field array and get the correct result.
In [9]: A.row_reduce()
Out[9]:
GF([[1, 0, 0, 0],
[0, 1, 0, 1],
[0, 0, 1, 0],
[0, 0, 0, 0]], order=2)
In [10]: np.linalg.matrix_rank(A)
Out[10]: 3
Hope this helps! If there is additional functionality desired, please open an issue on GitHub.
I'd some trouble understanding the angle parameter of greycomatrix in skimage. In the example mentioned in the documentation to compute GLCM's for the pixel to the right and up, they mention 4 angles. And they get 4 GLCM's.
>>> image = np.array([[0, 0, 1, 1],
... [0, 0, 1, 1],
... [0, 2, 2, 2],
... [2, 2, 3, 3]], dtype=np.uint8)
>>> result = greycomatrix(image, [1], [0, np.pi/4, np.pi/2, 3*np.pi/4], levels=4)
What should be the parameters for the pixel to the right and down?
There's a typo in the example included in the documentation of greycomatrix (emphasis mine):
Examples
Compute 2 GLCMs: One for a 1-pixel offset to the right, and one for a 1-pixel offset upwards.
>>> image = np.array([[0, 0, 1, 1],
... [0, 0, 1, 1],
... [0, 2, 2, 2],
... [2, 2, 3, 3]], dtype=np.uint8)
>>> result = greycomatrix(image, [1], [0, np.pi/4, np.pi/2, 3*np.pi/4],
... levels=4)
Indeed, result actually contains four different GLCM's rather than two. These four matrices correspond to the possible combinations of one distance and four angles. To calculate the GLCM corresponding to "1-pixel offset to the right" the distance and angle values should be 1 and 0, respectively:
result = greycomatrix(image, distances=[1], angles=[0], levels=4)
whereas to calculate the GLCM corresponding to "1-pixel offset upwards" the parameters should be 1 and np.pi/2:
result = greycomatrix(image, distances=[1], angles=[np.pi/2], levels=4)
In the example, distances=[1] and angles=[0, np.pi/4, np.pi/2, 3*np.pi/4]. To select a particular GLCM one has to specify the appropriate indices for angles and distances. Thus, the 1-pixel to the right GLCM is result[:, :, 0, 0] and the 1-pixel upwards GLCM is result[:, :, 0, 2].
Finally, if you wish to calculate the "1-pixel offset downwards" GLCM (↓) you just have to transpose the "1-pixel offset upwards" GLCM (↑). It is important to note that in most cases both GLCM's are very similar. In fact, you can ignore the order of the co-occurring intensities by setting parameter symmetric to True in the call to greycomatrix. By doing so, al the GLCM's returned by greycomatrix are symmetric.
I have an adjacency matrix of a bipartite graph (of 1's and 0's) and bi-clusters (array of arrays of rows and columns) for this matrix. How do I set different colours for elements (only 1's) in adjacency matrix which belong to different clusters with matplotlib matshow?
import numpy as np
import matplotlib.pyplot as plt
a_matrix = np.array([[0, 0, 1, 0, 1], [0, 0, 0, 1, 0], [0, 0, 1, 1, 1], [1, 1, 0, 0, 0], [0, 1, 0, 0 ,0]])
cluster_1 = np.array([[1, 2, 3], [3, 4, 5]])
cluster_2 = np.array([[4, 5], [1, 2]])
# plot matrix with one colour
plt.matshow(a_matrix, cmap='Greys', interpolation='nearest')
Adjacency matrix, bi-clusters, and a bipartite graph:
One approach might be to make a copy of your matrix and then give distinct values to the clusters you identify.
m = a_matrix.copy() # a copy we can change without altering the orignal
c = cluster_1 # an alias to save typing
# Naked NumPy doesn't seem to have a cartesian product, so roll our own
for i in range(c.shape[1]):
for j in range(c.shape[1]):
if m[c[0,i]-1,c[1,j]-1]:
m[c[0,i]-1,c[1,j]-1] = 2
plt.matshow(m, cmap='jet', interpolation='nearest')
plt.show()
For more clusters, loop over the above, setting a distinct value for each cluster (and maybe choose or define a better colormap). I'm sure there are more efficient implementations of the cartesian product as well...