I have a problem where I want to multiply many 2x2 square matrices. The elements in the matrix are of the from a+ib (complex number)
Here's how the code looks like:
Final_list=[]
for x in list_x:
Matrix=np.array([[1,0],[0,1]])
for y in list_y:
Matrix_y=[....] #create a 2x2 matrix from values x and y
Matrix= Matrix.dot(Matrix_y)
value= 5/(Matrix[0,0] + Matrix[1,0]....) #something like this
Final_list.append(abs(value))
Final_list is what I need.
My code works but it takes a while and I feel like there must be a better way to write this. If length of list_x=30000 and list_y=300 it takes a while to compute ~ a minute or so.
I was able to create a large matrix of shape (300,30000,2,2). I did that by array broadcasting and using np.shape. I hoped to multiply all matrices in a column to get an array of matrices of shape (30000,2,2). I am thinking that might help in the computation but havent figured out a way to do it.
Is there a better way to write this instead of using multiple for loops?
Thanks
I don't know a fully vectorized solution. Initially I thought that np.matmul.reduce might work, but reduce seems to be unsupported for non-commutative operations according to the old docs. There is also np.linalg.multi_dot, but it seems like an overkill for 2 by 2 matrices.
Here is a answer that runs in 1.5 seconds on my machine; it is similar to the approach here.
arr = np.random.rand(300, 30_000, 2, 2)
def my_mul(mats):
if len(mats) == 1:
return mats[0,...]
elif len(mats) == 2:
return mats[0,...] # mats[1,...]
else:
mid = len(mats) // 2
return my_mul(mats[:mid,...]) # my_mul(mats[mid:,...])
result = my_mul(arr) # 1.53 s
Related
I have two Arrays (which are very big), which have the same dimensions. In one of the both Arrays (in my code it's called "minMap") i want to save the smaller value of those both arrays
My current code looks like this:
for y in range(loadedMap.shape[1]):
for x in range(loadedMap.shape[0]):
if loadedMap[x][y] < minMap[x][y]:
minMap[x][y] = loadedMap[x][y]
It's working but im pretty sure it's a dumb solution, because I haven't used any numpy functionality. Maybe a solution with vectorization is faster? I didn't know how to do that :/
(Sorry for my bad english)
There is a function np.minimum() which does exactly what you want:
# a and b are 2 arrays with same shape
c = np.minimum(a, b)
# c will contain minimum values from a and b
I'm doing the following in python:
tmp = np.empty_like(J,dtype=X.dtype)
for idx, (ii, jj) in enumerate(zip(I, J)):
tmp[idx] = sum((X[ii] - X[jj])**2)
where X is a 50000 x 128 numpy array
and I and J are integer numpy arrays of size (763690,) (columns and rows of a sparse matrix)
Now the problem is that the above operation takes about 30 seconds to complete, and I don't see what I need to do to speed this up. I know it can be done faster, since I have a similar code in Matlab where it barely takes any time.
What am I doing wrong here?
Is it something about memory stride access?, not using builtin functions? or something else? should I parallelize/vectorize it?
(I know the title is terrible, but I couldn't figure out a good way to write it, suggestions are welcome!)
We can do this with:
np.sum((X[I]-X[J])**2, axis=1)
Here we thus first use subscripting to generate a 763 690×128 matrix X[I] where for each item in I we use the corresponding row in X. The same happens for X[j]. We then subtract the two, and obtain a 763 690×128 matrix. We can element-wise square the items, and then calculate the sum over the first axis. This thus means that fore every row, we obtain a single value. The result is thus a flat array with 763 690 elements.
Willems method worked wonderful!
np.sum((X[I]-X[J])**2, axis=1)
it took the operation time from ~30s to ~0.6s, thank you very much :)
Im struggling a little with stacking two matrices on top of each other. I'm using the pyKalman package, which when updated, returns a tuple of matrices. One with an updated estimate (new_pred a 1 x 2 vector) and the corresponding covariance matrix (new_cov a 2 x 2 matrix).
After the update, I want to stack the returned values to their corresponding outputs, for a recursive smoothing of the data, through these estimates.
The following is how it is currently implemented.
for meas in onlineObservations:
(new_pred, new_cov) = kf.filter_update(states_pred[-1], cov_pred[-1], meas)
states_pred = np.vstack((states_pred, new_pred))
cov_pred = np.stack((cov_pred, new_cov), axis=0)
Which works really well for the updated estimate (the 1x2 vector), but fails when i try to add new_cov to the array called cov_pred. For good measure:
states_pred.shape = (900,2)
cov_pred.shape = (900, 2, 2)
I've tried changing the axis of "stack" to no avail. It's probably something elementary, but i've been struggling with it for the past hour, and cannot seem to find a "simple" solution.
Thanks in advance.
This should work -
cov_pred = []
for meas in onlineObservations:
(new_pred, new_cov) = kf.filter_update(states_pred[-1], cov_pred[-1], meas)
states_pred = np.vstack((states_pred, new_pred))
cov_pred.append[new_cov]
cov_pred = np.stack(cov_pred, axis=0)
But since you want to update array which you are using already in code, you should use np.concatenate
for meas in onlineObservations:
(new_pred, new_cov) = kf.filter_update(states_pred[-1], cov_pred[-1], meas)
states_pred = np.vstack((states_pred, new_pred))
cov_pred = np.concatenate((cov_pred, np.reshape(new_cov, (1,2,2))), axis=0)
I've been able to make it work by converting cov_pred to a list, and then use:
cov_pred.append(new_cov)
And then re-convert it back again after the for loop. But it seems tedious - at least if there's an even better way!
You can keep your code inside a For Loop (While Loop will also do) and use 'Auto-index Enabled' and thats it....
At the output of Loop, LabVIEW will create a 3D data exactly as your requirement.
I am having a small issue understanding indexing in Numpy arrays. I think a simplified example is best to get an idea of what I am trying to do.
So first I create an array of zeros of the size I want to fill:
x = range(0,10,2)
y = range(0,10,2)
a = zeros(len(x),len(y))
so that will give me an array of zeros that will be 5X5. Now, I want to fill the array with a rather complicated function that I can't get to work with grids. My problem is that I'd like to iterate as:
for i in xrange(0,10,2):
for j in xrange(0,10,2):
.........
"do function and fill the array corresponding to (i,j)"
however, right now what I would like to be a[2,10] is a function of 2 and 10 but instead the index for a function of 2 and 10 would be a[1,4] or whatever.
Again, maybe this is elementary, I've gone over the docs and find myself at a loss.
EDIT:
In the end I vectorized as much as possible and wrote the simulation loops that I could not in Cython. Further I used Joblib to Parallelize the operation. I stored the results in a list because an array was not filling right when running in Parallel. I then used Itertools to split the list into individual results and Pandas to organize the results.
Thank you for all the help
Some tips for your to get the things done keeping a good performance:
- avoid Python `for` loops
- create a function that can deal with vectorized inputs
Example:
def f(xs, ys)
return x**2 + y**2 + x*y
where you can pass xs and ys as arrays and the operation will be done element-wise:
xs = np.random.random((100,200))
ys = np.random.random((100,200))
f(xs,ys)
You should read more about numpy broadcasting to get a better understanding about how the arrays's operations work. This will help you to design a function that can handle properly the arrays.
First, you lack some parenthesis with zeros, the first argument should be a tuple :
a = zeros((len(x),len(y)))
Then, the corresponding indices for your table are i/2 and j/2 :
for i in xrange(0,10,2):
for j in xrange(0,10,2):
# do function and fill the array corresponding to (i,j)
a[i/2, j/2] = 1
But I second Saullo Castro, you should try to vectorize your computations.
I am trying to implement the following equation using scipy's sparse package:
W = x[:,1] * y[:,1].T + x[:,2] * y[:,2].T + ...
where x & y are a nxm csc_matrix. Basically I'm trying to multiply each col of x by each col of y and sum the resulting nxn matrices together. I then want to make all non-zero elements 1.
This is my current implementation:
c = sparse.csc_matrix((n, n))
for i in xrange(0,m):
tmp = bam.id2sym_thal[:,i] * bam.id2sym_cort[:,i].T
minimum(tmp.data,ones_like(tmp.data),tmp.data)
maximum(tmp.data,ones_like(tmp.data),tmp.data)
c = c + tmp
This implementation has the following problems:
Memory usage seems to explode. As I understand it, memory should only increase as c becomes less sparse, but I am seeing that the loop starts eating up >20GB of memory with a n=10,000, m=100,000 (each row of x & y only has around 60 non-zero elements).
I'm using a python loop which is not very efficient.
My question: Is there a better way to do this? Controlling memory usage is my first concern, but it would be great to make it faster!
Thank you!
Note that a sum of outer products in the manner you describe is simply the same as multiplying two matrices together. In other words,
sum_i X[:,i]*Y[:,i].T == X*Y.T
So just multiply the matrices together.
Z = X*Y.T
For n=10000 and m=100000 and where each column has one nonzero element in both X and Y, it computes almost instantly on my laptop.
In terms of memory and performance, this might be a prime candidate for using Cython.
There is a section of the following paper describing its use with sparse scipy matricies:
http://folk.uio.no/dagss/cython_cise.pdf