So suppose i have two numpy ndarrays whose elements are matrices. I need element-wise multiplication for these two arrays, however, there should be matrix multiplication between the two matrix elements. Of course i would be able to implement this with for loops but i was looking to solve this problem without using an explicit for loop. How do i implement this?
EDIT: This for-loop does what I want to do. I'm on python 2.7
n = np.arange(8).reshape(2,2,1,2)
l = np.arange(1,9).reshape(2,2,2,1)
k = np.zeros((2,2))
for i in range(len(n)):
for j in range(len(n[i])):
k[i][j] = np.asscalar(n[i][j].dot(l[i][j]))
print k
Assuming your arrays of matrices are given as n+2 dimensional arrays A and B. What you want to achieve is as simple as C = A#B
Example
outer_dims = 2,3,4
inner_dims = 4,5,6
A = np.random.randint(0,10,(*outer_dims, *inner_dims[:2]))
B = np.random.randint(0,10,(*outer_dims, *inner_dims[1:]))
C = A#B
# check
for I in np.ndindex(outer_dims):
assert (C[I] == A[I]#B[I]).all()
UPDATE: Py2 version; thanks # hpaulj, Divakar
A = np.random.randint(0,10, outer_dims + inner_dims[:2])
B = np.random.randint(0,10, outer_dims + inner_dims[1:])
C = np.matmul(A,B)
# check
for I in np.ndindex(outer_dims):
assert (C[I] == np.matmul(A[I],B[I])).all()
If I understand correctly, this might work:
import numpy as np
a = np.array([[1,1],[1,0]])
b = np.array([[3,4],[5,4]])
x = np.array([[a,b],[b,a]])
y = np.array([[a,a],[b,b]])
result = np.array([_x # _y for _x, _y in zip(x,y)])
Related
Suppose A and B are two 4 dimensional numpy arrays with the same dimension.
A = np.random.rand(5,5,2,10)
B = np.random.rand(5,5,2,10)
a, b, c, d = A.shape
dat = []
for k in range(d):
sum = 0
for l in range(c):
sum = sum + np.einsum('ij,ji->', A[:,:,l,k], B[:,:,l,k])
dat.append(sum)
I was wondering whether I can use the "einsum" to replace the inner for loop, maybe even outer for loop, or maybe some matrix manipulation to replace all of it, casue the data set is large.
Is there any faster way to achieve this?
I am trying to conduct something similar to searchsorted, but in the case where the array is not completely monotonic. Say I have a scalar, c and a 1D array x, I want to find the indices i of all elements such that x[i] < c <= x[i + 1]. Importantly, x is not completely monotonic.
The following code works, but I just would like to know if this is the most efficient way to do this, or if there is a simper way:
x = np.array([1,2,3,1,2,3,1,2,3])
c = 2.5
t = c > x[:-1]
u = c <= x[1:]
v = t*u
i = v.nonzero()[0]
Or in one line of code:
i = ( (c > x[:-1]) * (c <= x[1:] ).nonzero()[0]
Is this the most efficient way to recover these indices?
Two additional questions.
Is there an easy way to extend this to the case where c is a 1D array and x is a 2D array, where c has as many elements as "rows" in x, and I perform this search for each element of c in the corresponding "row" of x?
My ultimate goal is to do this with a three dimensional case. That is, suppose c is still a 1D vector with n elements. Now, let x be a 3D array, with dimensions j by n by k. Is there a way to do #1 above for each "submatrix" in x? Basically, performing #1 above j times.
For example:
x1 = np.array([1,2,3,1,2,3],[1,2,3,1,2,3],[1,2,3,1,2,3])
x2 = x1 + 1
x = np.array([x1,x2])
c = np.array([1.5,2.5,3.5])
Under #1 above, when we compare c and x1, we would get: [[0,4],[1,5],[]]
When we compare c and x2, we would get: [[],[0,4],[1,5]]
Finally, under #2, I would like to get:
[[[0,4],[1,5],[]],
[[],[0,4],[1,5]]]
We could compare once to give us the boolean mask and re-use it with negation to get the other comparison array and also use slicing -
m = c > x
i = np.flatnonzero( m[:-1] & ~m[1:] )
We can extend it to x as 2D and c as 1D case with a loop, but do minimal computations with it by pre-computing on the masks generation in a vectorized manner, like so -
m = c[:,None] > x
m2 = m[:,:-1] & ~m[:,1:]
i = [np.flatnonzero( mi ) for mi in m2]
On such task, numpy make too much comparisons. You can win a 5X factor with Numba. No difficulties to adapt for 3 dimensions.
#numba.njit
def ind(x,c):
res = empty_like(x)
i=j=0
while i < x.size-1:
if x[i]<c and c<=x[i+1]:
res[j]=i
j+=1
i+=1
return res[:j]
Say I have 4 numpy arrays A,B,C,D , each the size of (256,256,1792).
I want to go through each element of those arrays and do something to it, but I need to do it in chunks of 256x256x256-cubes.
My code looks like this:
for l in range(7):
x, y, z, t = 0,0,0,0
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
t += D[m,n,o] * constant
x += A[m,n,o] * D[m,n,o] * constant
y += B[m,n,o] * D[m,n,o] * constant
z += C[m,n,o] * D[m,n,o] * constant
final = (x+y+z)/t
doOutput(final)
The code works and outputs exactly what I want, but its awfully slow. I've read online that those kind of nested for loops should be avoided in python. What is the cleanest solution to it? (right now I'm trying to do this part of my code in C and somehow import it via Cython or other tools, but I'd love a pure python solution)
Thanks
Add on
Willem Van Onsem's Solution to the first part seems to work just fine and I think I comprehend it. But now I want to modify my values before summing them. It looks like
(within the outer l loop)
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
R += (D[m,n,o] * constant * (A[m,n,o]**2
+ B[m,n,o]**2 + C[m,n,o]**2)/t - final**2)
doOutput(R)
I obviously can't just square the sum x = (A[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()**2*constant since (A²+B²) != (A+B)²
How can I redo this last for loops?
Since you update t with every element of m in range(a.shape[0]), n in range(a.shape[1]) and o in range(256*l,256*(l+1)), you can substitute:
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
t += D[m,n,o]
With:
t += D[:a.shape[0],:a.shape[1],256*l:256*(l+1)].sum()
The same for the other assignments. So you can rewrite your code to:
for l in range(7):
Dsub = D[:a.shape[0],:a.shape[1],256*l:256*(l+1)]
x = (A[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
y = (B[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
z = (C[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
t = Dsub.sum()*constant
final = (x+y+z)/t
doOutput(final)
Note that the * in numpy is the element-wise multiplication, not the matrix product. You can do the multiplication before the sum, but since the sum of a multiplications with a constant is equal to the multiplication of that constant with the sum, I think it is more efficient to do this out of the loop.
If a.shape[0] is equal to D.shape[0], etc. You can use : instead of :a.shape[0]. Based on your question, that seems to be the case. so:
# only when `a.shape[0] == D.shape[0], a.shape[1] == D.shape[1] (and so for A, B and C)`
for l in range(7):
Dsub = D[:,:,256*l:256*(l+1)]
x = (A[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
y = (B[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
z = (C[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
t = Dsub.sum()*constant
final = (x+y+z)/t
doOutput(final)
Processing the .sum() on the numpy level will boost performance since you do not convert values back and forth and with .sum(), you use a tight loop.
EDIT:
Your updated question does not change much. You can simply use:
m,n,_* = a.shape
lo,hi = 256*l,256*(l+1)
R = (D[:m,:n,lo:hi]*constant*(A[:m,:n,lo:hi]**2+B[:m,:n,lo:hi]**2+D[:m,:n,lo:hi]**2)/t-final**2)).sum()
doOutput(R)
how should i compare more than 2 numpy arrays?
import numpy
a = numpy.zeros((512,512,3),dtype=numpy.uint8)
b = numpy.zeros((512,512,3),dtype=numpy.uint8)
c = numpy.zeros((512,512,3),dtype=numpy.uint8)
if (a==b==c).all():
pass
this give a valueError, and i am not interested in comparing arrays two at a time.
For three arrays, you can check for equality among the corresponding elements between the first and second arrays and then second and third arrays to give us two boolean scalars and finally see if both of these scalars are True for final scalar output, like so -
np.logical_and( (a==b).all(), (b==c).all() )
For more number of arrays, you could stack them, get the differentiation along the axis of stacking and check if all of those differentiations are equal to zeros. If they are, we have equality among all input arrays, otherwise not. The implementation would look like so -
L = [a,b,c] # List of input arrays
out = (np.diff(np.vstack(L).reshape(len(L),-1),axis=0)==0).all()
For three arrays, you should really just compare them two at a time:
if np.array_equal(a, b) and np.array_equal(b, c):
do_whatever()
For a variable number of arrays, let's suppose they're all combined into one big array arrays. Then you could do
if np.all(arrays[:-1] == arrays[1:]):
do_whatever()
To expand on previous answers, I would use combinations from itertools to construct all pairs, then run your comparison on each pair. For example, if I have three arrays and want to confirm that they're all equal, I'd use:
from itertools import combinations
for pair in combinations([a, b, c], 2):
assert np.array_equal(pair[0], pair[1])
solution supporting different shapes and nans
compare against first element of array-list:
import numpy as np
a = np.arange(3)
b = np.arange(3)
c = np.arange(3)
d = np.arange(4)
lst_eq = [a, b, c]
lst_neq = [a, b, d]
def all_equal(lst):
for arr in lst[1:]:
if not np.array_equal(lst[0], arr, equal_nan=True):
return False
return True
print('all_equal(lst_eq)=', all_equal(lst_eq))
print('all_equal(lst_neq)=', all_equal(lst_neq))
output
all_equal(lst_eq)= True
all_equal(lst_neq)= False
for equal shape and without nan-support
Combine everything into one array, calculate the absolute diff along the new axis and check if the maximum element along the new dimension is equal 0 or lower than some threshold. This should be quite fast.
import numpy as np
a = np.arange(3)
b = np.arange(3)
c = np.arange(3)
d = np.array([0, 1, 3])
lst_eq = [a, b, c]
lst_neq = [a, b, d]
def all_equal(lst, threshold = 0):
arr = np.stack(lst, axis=0)
return np.max(np.abs(np.diff(arr, axis=0))) <= threshold
print('all_equal(lst_eq)=', all_equal(lst_eq))
print('all_equal(lst_neq)=', all_equal(lst_neq))
output
all_equal(lst_eq)= True
all_equal(lst_neq)= False
This might work.
import numpy
x = np.random.rand(10)
arrays = [x for _ in range(10)]
print(np.allclose(arrays[:-1], arrays[1:])) # True
arrays.append(np.random.rand(10))
print(np.allclose(arrays[:-1], arrays[1:])) # False
one-liner solution:
arrays = [a, b, c]
all([np.array_equal(a, b) for a, b in zip(arrays, arrays[1:])])
We test the equality of consecutive pairs of arrays
I am assigning values to a numpy array by looking up values in other numpy arrays. These arrays have potentially different indices. Here is an example:
import numpy as np
A=1; B=2; C=3; D=4; E=5
X = np.random.normal(0,1,(A,B,C,E))
Y = np.random.normal(0,1,(A,B,D))
Z = np.random.normal(0,1,(A,C))
Result = np.zeros((A,B,C,D,E))
for a in range(A):
for b in range(B):
for c in range(C):
for d in range(D):
for e in range(E):
Result[a,b,c,d,e] = Z[a,c] + Y[a,b,d] + X[a,b,c,e]
What is the best way to optimize this code? I can remove the E for loop using Result[a,b,c,d,:] = Z[a,c] + Y[a,b,d] + X[a,b,c,:]. But then how to remove the rest of the loops? I was also thinking that I could manipulate X,Y,Z before assignment so it merges easily with the dimensions of Result. There must be more elegant ways. Thanks for tips.
Here's one way:
Result = Z[:,None,:,None,None] + Y[:,:,None,:,None] + X[:,:,:,None,:]
To produce this vectorized version, all I did was replace the various indices into X, Y, and Z with full a,b,c,d,e-style indexing, inserting None where missing indices were found. For example, Y[a,b,d] becomes Y[a,b,None,d,None], which vectorizes into Y[:,:,None,:,None].
In numpy, indexing by None tells the array to pretend like it has an additional axis. This doesn't change the size of the array, but it does change how operations get broadcasted, which is what we need here. Check out the numpy broadcasting docs for more info.