I have a mXn numpy array called a:
I would like to write a function which returns an array with size (3, mxn) which contains for each couple (x,y) in the first array the correspondant value.
import numpy as np
m=5
n=10
a = np.random.random((m, n))
x = np.random.random((m, 1)) # x coordinates
y = np.random.random((1, n)) # y coordinates
b = np.empty((3, m*n)) # array to store coordinates
k=0
for i in range (0,m):
for j in range (0,n):
b[0,k] = a[i,0]
b[1,k] = a[0,j]
b[2,k] = a[i,j]
k=k+1
This seems to run ok, but is there a faster or better coded way to do this?
Steps :
Initialize a 3D array, such that m and n are separate ones. This lets us broadcast values.
Index into the first three elements along the first axis of output with appropriate elements off a and make sure that those shapes are broadcastable.
Reshape the output back to 2D.
That's all the play is about here! Here's the vectorized implementation -
b_out = np.empty((3, m,n),dtype=a.dtype) # 1. Initialize
b_out[0] = a[:,0,None] # 2. Assign
b_out[1] = a[0]
b_out[2] = a
b_out.shape = (3,m*n) # 3. Reshape back to 2D
Runtime test
Approaches -
def loopy_app(a):
m,n = a.shape
b = np.empty((3, m*n),dtype=a.dtype)
k=0
for i in range (0,m):
for j in range (0,n):
b[0,k] = a[i,0]
b[1,k] = a[0,j]
b[2,k] = a[i,j]
k=k+1
return b
def vectorized_app(a):
b_out = np.empty((3, m,n),dtype=a.dtype)
b_out[0] = a[:,0,None]
b_out[1] = a[0]
b_out[2] = a
b_out.shape = (3,m*n)
return b_out
Timings -
In [194]: m=5
...: n=10
...: a = np.random.random((m, n))
...:
In [195]: %timeit loopy_app(a)
...: %timeit vectorized_app(a)
...:
10000 loops, best of 3: 28.2 µs per loop
100000 loops, best of 3: 2.48 µs per loop
In [196]: m=50
...: n=100
...: a = np.random.random((m, n))
...:
In [197]: %timeit loopy_app(a)
...: %timeit vectorized_app(a)
...:
100 loops, best of 3: 2.56 ms per loop
100000 loops, best of 3: 6.31 µs per loop
In [198]: 2560/6.31
Out[198]: 405.7052297939778
400x+ speedup on large datasets and more on larger ones!
Related
I have two very large numpy arrays, which are both 3D. I need to find an efficient way to check if they are overlapping, because turning them both into sets first takes too long. I tried to use another solution I found here for this same problem but for 2D arrays, but I didn't manage to make it work for 3D.
Here is the solution for 2D:
nrows, ncols = A.shape
dtype={'names':['f{}'.format(i) for i in range(ndep)],
'formats':ndep * [A.dtype]}
C = np.intersect1d(A.view(dtype).view(dtype), B.view(dtype).view(dtype))
# This last bit is optional if you're okay with "C" being a structured array...
C = C.view(A.dtype).reshape(-1, ndep)
(where A and B are the 2D arrays)
I need to find the number of overlapping numpy arrays, but not the specific ones.
We could leverage views using a helper function that I have used across few Q&As. To get the presence of subarrays, we could use np.isin on the views or use a more laborious one with np.searchsorted.
Approach #1 : Using np.isin -
# https://stackoverflow.com/a/45313353/ #Divakar
def view1D(a, b): # a, b are arrays
a = np.ascontiguousarray(a)
b = np.ascontiguousarray(b)
void_dt = np.dtype((np.void, a.dtype.itemsize * a.shape[1]))
return a.view(void_dt).ravel(), b.view(void_dt).ravel()
def isin_nd(a,b):
# a,b are the 3D input arrays to give us "isin-like" functionality across them
A,B = view1D(a.reshape(a.shape[0],-1),b.reshape(b.shape[0],-1))
return np.isin(A,B)
Approach #2 : We could also leverage np.searchsorted upon the views -
def isin_nd_searchsorted(a,b):
# a,b are the 3D input arrays
A,B = view1D(a.reshape(a.shape[0],-1),b.reshape(b.shape[0],-1))
sidx = A.argsort()
sorted_index = np.searchsorted(A,B,sorter=sidx)
sorted_index[sorted_index==len(A)] = len(A)-1
idx = sidx[sorted_index]
return A[idx] == B
So, these two solutions give us the mask of presence of each of the subarrays from a in b. Hence, to get our desired count, it would be - isin_nd(a,b).sum() or isin_nd_searchsorted(a,b).sum().
Sample run -
In [71]: # Setup with 3 common "subarrays"
...: np.random.seed(0)
...: a = np.random.randint(0,9,(10,4,5))
...: b = np.random.randint(0,9,(7,4,5))
...:
...: b[1] = a[4]
...: b[3] = a[2]
...: b[6] = a[0]
In [72]: isin_nd(a,b).sum()
Out[72]: 3
In [73]: isin_nd_searchsorted(a,b).sum()
Out[73]: 3
Timings on large arrays -
In [74]: # Setup
...: np.random.seed(0)
...: a = np.random.randint(0,9,(100,100,100))
...: b = np.random.randint(0,9,(100,100,100))
...: idxa = np.random.choice(range(len(a)), len(a)//2, replace=False)
...: idxb = np.random.choice(range(len(b)), len(b)//2, replace=False)
...: a[idxa] = b[idxb]
# Verify output
In [82]: np.allclose(isin_nd(a,b),isin_nd_searchsorted(a,b))
Out[82]: True
In [75]: %timeit isin_nd(a,b).sum()
10 loops, best of 3: 31.2 ms per loop
In [76]: %timeit isin_nd_searchsorted(a,b).sum()
100 loops, best of 3: 1.98 ms per loop
I'm trying to compute the pairwise np.vdot of a complex 2D array x with itself. So the behaviour I want is:
X = np.empty((x.shape[0], x.shape[0]), dtype='complex128')
for i in range(x.shape[0]):
for j in range(x.shape[0]):
X[i, j] = np.vdot(x[i], x[j])
Is there a way to do this without the explicit loops? I tried using pairwise_kernel from sklearn but it assumes the input arrays are real numbers. I also tried broadcasting, but vdot flattens its inputs.
X = np.einsum('ik,jk->ij', np.conj(x), x)
is equivalent to
X = np.empty((x.shape[0], x.shape[0]), dtype='complex128')
for i in range(x.shape[0]):
for j in range(x.shape[0]):
X[i, j] = np.vdot(x[i], x[j])
np.einsum
takes a sum of products. The subscript 'ik,jk->ij' tells np.einsum that the second argument,
np.conj(x) is an array with subscripts ik and the third argument, x has
subscripts jk. Thus, the product np.conj(x)[i,k]*x[j,k] is computed for all
i,j,k. The sum is taken over the repeated subscript, k, and since that
leaves i and j remaining, they become the subscripts of the resultant array.
For example,
import numpy as np
N, M = 10, 20
a = np.random.random((N,M))
b = np.random.random((N,M))
x = a + b*1j
def orig(x):
X = np.empty((x.shape[0], x.shape[0]), dtype='complex128')
for i in range(x.shape[0]):
for j in range(x.shape[0]):
X[i, j] = np.vdot(x[i], x[j])
return X
def alt(x):
return np.einsum('ik,jk->ij', np.conj(x), x)
assert np.allclose(orig(x), alt(x))
In [307]: %timeit orig(x)
10000 loops, best of 3: 143 µs per loop
In [308]: %timeit alt(x)
100000 loops, best of 3: 8.63 µs per loop
To extend the np.vdot to all rows, you can use np.tensordot and I am borrowing the conjugate idea straight off #unutbu's solution , like so -
np.tensordot(np.conj(x),x,axes=(1,1))
Basically with np.tensordot, we specify the axes to be reduced, which in this case is the last axis for the conjugate version of x and the array itself, when applied on those two.
Runtime test -
Let's time #unutbu's solution with np.einsum and the proposed solution in this post -
In [27]: import numpy as np # From #unutbu's` solution again
...:
...: N, M = 1000, 1000
...: a = np.random.random((N,M))
...: b = np.random.random((N,M))
...: x = a + b*1j
...:
In [28]: %timeit np.einsum('ik,jk->ij', np.conj(x), x) # #unutbu's` solution
1 loops, best of 3: 4.45 s per loop
In [29]: %timeit np.tensordot(np.conj(x),x,axes=(1,1))
1 loops, best of 3: 3.76 s per loop
Suppose I have an N-dimensional numpy array x and an (N-1)-dimensional index array m (for example, m = x.argmax(axis=-1)). I'd like to construct (N-1) dimensional array y such that y[i_1, ..., i_N-1] = x[i_1, ..., i_N-1, m[i_1, ..., i_N-1]] (for the argmax example above it would be equivalent to y = x.max(axis=-1)).
For N=3 I could achieve what I want by
y = x[np.arange(x.shape[0])[:, np.newaxis], np.arange(x.shape[1]), m]
The question is, how do I do this for an arbitrary N?
you can use indices :
firstdims=np.indices(x.shape[:-1])
And add yours :
ind=tuple(firstdims)+(m,)
Then x[ind] is what you want.
In [228]: allclose(x.max(-1),x[ind])
Out[228]: True
Here's one approach using reshaping and linear indexing to handle multi-dimensional arrays of arbitrary dimensions -
shp = x.shape[:-1]
n_ele = np.prod(shp)
y_out = x.reshape(n_ele,-1)[np.arange(n_ele),m.ravel()].reshape(shp)
Let's take a sample case with a ndarray of 6 dimensions and let's say we are using m = x.argmax(axis=-1) to index into the last dimension. So, the output would be x.max(-1). Let's verify this for the proposed solution -
In [121]: x = np.random.randint(0,9,(4,5,3,3,2,4))
In [122]: m = x.argmax(axis=-1)
In [123]: shp = x.shape[:-1]
...: n_ele = np.prod(shp)
...: y_out = x.reshape(n_ele,-1)[np.arange(n_ele),m.ravel()].reshape(shp)
...:
In [124]: np.allclose(x.max(-1),y_out)
Out[124]: True
I liked #B. M.'s solution for its elegance. So, here's a runtime test to benchmark these two -
def reshape_based(x,m):
shp = x.shape[:-1]
n_ele = np.prod(shp)
return x.reshape(n_ele,-1)[np.arange(n_ele),m.ravel()].reshape(shp)
def indices_based(x,m): ## #B. M.'s solution
firstdims=np.indices(x.shape[:-1])
ind=tuple(firstdims)+(m,)
return x[ind]
Timings -
In [152]: x = np.random.randint(0,9,(4,5,3,3,4,3,6,2,4,2,5))
...: m = x.argmax(axis=-1)
...:
In [153]: %timeit indices_based(x,m)
10 loops, best of 3: 30.2 ms per loop
In [154]: %timeit reshape_based(x,m)
100 loops, best of 3: 5.14 ms per loop
Given a list of rotation angles (lets say about the X axis):
import numpy as np
x_axis_rotations = np.radians([0,10,32,44,165])
I can create an array of matrices matching these angles by doing so:
matrices = []
for angle in x_axis_rotations:
matrices.append(np.asarray([[1 , 0 , 0],[0, np.cos(angle), -np.sin(angle)], [0, np.sin(angle), np.cos(angle)]]))
matrices = np.array(matrices)
This will work but it doesn't take advantage of numpy's strengths for dealing with large arrays... So if my array of angles is in the millions, doing it this way won't be very fast.
Is there a better (faster) way to do create an array of transform matrices from an array of inputs?
Here's a direct and simple approach:
c = np.cos(x_axis_rotations)
s = np.sin(x_axis_rotations)
matrices = np.zeros((len(x_axis_rotations), 3, 3))
matrices[:, 0, 0] = 1
matrices[:, 1, 1] = c
matrices[:, 1, 2] = -s
matrices[:, 2, 1] = s
matrices[:, 2, 2] = c
timings, for the curious:
In [30]: angles = 2 * np.pi * np.random.rand(1000)
In [31]: timeit OP(angles)
100 loops, best of 3: 5.46 ms per loop
In [32]: timeit askewchan(angles)
10000 loops, best of 3: 39.6 µs per loop
In [33]: timeit divakar(angles)
10000 loops, best of 3: 93.8 µs per loop
In [34]: timeit divakar_oneline(angles)
10000 loops, best of 3: 56.1 µs per loop
In [35]: timeit divakar_combine(angles)
10000 loops, best of 3: 43.9 µs per loop
All are much faster than your loop, so use whichever you like the most :)
You can use linear indexing to help out, like so -
# Get cosine and sine values in one-go
cosv = np.cos(x_axis_rotations)
sinv = np.sin(x_axis_rotations)
# Get size parameter
N = x_axis_rotations.size
# Initialize output array
out = np.zeros((N,3,3))
# Set the first element in each 3D slice as 1
out[:,0,0] = 1
# Calculate the first of positions where cosine valued elements are to be put
idx1 = 4 + 9*np.arange(N)[:,None]
# One by one put those 4 values in 2x2 blocks across all 3D slices
out.ravel()[idx1] = cosv
out.ravel()[idx1+1] = -sinv
out.ravel()[idx1+3] = sinv
out.ravel()[idx1+4] = cosv
Alternatively, you can set the elements in one-go after you have initialized the output array with zeros and set the first element in each slice as 1, like so -
out.reshape(N,-1)[:,[4,5,7,8]] = np.column_stack((cosv,-sinv,sinv,cosv))
Between the above mentioned two approaches, two more middleground approaches could evolve, again put right after initializing with zeros and setting the first element in each 3D slice as 1, like so -
out.reshape(N,-1)[:,[4,8]] = cosv[:,None]
out.reshape(N,-1)[:,[5,7]] = np.column_stack((-sinv[:,None],sinv[:,None]))
The last one would be -
out.reshape(N,-1)[:,[4,8]] = cosv[:,None]
out.reshape(N,-1)[:,5] = -sinv
out.reshape(N,-1)[:,7] = sinv
Title might be not as precise than I hoped, but here is the problem. Basically I'm filling a 4d numpy array from the entries of the product of two matrices. Right now the code is the following :
M = P.dot(U)
C_arr = np.zeros((b_size,b_size,N,N))
for alpha in xrange(b_size):
for beta in xrange(b_size):
for i in xrange(N):
for j in xrange(N):
C_arr[alpha,beta,i,j] = np.conjugate(M[i,alpha])*M[j,beta]
It turns out that this function is called quite ofen and appears to be very time-consumming. I'm just beginning with Python and I suspect that there could be a more efficient way to write this function by avoiding those loops, but haven't been able to figure it out by myself...
You can use numpy.einsum:
C = np.einsum('ia,jb->abij', M.conj(), M)
Or, since there is no actual sum being computed (i.e. this is a form of an outer product), you can use numpy broadcasting with regular array multiplication after reshaping the input matrix M appropriately:
nrows, ncols = M.shape
C = M.T.reshape(1, ncols, 1, nrows) * M.T.conj().reshape(ncols, 1, nrows, 1)
Apart from the terse code with np.einsum listed in the other solution, you can also use outer-product with np.outer like so -
np.outer(M.conj().ravel(),M.ravel()).reshape(N,b_size,N,b_size).transpose(1,3,0,2)
Runtime tests -
In [54]: # Create input and get shape parameters
...: M = np.random.rand(10,10)
...: N,b_size = M.shape
...:
In [55]: %timeit np.einsum('ia,jb->abij', M.conj(), M)
10000 loops, best of 3: 26 µs per loop
In [56]: %timeit np.outer(M.conj().ravel(),M.ravel()).reshape(N,b_size,N,b_size).transpose(1,3,0,2)
10000 loops, best of 3: 55.6 µs per loop
In [57]: # Create input and get shape parameters
...: M = np.random.rand(40,40)
...: N,b_size = M.shape
...:
In [58]: %timeit np.einsum('ia,jb->abij', M.conj(), M)
10 loops, best of 3: 31 ms per loop
In [59]: %timeit np.outer(M.conj().ravel(),M.ravel()).reshape(N,b_size,N,b_size).transpose(1,3,0,2)
10 loops, best of 3: 24.2 ms per loop
In [60]: # Create input and get shape parameters
...: M = np.random.rand(80,80)
...: N,b_size = M.shape
...:
In [61]: %timeit np.einsum('ia,jb->abij', M.conj(), M)
1 loops, best of 3: 497 ms per loop
In [62]: %timeit np.outer(M.conj().ravel(),M.ravel()).reshape(N,b_size,N,b_size).transpose(1,3,0,2)
1 loops, best of 3: 399 ms per loop
Thus, depending on the shape of the input array, you can go either way.