I have tried to apply the function np.nditer() like zip() with arrays of different dimensions, where the iterator should use only the first dimensions.
Minimal example
a_all = np.arange(6).reshape(2,3)
idx_all = np.arange(12).reshape(2,3,2)
for a, idx in np.nditer([a_all, idx_all]):
print((a, idx))
Which throws the error:
ValueError: operands could not be broadcast together with shapes (2,3) (2,3,2)
My use case
I have two arrays with data which I want to calculate each other. Furthermore I have an index list for another array. So I try:
a_all = np.arange(6).reshape(2,3)
b_all = np.arange(6).reshape(2,3)
idx_all = (
((0,0), (0,1), (0,2)),
((1,0), (1,1), (1,2))
)
result = np.zeros((2,3))
for a, b, idx in np.nditer([a_all, b_all, idx_all]):
result[idx] += a*b
Which throws the same error like the minimal example.
I assume the problem is that np.nditer() tries to iterate over all dimensions of idx_all, but I couldn't figure out how to limit it to the first two.
zip() I do not want to use, otherwise I would need two loops:
for a_, b_, idx_ in zip(a_all, b_all, idx_all):
for a, b, idx in zip(a_, b_, idx_):
result[idx] += a*b
More sensible example
a_all = np.random.randn(2,3)
b_all = np.random.randn(2)
idx_all = (
((1,1), (2,2))
)
result = np.zeros(2)
for a, b, idx, res in np.nditer([a_all, b_all, idx_all, result], op_flags=['readwrite']):
res += a[idx] + b
Look at the first case, corrected so the arrays do broadcast together (if you don't understand what I've changed, you have read enough basic numpy docs.)
In [14]: a_all = np.arange(6).reshape(2,3,1)
...: idx_all = np.arange(12).reshape(2,3,2)
...:
...: for a, idx in np.nditer([a_all, idx_all]):
...: print((a, idx))
...:
(array(0), array(0))
(array(0), array(1))
(array(1), array(2))
(array(1), array(3))
(array(2), array(4))
(array(2), array(5))
(array(3), array(6))
(array(3), array(7))
(array(4), array(8))
(array(4), array(9))
(array(5), array(10))
(array(5), array(11))
nditer iterates in a 'flat' sense, passing single element arrays (0d) to the body. It's not like zip which just iterates on the first dimension (or outer layer of nested lists).
np.vectorize (which I don't recommend either), does the same sort of broadcasting, but passes python scalar elements to the function instead:
In [15]: np.vectorize(lambda a,idx: print((a,idx)))(a_all,idx_all)
(0, 0) # test run
(0, 0)
(0, 1)
(1, 2)
(1, 3)
(2, 4)
(2, 5)
(3, 6)
(3, 7)
(4, 8)
(4, 9)
(5, 10)
(5, 11)
Out[15]:
array([[[None, None],
[None, None],
[None, None]],
[[None, None],
[None, None],
[None, None]]], dtype=object)
nditer needs the same sort of performance disclaimer as np.vectorize. It doesn't help, at least not when using in python code. In cython it can be useful, as demonstrated in the larger nditer documentation page.
Also nditer inputs can be complex, as shown by the TypeError that your last example produces.
Your last example:
I had to change idx_all to array, not tuple, so it can be readwrite able. Read the op_flags docs more carefully.
And we still get the broadcasting error. It isn't iterating the 'first layer'.
In [24]: a_all = np.random.randn(2,3)
...: b_all = np.random.randn(2)
...: idx_all = (
...: ((1,1), (2,2))
...: ); idx_all=np.array(idx_all)
...: result = np.zeros(2)
...:
...: for a, b, idx, res in np.nditer([a_all, b_all, idx_all, result], op_flags=['readwrite']):
...: res += a[idx] + b
...:
ValueError: operands could not be broadcast together with shapes (2,3) (2,) (2,2) (2,)
Related
I have a function, remrow which takes as input an arbitrary numpy nd array, arr, and an integer, n. My function should remove the last row from arr in the nth dimension. For example, if call my function like so:
remrow(arr,2)
with arr as a 3d array, then my function should return:
arr[:,:,:-1]
Similarly if I call;
remrow(arr,1)
and arr is a 5d array, then my function should return:
arr[:,:-1,:,:,:]
My problem is this; my function must work for all shapes and sizes of arr and all compatible n. How can I do this with numpy array indexing?
Construct an indexing tuple, consisting of the desired combination of slice(None) and slice(None,-1) objects.
In [75]: arr = np.arange(24).reshape(2,3,4)
In [76]: idx = [slice(None) for _ in arr.shape]
In [77]: idx
Out[77]: [slice(None, None, None), slice(None, None, None), slice(None, None, None)]
In [78]: idx[1]=slice(None,-1)
In [79]: arr[tuple(idx)].shape
Out[79]: (2, 2, 4)
In [80]: idx = [slice(None) for _ in arr.shape]
In [81]: idx[2]=slice(None,-1)
In [82]: arr[tuple(idx)].shape
Out[82]: (2, 3, 3)
Say I have two numpy arrays, for example
import numpy as np
A = np.arange(5*3*3*2).reshape(5, 3, 3, 2)
B = np.arange(3*3).reshape(3, 3)
If I want to add A and B across a shared axis, I would just do
C = A + B[None, :, :, None]
# C has shape (5, 3, 3, 2) which is what I want
I want to write a write function that generalizes this kind of summation but am not how to get started. It would look something like
def mask(M, Mshape, out_shape):
# not sure what to put here
pass
def add_tensors(A, B, Ashape, Bshape, out_shape):
# Here I mask A, B so that it has shape out_shape
A = mask(A, Aaxis, out_shape)
B = mask(B, Baxis, out_shape)
return A + B
Any suggestions? Is it possible to make this a ufunc?
In [447]: A = np.arange(5*3*3*2).reshape(5, 3, 3, 2)
...: B = np.arange(3*3).reshape(3, 3)
These are all equivalent:
In [448]: A + B[None,:, :, None];
In [449]: A + B[:, :, None]; # initial None is automatic
build the index tuple from list:
In [454]: tup = [slice(None)]*3; tup[-1] = None; tup = tuple(tup)
In [455]: tup
Out[455]: (slice(None, None, None), slice(None, None, None), None)
In [456]: A + B[tup];
or the equivalent shape:
In [457]: sh = B.shape + (1,)
In [458]: sh
Out[458]: (3, 3, 1)
In [459]: A + B.reshape(sh);
expand_dims also uses a parameterized reshape:
In [462]: np.expand_dims(B,2).shape
Out[462]: (3, 3, 1)
In [463]: A+np.expand_dims(B,2);
I have two 3D matrices:
a = np.random.normal(size=[3,2,5])
b = np.random.normal(size=[5,2,3])
I want the dot product of each slice along 2 and 0 axes respectively:
c = np.zeros([3,3,5]) # c.size is 45
c[:,:,0] = a[:,:,0].dot(b[0,:,:])
c[:,:,1] = a[:,:,1].dot(b[1,:,:])
...
I would like to do that using np.tensordot (for efficiency and speed)
I have tried:
c = np.tensordot(a, b, axes=[2,0])
but I get a 4D array with 36 elements (instead of 45). c.shape, c.size = ((3L, 2L, 2L, 3L), 36). I have found a similar question here (Numpy tensor: Tensordot over frontal slices of tensor) but it's not exactly what I want, and I was unable to extrapolate that solution to my problem.
To summarise, can I use np.tensordot to compute c array show above?
Update #1
The answer by #hpaulj is what I wanted, however in my system (python 2.7 and np 1.13.3) those aproaches are pretty slow:
n = 3000
a = np.random.normal(size=[n, 20, 5])
b = np.random.normal(size=[5, 20, n])
t = time.clock()
c_slice = a[:,:,0].dot(b[0,:,:])
print('one slice_x_5: {:.3f} seconds'.format( (time.clock()-t)*5 ))
t = time.clock()
c = np.zeros([n, n, 5])
for i in range(5):
c[:,:,i] = a[:,:,i].dot(b[i,:,:])
print('for loop: {:.3f} seconds'.format(time.clock()-t))
t = time.clock()
d = np.einsum('abi,ibd->adi', a, b)
print('einsum: {:.3f} seconds'.format(time.clock()-t))
t = time.clock()
e = np.tensordot(a,b,[1,1])
e1 = e.transpose(0,3,1,2)[:,:,np.arange(5),np.arange(5)]
print('tensordot: {:.3f} seconds'.format(time.clock()-t))
a = a.transpose(2,0,1)
t = time.clock()
f = np.matmul(a,b)
print('matmul: {:.3f} seconds'.format(time.clock()-t))
It's easier to work with einsum than tensordot. So let's start there:
In [469]: a = np.random.normal(size=[3,2,5])
...: b = np.random.normal(size=[5,2,3])
...:
In [470]: c = np.zeros([3,3,5]) # c.size is 45
In [471]: for i in range(5):
...: c[:,:,i] = a[:,:,i].dot(b[i,:,:])
...:
In [472]: d = np.einsum('abi,ibd->iad', a, b)
In [473]: d.shape
Out[473]: (5, 3, 3)
In [474]: d = np.einsum('abi,ibd->adi', a, b)
In [475]: d.shape
Out[475]: (3, 3, 5)
In [476]: np.allclose(c,d)
Out[476]: True
I had to think a bit about to match up the dimensions. It helped to focus on a[:,:,i] as 2d, and similarly for b[i,:,:]. So the dot sum is over the middle dimension of both arrays (size 2).
In testing ideas it might help if the first 2 dimensions of c were different. There'd be less chance of mixing them up.
It's easy to specify the dot summation axis (axes) in tensordot, but harder to constrain the handling of the other dimensions. That's why you get a 4d array.
I can get it to work with a transpose, followed by taking the diagonal:
In [477]: e = np.tensordot(a,b,[1,1])
In [478]: e.shape
Out[478]: (3, 5, 5, 3)
In [479]: e1 = e.transpose(0,3,1,2)[:,:,np.arange(5),np.arange(5)]
In [480]: e1.shape
Out[480]: (3, 3, 5)
In [481]: np.allclose(c,e1)
Out[481]: True
I've calculated a lot more values than needed, and thrown most of them away.
matmul with some transposing might work better.
In [482]: f = a.transpose(2,0,1)#b
In [483]: f.shape
Out[483]: (5, 3, 3)
In [484]: np.allclose(c, f.transpose(1,2,0))
Out[484]: True
I think of the 5 dimension as 'going-along-for-ride'. That's what your loop does. In einsum the i is the same in all parts.
Normally, when we know where should we insert the newaxis, we can do a[:, np.newaxis,...]. Is there any good way to insert the newaxis at certain axis?
Here is how I do it now. I think there must be some much better ways than this:
def addNewAxisAt(x, axis):
_s = list(x.shape)
_s.insert(axis, 1)
return x.reshape(tuple(_s))
def addNewAxisAt2(x, axis):
ind = [slice(None)]*x.ndim
ind.insert(axis, np.newaxis)
return x[ind]
That singleton dimension (dim length = 1) could be added as a shape criteria to the original array shape with np.insert and thus directly change its shape, like so -
x.shape = np.insert(x.shape,axis,1)
Well, we might as well extend this to invite more than one new axes with a bit of np.diff and np.cumsum trick, like so -
insert_idx = (np.diff(np.append(0,axis))-1).cumsum()+1
x.shape = np.insert(x.shape,insert_idx,1)
Sample runs -
In [151]: def addNewAxisAt(x, axis):
...: insert_idx = (np.diff(np.append(0,axis))-1).cumsum()+1
...: x.shape = np.insert(x.shape,insert_idx,1)
...:
In [152]: A = np.random.rand(4,5)
In [153]: addNewAxisAt(A, axis=1)
In [154]: A.shape
Out[154]: (4, 1, 5)
In [155]: A = np.random.rand(5,6,8,9,4,2)
In [156]: addNewAxisAt(A, axis=5)
In [157]: A.shape
Out[157]: (5, 6, 8, 9, 4, 1, 2)
In [158]: A = np.random.rand(5,6,8,9,4,2,6,7)
In [159]: addNewAxisAt(A, axis=(1,3,4,6))
In [160]: A.shape
Out[160]: (5, 1, 6, 1, 1, 8, 1, 9, 4, 2, 6, 7)
np.insert does
slobj = [slice(None)]*ndim
...
slobj[axis] = slice(None, index)
...
new[slobj] = arr[slobj2]
Like you it constructs a list of slices, and modifies one or more elements.
apply_along_axis constructs an array, and converts it to indexing tuple
outarr[tuple(i.tolist())] = res
Other numpy functions work this way as well.
My suggestion is to make initial list large enough to hold the None. Then I don't need to use insert:
In [1076]: x=np.ones((3,2,4),int)
In [1077]: ind=[slice(None)]*(x.ndim+1)
In [1078]: ind[2]=None
In [1080]: x[ind].shape
Out[1080]: (3, 2, 1, 4)
In [1081]: x[tuple(ind)].shape # sometimes converting a list to tuple is wise
Out[1081]: (3, 2, 1, 4)
Turns out there is a np.expand_dims
In [1090]: np.expand_dims(x,2).shape
Out[1090]: (3, 2, 1, 4)
It uses reshape like you do, but creates the new shape with tuple concatenation.
def expand_dims(a, axis):
a = asarray(a)
shape = a.shape
if axis < 0:
axis = axis + len(shape) + 1
return a.reshape(shape[:axis] + (1,) + shape[axis:])
Timings don't tell me much about which is better. They are the 2 µs range, where simply wrapping the code in a function makes a difference.
I have a two 1 dimensional arrays, a such that np.shape(a) == (n,) and b such that np.shape(b) == (m,).
I want to make a (3rd order) tensor c such that np.shape(c) == (n,n,m,)by doing c = np.outer(np.outer(a,a),b).
But when I do this, I get:
>> np.shape(c)
(n*n,m)
which is just a rectangular matrix. How can I make a 3D tensor like I want?
You could perhaps use np.multiply.outer instead of np.outer to get the required outer product:
>>> a = np.arange(4)
>>> b = np.ones(5)
>>> mo = np.multiply.outer
Then we have:
>>> mo(mo(a, a), b).shape
(4, 4, 5)
A better way could be to use np.einsum (this avoids creating intermediate arrays):
>>> c = np.einsum('i,j,k->ijk', a, a, b)
>>> c.shape
(4, 4, 5)