I have 2 arrays:
>>> a.shape
(9, 3, 11)
>>> b.shape
(9,)
I would like to compute the equivalent of c[i, j] = f(a[i, j, :], b[i]) where f(a0, b0) is a function that takes 2 parameters, with len(a0) == 11 and len(b0) == 9. Here, i is iterating on range(9) and j is iterating on range(3).
Is there a way to code this using numpy.vectorize? Or is it simpler with some clever broadcasting?
I have been trying for 2 hours and I just don't understand how to make it work... I tried to broadcast or to use signatures but to no avail.
In the end, I could make it work like this:
>>> f = np.vectorize(f, signature="(k),(1)->()")
>>> print(a.shape)
(9, 3, 11)
>>> print(b.shape)
(9,)
>>> print(f(a, b[:, None, None]).shape)
(9, 3)
This ensures that f gets called with the correct shapes and iterates properly. It is frankly not straightforward from the Numpy documentation to understand the trick to use a (1) in the signature for this purpose.
numpy.apply_along_axis is what you need.
import numpy as np
a = np.ones( (9,3,11) )
b = np.ones( 9 )
def f(a0, b0):
return sum(a0[:9]+b0)
c = np.apply_along_axis( f, 2, a, b )
print(c)
c's shape is (9,3).
Related
I have tried to apply the function np.nditer() like zip() with arrays of different dimensions, where the iterator should use only the first dimensions.
Minimal example
a_all = np.arange(6).reshape(2,3)
idx_all = np.arange(12).reshape(2,3,2)
for a, idx in np.nditer([a_all, idx_all]):
print((a, idx))
Which throws the error:
ValueError: operands could not be broadcast together with shapes (2,3) (2,3,2)
My use case
I have two arrays with data which I want to calculate each other. Furthermore I have an index list for another array. So I try:
a_all = np.arange(6).reshape(2,3)
b_all = np.arange(6).reshape(2,3)
idx_all = (
((0,0), (0,1), (0,2)),
((1,0), (1,1), (1,2))
)
result = np.zeros((2,3))
for a, b, idx in np.nditer([a_all, b_all, idx_all]):
result[idx] += a*b
Which throws the same error like the minimal example.
I assume the problem is that np.nditer() tries to iterate over all dimensions of idx_all, but I couldn't figure out how to limit it to the first two.
zip() I do not want to use, otherwise I would need two loops:
for a_, b_, idx_ in zip(a_all, b_all, idx_all):
for a, b, idx in zip(a_, b_, idx_):
result[idx] += a*b
More sensible example
a_all = np.random.randn(2,3)
b_all = np.random.randn(2)
idx_all = (
((1,1), (2,2))
)
result = np.zeros(2)
for a, b, idx, res in np.nditer([a_all, b_all, idx_all, result], op_flags=['readwrite']):
res += a[idx] + b
Look at the first case, corrected so the arrays do broadcast together (if you don't understand what I've changed, you have read enough basic numpy docs.)
In [14]: a_all = np.arange(6).reshape(2,3,1)
...: idx_all = np.arange(12).reshape(2,3,2)
...:
...: for a, idx in np.nditer([a_all, idx_all]):
...: print((a, idx))
...:
(array(0), array(0))
(array(0), array(1))
(array(1), array(2))
(array(1), array(3))
(array(2), array(4))
(array(2), array(5))
(array(3), array(6))
(array(3), array(7))
(array(4), array(8))
(array(4), array(9))
(array(5), array(10))
(array(5), array(11))
nditer iterates in a 'flat' sense, passing single element arrays (0d) to the body. It's not like zip which just iterates on the first dimension (or outer layer of nested lists).
np.vectorize (which I don't recommend either), does the same sort of broadcasting, but passes python scalar elements to the function instead:
In [15]: np.vectorize(lambda a,idx: print((a,idx)))(a_all,idx_all)
(0, 0) # test run
(0, 0)
(0, 1)
(1, 2)
(1, 3)
(2, 4)
(2, 5)
(3, 6)
(3, 7)
(4, 8)
(4, 9)
(5, 10)
(5, 11)
Out[15]:
array([[[None, None],
[None, None],
[None, None]],
[[None, None],
[None, None],
[None, None]]], dtype=object)
nditer needs the same sort of performance disclaimer as np.vectorize. It doesn't help, at least not when using in python code. In cython it can be useful, as demonstrated in the larger nditer documentation page.
Also nditer inputs can be complex, as shown by the TypeError that your last example produces.
Your last example:
I had to change idx_all to array, not tuple, so it can be readwrite able. Read the op_flags docs more carefully.
And we still get the broadcasting error. It isn't iterating the 'first layer'.
In [24]: a_all = np.random.randn(2,3)
...: b_all = np.random.randn(2)
...: idx_all = (
...: ((1,1), (2,2))
...: ); idx_all=np.array(idx_all)
...: result = np.zeros(2)
...:
...: for a, b, idx, res in np.nditer([a_all, b_all, idx_all, result], op_flags=['readwrite']):
...: res += a[idx] + b
...:
ValueError: operands could not be broadcast together with shapes (2,3) (2,) (2,2) (2,)
I'm currently struggling with a probably rather simple question but I can't get my head around it.
Assuming I have the follow two 2d arrays with different shapes, I can combine them into a new array using:
a = np.zeros((2, 3))
b = np.zeros((4, 5))
c = np.array([a, b])
print(c.shape)
# Output
# (2,)
for elements in c:
print(elements.shape)
# Output:
# (2, 3)
# (4, 5)
So far so good!
But how would I do this if I have a large list where I'd have to iterate over? Here is a simple example with just 4 different 2d arrays:
This works as expected:
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = np.array([a, b, c, d])
print(e.shape)
# Output
# (4,)
for elements in e:
print(elements.shape)
# Output
# (2, 3)
# (4, 5)
# (6, 7)
# (8, 9)
This doesn't work as expected and my question would be how to do this in an iterative way:
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = None
for elements in [a, b, c, d]:
e = np.array([e, elements])
print(e.shape)
# Output
# (2,) <--- This should be (4,) as in the upper example, but I don't know how to achieve that :-/
for elements in e:
print(elements.shape)
# (2,)
# (8, 9)
I understand that in each iteration I'm just combining two arrays why it always stays at shape of (2,), but I wonder how this can be done in an elegant way.
So basically I want to have a third dimension which states the count or amount of arrays that are stored. E.g. if I iterate of 1000 different 2d arrays I'd expect to have a shape of (1000,)
Hope my question is understandable - if not let me know!
Thanks a lot!
If I understood your issue correctly, you can achieve what you want in a list comprehension. This will yield the exact same solution as your code above that you described as working.
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = np.array([element for element in [a, b, c, d]])
print(e.shape)
for elements in e:
print(elements.shape)
I want to use tensordot to compute the dot product of a specific dim of two tensors. Like:
A is a tensor, whose shape is (3, 4, 5)
B is a tensor, whose shape is (3, 5)
I want to do a dot use A's third dim and B's second dim, and get a output whose dims is (3, 4)
Like below:
for i in range(3):
C[i] = dot(A[i], B[i])
How to do it by tensordot?
Well, do you want this in numpy or in Theano?
In the case, where, as you state, you would like to contract axis 3 of A against axis 2 of B, both are straightforward:
import numpy as np
a = np.arange(3 * 4 * 5).reshape(3, 4, 5).astype('float32')
b = np.arange(3 * 5).reshape(3, 5).astype('float32')
result = a.dot(b.T)
in Theano this writes as
import theano.tensor as T
A = T.ftensor3()
B = T.fmatrix()
out = A.dot(B.T)
out.eval({A: a, B: b})
however, the output then is of shape (3, 4, 3). Since you seem to want an output of shape (3, 4), the numpy alternative uses einsum, like so
einsum_out = np.einsum('ijk, ik -> ij', a, b)
However, einsum does not exist in Theano. So the specific case here can be emulated as follows
out = (a * b[:, np.newaxis]).sum(2)
which can also be written in Theano
out = (A * B.dimshuffle(0, 'x', 1)).sum(2)
out.eval({A: a, B: b})
In this specific case, einsum is probably easier to understand than tensordot. For example:
c = np.einsum('ijk,ik->ij', a, b)
I'm going to over-simplify the explanation a bit to make things more immediately understandable. We have two input arrays (separated by the comma) and this yields our output array (to the right of the ->).
a has shape 3, 4, 5 and we'll refer to it as ijk
b has shape 3, 5 (ik)
We want the output c to have shape 3, 4 (ij)
Seems a bit magical, right? Let's break that down a bit.
The letters we "lose" as we cross the -> are axes that will be summed over. That's what dot is doing, as well.
We want output with shape 3, 4, so we're eliminating k
Therefore, the output c should be ij
This means we'll refer to b as ik.
As a full example:
import numpy as np
a = np.random.random((3, 4, 5))
b = np.random.random((3, 5))
# Looping through things
c1 = []
for i in range(3):
c1.append(a[i].dot(b[i]))
c1 = np.array(c1)
# Using einsum instead
c2 = np.einsum('ijk,ik->ij', a, b)
assert np.allclose(c1, c2)
You can do this with tensordot as well. I'll add an example of that as soon as I have a bit more time. (Of course, if anyone else would like to add a tensordot example as another answer in the meantime, feel free!)
I have a two 1 dimensional arrays, a such that np.shape(a) == (n,) and b such that np.shape(b) == (m,).
I want to make a (3rd order) tensor c such that np.shape(c) == (n,n,m,)by doing c = np.outer(np.outer(a,a),b).
But when I do this, I get:
>> np.shape(c)
(n*n,m)
which is just a rectangular matrix. How can I make a 3D tensor like I want?
You could perhaps use np.multiply.outer instead of np.outer to get the required outer product:
>>> a = np.arange(4)
>>> b = np.ones(5)
>>> mo = np.multiply.outer
Then we have:
>>> mo(mo(a, a), b).shape
(4, 4, 5)
A better way could be to use np.einsum (this avoids creating intermediate arrays):
>>> c = np.einsum('i,j,k->ijk', a, a, b)
>>> c.shape
(4, 4, 5)
I have an array comprised of N 3x3 arrays (a collection of matrices, although the data type is np.ndarray) and I have an array comprised of N 3x1 arrays (a collection of vectors). What I want to do is multiply each matrix by each vector, so I expect to get back N 3x1 arrays.
Simple example:
A = np.ones((6,3,3))
B = np.ones((6,3,1))
np.dot(A,B) # This gives me a 6x3x6x1 array, which is not what I want
np.array(map(np.dot,A,B)) # This gives me exactly what I want, but I don't want to have to rely on map
I've tired all kinds of reshaping, explored einsum, etc., but can't get this to work the way I want it to. How do I get this to work with numpy broadcasting? This operation will ultimately need to be done many thousands of times, and I don't want map or list comprehension operations to slow things down.
You can use np.einsum to calculate the dot products and create the matrix of the desired shape:
np.einsum('ijk,ikl->ijl', A, B)
One could use the built-in matrix multiplication in Python 3.5 or above,
introduced in PEP 465.
$ python --version
Python 3.6.6
>>> import numpy as np
>>> A = np.ones((6,3,3))
>>> B = np.ones((6,3,1))
>>> C = A # B
>>> print(C)
[[[3.]
[3.]
[3.]]
[[3.]
[3.]
[3.]]
[[3.]
[3.]
[3.]]
[[3.]
[3.]
[3.]]
[[3.]
[3.]
[3.]]
[[3.]
[3.]
[3.]]]
A = np.random.rand(6, 3, 3)
B = np.random.rand(6, 3, 1)
C = np.array(map(np.dot, A, B))
D = np.sum(A*B.swapaxes(1, 2), axis=2)[..., None]
assert np.allclose(C, D)
assert C.shape == D.shape == (6, 3, 1)
The "allclose" is because there's some floating point rounding difference between the two methods on the order of 1e-16.
The .swapaxis and the [..., None] are just to get the arrays to conform to the shapes you specified. You could also represent it more simply with:
A = np.random.rand(6, 3, 3)
B = np.random.rand(6, 3)
C = np.array(map(np.dot, A, B))
D = np.sum(A*B[:, None, :], axis=2)
assert np.allclose(C, D)
assert C.shape == D.shape == (6, 3)