Develop Reference

Loading multiple 2d arrays with different shapes into a new array on a third dimension

I'm currently struggling with a probably rather simple question but I can't get my head around it.
Assuming I have the follow two 2d arrays with different shapes, I can combine them into a new array using:
a = np.zeros((2, 3))
b = np.zeros((4, 5))
c = np.array([a, b])
print(c.shape)
# Output
# (2,)
for elements in c:
print(elements.shape)
# Output:
# (2, 3)
# (4, 5)
So far so good!
But how would I do this if I have a large list where I'd have to iterate over? Here is a simple example with just 4 different 2d arrays:
This works as expected:
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = np.array([a, b, c, d])
print(e.shape)
# Output
# (4,)
for elements in e:
print(elements.shape)
# Output
# (2, 3)
# (4, 5)
# (6, 7)
# (8, 9)
This doesn't work as expected and my question would be how to do this in an iterative way:
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = None
for elements in [a, b, c, d]:
e = np.array([e, elements])
print(e.shape)
# Output
# (2,) <--- This should be (4,) as in the upper example, but I don't know how to achieve that :-/
for elements in e:
print(elements.shape)
# (2,)
# (8, 9)
I understand that in each iteration I'm just combining two arrays why it always stays at shape of (2,), but I wonder how this can be done in an elegant way.
So basically I want to have a third dimension which states the count or amount of arrays that are stored. E.g. if I iterate of 1000 different 2d arrays I'd expect to have a shape of (1000,)
Hope my question is understandable - if not let me know!
Thanks a lot!

If I understood your issue correctly, you can achieve what you want in a list comprehension. This will yield the exact same solution as your code above that you described as working.
a = np.zeros((2,3))
b = np.zeros((4,5))
c = np.zeros((6,7))
d = np.zeros((8,9))
e = np.array([element for element in [a, b, c, d]])
print(e.shape)
for elements in e:
print(elements.shape)

Sort numpy array in-place using order from a second array

Let two ndarrays: A of shape (n, *m), and B of shape (n, ). Is there a way to sort A in-place using the order that would sort B?
Sorting A with B is easy using np.argsort, but this is not done in-place:
A = A[np.argsort(B)]
Comments:
A and B have different dtypes, and A can have more than two dimensions. Hence they can’t be stacked to use ndarray.sort().
A takes up a lot of space, which is why it needs to be sorted in-place. Any solution requiring twice the space occupied by A would therefore defeat this purpose.
The title of this question “Re-arranging numpy array in place” may sound related, but the question itself is not very clear, and the answers do not match my question.

Here is a solution that works by following cycles in the index array. It can optionally be compiled using pythran giving a significant speedup if rows are small (80x for 10 elements) and a small speedup if rows are large (30% for 1000 elements).
To keep it pythran compatible I had to simplify it a bit, so it only accepts 2D arrays and it only sorts along axis 0.
Code:
import numpy as np
#pythran export take_inplace(float[:, :] or int[:, :], int[:])
def take_inplace(a, idx):
n, m = a.shape
been_there = np.zeros(n, bool)
keep = np.empty(m, a.dtype)
for i in range(n):
if been_there[i]:
continue
keep[:] = a[i]
been_there[i] = True
j = i
k = idx[i]
while not been_there[k]:
a[j] = a[k]
been_there[k] = True
j = k
k = idx[k]
a[j] = keep
Sample run using compiled version. As indicated above compilation is only required for small rows, for larger rows pure python should be fast enough.
>>> from timeit import timeit
>>> import numpy as np
>>> import take_inplace
>>>
>>> a = np.random.random((1000, 10))
>>> idx = a[:, 4].argsort()
>>>
>>> take_inplace.take_inplace(a, idx)
>>>
# correct
>>> np.all(np.arange(1000) == a[:, 4].argsort())
True
>>>
# speed
>>> timeit(lambda: take_inplace.take_inplace(a, idx), number=1000)
0.011950935004279017
>>>
# for comparison
>>> timeit(lambda: a[idx], number=1000)
0.02985276997787878

If you can set A beforehand as a structured array whose datatype is composed of a subarray of shape (m, ) and a scalar of the same type (e.g., np.int32), then you can sort it in-place with respect to B. For example:
import numpy as np
B = np.array([3, 1, 2])
A = np.array([[10, 11], [20, 21], [30, 31]])
(n, m) = A.shape
dt = np.dtype([('a', np.int32, (m, )), ('b', int)])
A2 = np.array([(a, b) for a, b in zip(A, B)], dtype=dt)
A2.sort(order='b')
print A2

Python, all combinations of arrays array [duplicate]

I want to use tensordot to compute the dot product of a specific dim of two tensors. Like:
A is a tensor, whose shape is (3, 4, 5)
B is a tensor, whose shape is (3, 5)
I want to do a dot use A's third dim and B's second dim, and get a output whose dims is (3, 4)
Like below:
for i in range(3):
C[i] = dot(A[i], B[i])
How to do it by tensordot?

Well, do you want this in numpy or in Theano?
In the case, where, as you state, you would like to contract axis 3 of A against axis 2 of B, both are straightforward:
import numpy as np
a = np.arange(3 * 4 * 5).reshape(3, 4, 5).astype('float32')
b = np.arange(3 * 5).reshape(3, 5).astype('float32')
result = a.dot(b.T)
in Theano this writes as
import theano.tensor as T
A = T.ftensor3()
B = T.fmatrix()
out = A.dot(B.T)
out.eval({A: a, B: b})
however, the output then is of shape (3, 4, 3). Since you seem to want an output of shape (3, 4), the numpy alternative uses einsum, like so
einsum_out = np.einsum('ijk, ik -> ij', a, b)
However, einsum does not exist in Theano. So the specific case here can be emulated as follows
out = (a * b[:, np.newaxis]).sum(2)
which can also be written in Theano
out = (A * B.dimshuffle(0, 'x', 1)).sum(2)
out.eval({A: a, B: b})

In this specific case, einsum is probably easier to understand than tensordot. For example:
c = np.einsum('ijk,ik->ij', a, b)
I'm going to over-simplify the explanation a bit to make things more immediately understandable. We have two input arrays (separated by the comma) and this yields our output array (to the right of the ->).
a has shape 3, 4, 5 and we'll refer to it as ijk
b has shape 3, 5 (ik)
We want the output c to have shape 3, 4 (ij)
Seems a bit magical, right? Let's break that down a bit.
The letters we "lose" as we cross the -> are axes that will be summed over. That's what dot is doing, as well.
We want output with shape 3, 4, so we're eliminating k
Therefore, the output c should be ij
This means we'll refer to b as ik.
As a full example:
import numpy as np
a = np.random.random((3, 4, 5))
b = np.random.random((3, 5))
# Looping through things
c1 = []
for i in range(3):
c1.append(a[i].dot(b[i]))
c1 = np.array(c1)
# Using einsum instead
c2 = np.einsum('ijk,ik->ij', a, b)
assert np.allclose(c1, c2)
You can do this with tensordot as well. I'll add an example of that as soon as I have a bit more time. (Of course, if anyone else would like to add a tensordot example as another answer in the meantime, feel free!)

Creating 3rd order tensors with python and numpy

I have a two 1 dimensional arrays, a such that np.shape(a) == (n,) and b such that np.shape(b) == (m,).
I want to make a (3rd order) tensor c such that np.shape(c) == (n,n,m,)by doing c = np.outer(np.outer(a,a),b).
But when I do this, I get:
>> np.shape(c)
(n*n,m)
which is just a rectangular matrix. How can I make a 3D tensor like I want?

You could perhaps use np.multiply.outer instead of np.outer to get the required outer product:
>>> a = np.arange(4)
>>> b = np.ones(5)
>>> mo = np.multiply.outer
Then we have:
>>> mo(mo(a, a), b).shape
(4, 4, 5)
A better way could be to use np.einsum (this avoids creating intermediate arrays):
>>> c = np.einsum('i,j,k->ijk', a, a, b)
>>> c.shape
(4, 4, 5)

Complicated numpy array multiplications

I have two arrays:
a = [[a11,a12],
[a21,a22]]
b = [[b11,b12],
[b21,b22]]
What I would like to do is build up a matrix as follows:
xx = np.mean(a[:,0]*b[:,0])
xy = np.mean(a[:,0]*b[:,1])
yx = np.mean(a[:,1]*b[:,0])
yy = np.mean(a[:,1]*b[:,1])
and return an array c such that
c = [[xx,xy],
yx,yy]]
Is there a nice pythonic way to do this in numpy? Because at the moment I have done it by hand, exactly as above, so the dimensions of the output array are coded in by hand, rather than determined as according to the size of the input arrays a and b.

Is there an error in your third element? If, as seems reasonable, you want yx = np.mean(a[:,1]*b[:,0]) instead of yx = np.mean(b[:,1]*a[:,0]), then you can try the following:
a = np.random.rand(2, 2)
b = np.random.rand(2, 2)
>>> c
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])
>>> np.mean(a.T[:, None, :]*b.T, axis=-1)
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])
It will actually be faster to avoid the intermediate array and express your result as a matrix multiplication:
>>> np.dot(a.T, b) / a.shape[0]
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])