I'm trying to understand why numpy's dot function behaves as it does:
M = np.ones((9, 9))
V1 = np.ones((9,))
V2 = np.ones((9, 5))
V3 = np.ones((2, 9, 5))
V4 = np.ones((3, 2, 9, 5))
Now np.dot(M, V1) and np.dot(M, V2) behave as
expected. But for V3 and V4 the result surprises
me:
>>> np.dot(M, V3).shape
(9, 2, 5)
>>> np.dot(M, V4).shape
(9, 3, 2, 5)
I expected (2, 9, 5) and (3, 2, 9, 5) respectively. On the other hand, np.matmul
does what I expect: the matrix multiply is broadcast
over the first N - 2 dimensions of the second argument and
the result has the same shape:
>>> np.matmul(M, V3).shape
(2, 9, 5)
>>> np.matmul(M, V4).shape
(3, 2, 9, 5)
So my question is this: what is the rationale for
np.dot behaving as it does? Does it serve some particular purpose,
or is it the result of applying some general rule?
From the docs for np.dot:
For 2-D arrays it is equivalent to matrix multiplication, and for 1-D arrays to inner product of vectors (without complex conjugation). For N dimensions it is a sum product over the last axis of a and the second-to-last of b:
dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])
For np.dot(M, V3),
(9, 9), (2, 9, 5) --> (9, 2, 5)
For np.dot(M, V4),
(9, 9), (3, 2, 9, 5) --> (9, 3, 2, 5)
The strike-through represents dimensions that are summed over, and are therefore not present in the result.
In contrast, np.matmul treats N-dimensional arrays as 'stacks' of 2D matrices:
The behavior depends on the arguments in the following way.
If both arguments are 2-D they are multiplied like conventional matrices.
If either argument is N-D, N > 2, it is treated as a stack of matrices residing in the last two indexes and broadcast accordingly.
The same reductions are performed in both cases, but the order of the axes is different. np.matmul essentially does the equivalent of:
for ii in range(V3.shape[0]):
out1[ii, :, :] = np.dot(M[:, :], V3[ii, :, :])
and
for ii in range(V4.shape[0]):
for jj in range(V4.shape[1]):
out2[ii, jj, :, :] = np.dot(M[:, :], V4[ii, jj, :, :])
From the documentation of numpy.matmul:
matmul differs from dot in two important ways.
Multiplication by scalars is not allowed.
Stacks of matrices are broadcast together as if the matrices were elements.
In conclusion, this is the standard matrix-matrix multiplication you would expect.
On the other hand, numpy.dot is only equivalent to the matrix-matrix multiplication for two-dimensional arrays. For larger dimensions, ...
it is a sum product over the last axis of a and the second-to-last of b:
dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])
[source: documentation of numpy.dot]
This resembles the inner (dot) product. In case of vectors, numpy.dot returns the dot product. Arrays are considered collections of vectors, and the dot product of them is returned.
For the why :
dot and matmult are both generalizations of 2D*2D matrix multiplication. But they are a lot of possible choices, according to mathematics properties, broadcasting rules, ...
The choices are for dot and matmul are very different:
For dot, some dimensions (green here) are dedicated to the first array ,
others (blue) for the second.
matmul need an alignement of stacks regarding to broadcasting rules.
Numpy is born in an image analysis context, and dot can manage easily some tasks by a out=dot(image(s),transformation(s)) way. (see the dot docs in early version of numpy book, p92).
As an illustration :
from pylab import *
image=imread('stackoverflow.png')
identity=eye(3)
NB=ones((3,3))/3
swap_rg=identity[[1,0,2]]
randoms=[rand(3,3) for _ in range(6)]
transformations=[identity,NB,swap_rg]+randoms
out=dot(image,transformations)
for k in range(9):
subplot(3,3,k+1)
imshow (out[...,k,:])
The modern matmul can do the same thing as the old dot, but the stack of matrix must be take in account. (matmul(image,transformations[:,None]) here).
No doubt that it is better in other contexts.
The equivalent einsum expressions are:
In [92]: np.einsum('ij,kjm->kim',M,V3).shape
Out[92]: (2, 9, 5)
In [93]: np.einsum('ij,lkjm->lkim',M,V4).shape
Out[93]: (3, 2, 9, 5)
Expressed this way, the dot equivalent, 'ij,lkjm->ilkm', looks just as natural as the 'matmul' equivalent, 'ij,lkjm->lkim'.
Related
For example, I have a matrix with shape:
x = np.random.rand(3, 10, 2, 6)
As you can see, there are only two arrays along an axis=2.
I have a function that accepts these two arrays:
def f(arr1, arr2): # arr1 with shape (6, ) and arr2 with (6, )
return np.sum(arr1, arr2) # for simplicity
How can I apply this function along the second axis to x array in a vectorized way? Such that resulting array will be of shape [3, 10, dim of output].
I came across apply_along_axis routine, but it requires that f accepts only 1D slice.
You can't do it entirely arbitrarily, but your particular case reduces to
x.sum(axis=2)
If you want to add the arrays as in your code:
x[:, :, 0, :] + x[:, :, 1, :]
Let's suppose I have these two variables
matrices = np.random.rand(4,3,3)
vectors = np.random.rand(4,3,1)
What I would like to perform is the following:
dot_products = [matrix # vector for (matrix,vector) in zip(matrices,vectors)]
Therefore, I've tried using the np.tensordot method, which at first seemed to make sense, but this happened when testing
>>> np.tensordot(matrices,vectors,axes=([-2,-1],[-2,-1]))
...
ValueError: shape-mismatch for sum
>>> np.tensordot(matrices,vectors,axes=([-2,-1]))
...
ValueError: shape-mismatch for sum
Is it possible to achieve these multiple dot products with the mentioned Numpy method? If not, is there another way that I can accomplish this using Numpy?
The documentation for # is found at np.matmul. It is specifically designed for this kind of 'batch' processing:
In [76]: matrices = np.random.rand(4,3,3)
...: vectors = np.random.rand(4,3,1)
In [77]: dot_products = [matrix # vector for (matrix,vector) in zip(matrices,vectors)]
In [79]: np.array(dot_products).shape
Out[79]: (4, 3, 1)
In [80]: (matrices # vectors).shape
Out[80]: (4, 3, 1)
In [81]: np.allclose(np.array(dot_products), matrices#vectors)
Out[81]: True
A couple of problems with tensordot. The axes parameter specify which dimensions are summed, "dotted", In your case it would be the last of matrices and 2nd to the last of vectors. That's the standard dot paring.
In [82]: np.dot(matrices, vectors).shape
Out[82]: (4, 3, 4, 1)
In [84]: np.tensordot(matrices, vectors, (-1,-2)).shape
Out[84]: (4, 3, 4, 1)
You tried to specify 2 pairs of axes for summing. Also dot/tensordot does a kind of outer product on the other dimensions. You'd have to take the "diagonal" on the 4's. tensordot is not what you want for this operation.
We can be more explicit about the dimensions with einsum:
In [83]: np.einsum('ijk,ikl->ijl',matrices, vectors).shape
Out[83]: (4, 3, 1)
I have two matrices:
>>> a.shape
(100, 3, 1)
>>> b.shape
(100, 3, 3)
I'd like to perform a dot product such that my end result is (100, 3, 1). However, currently I receive:
>>> c = np.dot(b, a)
>>> c.shape
(100, 3, 100, 1)
Could somebody explain what is happening? I'm reading the docs and can't figure it out.
Edit:
So per the docs (overlooked it):
If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a # b is preferred.
And that gives the desired result, but I'm still curious, what is taking place here? What rule of the np.dot function is being applied to yield a result of (100, 3, 100, 1)?
This is how dot works in your case:
dot(b, a)[i,j,k,m] = sum(b[i,j,:] * a[k,:,m])
Your output shape is exactly how the docs specify it:
(b.shape[0], b.shape[1], a.shape[0], a.shape[2])
If it's not what you expected, you might be looking for another matrix multiplication.
dot will return all the possible products of the matrices stored in the last two dimensions of your arrays. Use matmul aka the # operator to broadcast the leading dimensions instead of combining them:
np.matmul(b, a)
Or
b # a
The Swiss Army knife of sum-products is einsum, so you can use that too:
np.einsum('aij,ajk->aik', b, a)
Or
np.einsum('ajk,aij->aik', a, b)
I'm trying to take array
a = [1,5,4,5,7,8,9,8,4,13,43,42]
and array
b = [3,5,6,2,7]
And I want b to be the indexes in a, e.g. a new array that is
[a[b[0]], a[b[1]], a[b[2]], a[b[3]] ...]
So the values in b are indexes into a.
And there are 500k entries in a and 500k in b (approximately).
Is there a fast way to kick in all cores in numpy to do this?
I already do it just fine in for loops and it is sloooooooowwwwww.
Edit to clarify. The solution has to work for 2D and 3D arrays.
so maybe
b = [(2,3), (5,4), (1,2), (1,0)]
and we want
c = [a[b[0], a[b[1], ...]
Not saying it is fast, but the numpy way would simply be:
a[b]
outputs:
array([5, 8, 9, 4, 8])
This can be done in NumPy using advanced indexing. As Christian's answer pointed out, in the 1-D case, you would simply write:
a[b]
and that is equivalent to:
[a[b[x]] for x in range(b.shape[0])]
In higher-dimensional cases, however, you need to have separate lists for each dimension of the indices. Which means, you can't do:
a = np.random.randn(7, 8, 9) # 3D array
b = [(2, 3, 0), (5, 4, 1), (1, 2, 2), (1, 0, 3)]
print(a[b]) # this is incorrect
but you can do:
b0, b1, b2 = zip(*b)
print(a[b0, b1, b2])
you can also use np.take:
print(np.take(a, b))
I solved this by writing a C extension to numpy called Tensor Weighted Interpolative Transfer, in order to get speed and multi-threading. In pure python it is 3 seconds per 200x100x3 image scale and fade across, and in multi-threaded C with 8 cores is 0.5 milliseconds for the same operation.
The core C code ended up being like
t2[dstidxs2[i2] + doff1] += t1[srcidxs2[i2] + soff1] * w1 * ws2[i2];
Where the doff1 is the offset in the destination array etc. The w1 and ws2 are the interpolated weights.
All the code is ultra optimized in C for speed. (not code size or maintainability)
All code is available on https://github.com/RMKeene/twit and on PyPI.
I expect furthur optimization in the future such as special cases if all weights are 1.0.
Suppose I have 3 numpy arrays a, b, c, of the same shape, say
a.shape == b.shape == c.shape == (7,9)
Now I'd like to create a 3-dimensional array of size (7,9,3), say x, such that
x[:,:,0] == a
x[:,:,1] == b
x[:,:,2] == c
What is the "pythonic" way of doing it (perhaps in one line)?
Thanks in advance!
There's a function that does exactly that: numpy.dstack ("d" for "depth"). For example:
In [10]: import numpy as np
In [11]: a = np.ones((7, 9))
In [12]: b = a * 2
In [13]: c = a * 3
In [15]: x = np.dstack((a, b, c))
In [16]: x.shape
Out[16]: (7, 9, 3)
In [17]: (x[:, :, 0] == a).all()
Out[17]: True
In [18]: (x[:, :, 1] == b).all()
Out[18]: True
In [19]: (x[:, :, 2] == c).all()
Out[19]: True
TL;DR:
Use numpy.stack (docs), which joins a sequence of arrays along a new axis of your choice.
Although #NPE answer is very good and cover many cases, there are some scenarios in which numpy.dstack isn't the right choice (I've just found that out while trying to use it). That's because numpy.dstack, according to the docs:
Stacks arrays in sequence depth wise (along third axis).
This is equivalent to concatenation along the third axis after 2-D
arrays of shape (M,N) have been reshaped to (M,N,1) and 1-D arrays of
shape (N,) have been reshaped to (1,N,1).
Let's walk through an example in which this function isn't desirable. Suppose you have a list with 512 numpy arrays of shape (3, 3, 3) and want to stack them in order to get a new array of shape (3, 3, 3, 512). In my case, those 512 arrays were filters of a 2D-convolutional layer. If you use numpy.dstack:
>>> len(arrays_list)
512
>>> arrays_list[0].shape
(3, 3, 3)
>>> numpy.dstack(arrays_list).shape
(3, 3, 1536)
That's because numpy.dstack always stacks the arrays along the third axis! Alternatively, you should use numpy.stack (docs), which joins a sequence of arrays along a new axis of your choice:
>>> numpy.stack(arrays_list, axis=-1).shape
(3, 3, 3, 512)
In my case, I passed -1 to the axis parameter because I wanted the arrays stacked along the last axis.