I have a matrix A with shape=(N, N) and a matrix B with the same shape=(N, N).
I am constructing a matrix M using the following einsum (using the opt_einsum library):
M = oe.contract('nm,in,jm,pn,qm->ijpq', A, B, B, B, B)
This is evaluating the following sum:
This yeilds matrix M with shape (N, N, N, N). I then reshape this to a 2D array of shape (N**2, N**2)
M = M.reshape((N**2, N**2))
This must be 2D as it is treated as a linear operator.
I want to use the sparse library, as M is sparse, and becomes too large to store for large N. I can make A and B sparse, and insert them into the oe.contract.
The problem is, sparse only supports 2D arrays and so fails to produce the 4D output of shape (N, N, N. N). Is there a way to combine the einsum and reshape steps to allow sparse to be used in this way, as the final shape of M is 2D?
This may not help with your use of opt_einsum, but with a bit of reorganizing I can speed up np.einsum quite a bit, at least for small arrays.
Do a partial product of two B:
c1 = np.einsum('in,jm->ijnm',B,B).reshape(N*N,N,N)
The pq pair is the same, so we don't need to recalculate it:
c2 = np.einsum('nm,onm,pnm->op',A,c1,c1)
I verified that this works for two (3,3) arrays, and the speed up is about 10x.
We can even reshape the nm to 1d, though this doesn't improve speed:
c1 = np.einsum('in,jm->ijnm',B,B).reshape(N*N,N*N)
c3 = np.einsum('n,on,pn->op',A.reshape(N*N),c1,c1)
I did not correctly interpret the error given by opt_einsum.
The problem is not that sparse does not support ND sparse arrays (it does!), but that I was not using a true einsum, as the indices summed over appear more than twice (n and m). As stated in the opt_einsum documentation this will result in the use of the sparse.einsum function, of which none exists. Using only 1 or 2 of each index works. Using a differerent path, one suggested for example by hpaulj can be used to solve the problem.
Related
I need to do the following two operations:
solve Ax=b by inverting the n-by-n matrix A, and
solve r=Ar using power iteration (i.e. by repeated multiplying current vector r by A) such as one would do for the PageRank algorithm.
My question is: When computing the matrix-vector product A^{-1}b or the matrix-vector product Ar, is it better to use numpy.dot or numpy.matmul? (I understand there might be differences in higher dimensions, but my question is only for the case where A is a 2D array and b, r are vectors.)
From the numpy doc for np.dot:
Dot product of two arrays. Specifically, If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a # b is
preferred.
So basically for your case, it does not matter, although matmul is preferred according to the doc.
Also since one of your arrays is 1-D, from docs for np.matmul:
If the second argument is 1-D, it is promoted to a matrix by appending
a 1 to its dimensions. After matrix multiplication the appended 1 is
removed.
And:
matmul differs from dot in two important ways:
Multiplication by scalars is not allowed, use * instead. Stacks of matrices are
broadcast together as if the matrices were elements, respecting the
signature
Therefore, they would work the same in your case, but I would go with numpy doc's recommendation on using matmul.
I have a matrix A with size (5,7,3) and a matrix B with size (5,3,8). I want to multiply them C = A.B, and the size of C is (5,7,8).
It means that one 2D submatrix with size (7,3) in matrix A will be multiplied with one 2D submatrix with size (3,8) in matrix B respectively. So I have to multiply 5 times.
The simplest way is using a loop and numpy:
for u in range(5):
C[u] = numpy.dot(A[u],B[u])
Is there any way to do this without using a loop?
Is there any equivalent method in Theano to do this without using scan?
Can be done pretty simply with np.einsum in numpy.
C = numpy.einsum('ijk,ikl->ijl', A, B)
It can also simply be:
C = numpy.matmul(A,B)
Since the docs state:
If either argument is N-D, N > 2, it is treated as a stack of matrices residing in the last two indexes and broadcast accordingly
Theano has similar functionaly of batched_dot so it would be
C = theano.tensor.batched_dot(A, B)
I have a linear system in which all the matrices are block diagonal. They have N blocks identical in shape.
Matrices are stored in compressed format as numpy arrays with shape (N, n, m), while the shape of the vectors is (N, m).
I currently implemented the matrix-vector product as
import numpy as np
def mvdot(m, v):
return (m * np.expand_dims(v, -2)).sum(-1)
Thanks to broadcasting rules, if all the blocks of a matrix are the same I have to store it only once in an array with shape (1, n, m): the product with a vector (N, m) still gives the correct (N, n) vector.
My questions are:
how to implement an efficient matrix-matrix product that yields the matrix with shape (N, n, m) from two matrices with shapes (N, n, p) and (N, p, m)?
is there a way to perform these operations with a numpy built-in (possibly faster) function? Functions like np.linalg.inv make me think that numpy was designed to support this compressed format for block diagonal matrices.
If I understand your question correctly, you have two arrays of shape (N,n,p) and (N,p,m), respectively, and their product should be of shape (N,n,m) where element [i,:,:] is the matrix product of M1[i,:,:] and M2[i,:,:]. This can be achieved using numpy.einsum:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = np.einsum('ijk,ikl->ijl',M1,M2)
# check if all the submatrices are what we expect
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
# True
Numpy's einsum is an incredibly versatile construction for complicated linear operations, and it's usually pretty efficient with two operands. The idea is to rewrite your operation in an indexed way: what you need is to multiply M1[i,j,k] with M2[i,k,l] for each i,j,l, and sum over k. This is exactly what the above call to einsum does: it collapses the index k, and performs the necessary products and assignments along the remaining dimensions in the given order.
The matrix-vector product can be done similarly:
M = np.random.rand(N,n,m)
v = np.random.rand(N,m)
Mvprod = np.einsum('ijk,ik->ij',M,v)
It's possible that numpy.dot can be coerced with the proper transposes and dimension tricks to directly do what you want, but I couldn't make that work.
Both of the above operations can be done in the same function call by allowing an implicit number of dimensions within einsum:
def mvdot(M1,M2):
return np.einsum('ijk,ik...->ij...',M1,M2)
Mprod = mvdot(M1,M2)
Mvprod = mvdot(M,v)
In case the input argument M2 is a block matrix, there will be a leading dimension appended to the result, creating a block matrix. In case M2 is a "block vector", the result will be a block vector.
Since Python 3.5 and above, the previous example can be simplified using the matrix multiplication operator # (numpy.matmul) which treats this case as a stack of matrices residing in the last two indexes and broadcast accordingly:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = M1 # M2 # similar to np.matmul(M1, M2)
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
#True
In python I have 2 three dimensional arrays:
T with size (n,n,n)
U with size (k,n,n)
T and U can be seen as many 2-D arrays one next to the other. I need to multiply all those matrices, ie I have to perform the following operation:
for i in range(n):
H[:,:,i] = U[:,:,i].dot(T[:,:,i]).dot(U[:,:,i].T)
As n might be very big I am wondering if this operation could be in some way speed up with numpy.
Carefully looking into the iterators and how they are involved in those dot product reductions, we could translate all of those into one np.einsum implementation like so -
H = np.einsum('ijk,jlk,mlk->imk',U,T,U)
in a current project I have a large multidimensional array of shape (I,J,K,N) and a square matrix of dim N.
I need to perform a matrix vector multiplication of the last axis of the array with the square matrix.
So the obvious solution would be:
for i in range(I):
for j in range(J):
for k in range(K):
arr[i,j,k] = mat.dot(arr[i,j,k])
but of course this is rather slow. So I also tried numpy's tensordot but had little success.
I would expect that something like:
arr = tensordot(mat,arr,axes=((0,1),(3)))
should do the trick but I get a shape mismatch error.
Has someone a better solution or knows how to correctly use tensordot?
Thank you!
This should do what your loops, but with vectorized looping:
from numpy.core.umath_tests import matrix_multiply
arr[..., np.newaxis] = matrix_multiply(mat, arr[..., np.newaxis])
matrix_multiply and its sister inner1d are hidden, undocumented, gems of numpy, although a full set of linear algebra gufuncs should see the light with numpy 1.8. matrix_multiply does matrix multiplication on the last two dimensions of its inputs, and broadcasting on the rest. The only tricky part is setting an additional dimension, so that it sees column vectors when multiplying, and adding it also on assignment back into array, so that there is no shape mismatch.
I think your for loop is wrong, and for this case dot seems to be enough:
# a is your IJKN
# b is your NN
c = dot(a, b)
Here c will be a IJKN array. If you want to sum over the last dimension to get the IJK array:
arr = dot(a,b).sum(axis=3)
BUT I'm NOT SURE IF THIS IS WHAT YOU WANT...