I am trying to find out a way to do a matrix power for a sparse matrix M: M^k = M*...*M k times where * is the matrix multiplication (numpy.dot), and not element-wise multiplication.
I know how to do it for a normal matrix:
import numpy as np
import scipy as sp
N=100
k=3
M=(sp.sparse.spdiags(np.ones(N), 0, N, N)-sp.sparse.spdiags(np.ones(N), 2, N, N)).toarray()
np.matrix_power(M,k)
How can I do it for sparse M:
M=(sp.sparse.spdiags(np.ones(N), 0, N, N)-sp.sparse.spdiags(np.ones(N), 2, N, N))
Of course, I can do this by recursive multiplications, but I am wondering if there is a functionality like matrix_power for sparse matrices in scipy.
Any help is much much appreciated. Thanks in advance.
** has been implemented for csr_matrix. There is a __pow__ method.
After handling some special cases this __pow__ does:
tmp = self.__pow__(other//2)
if (other % 2):
return self * tmp * tmp
else:
return tmp * tmp
For sparse matrix, * is the matrix product (dot for ndarray). So it is doing recursive multiplications.
As math noted, np.matrix also implements ** (__pow__) as matrix power. In fact it ends up calling np.linalg.matrix_power.
np.linalg.matrix_power(M, n) is written in Python, so you can easily see what it does.
For n<=3 is just does the repeated dot.
For larger n, it does a binary decomposition to reduce the total number of dots. I assume that means for n=4:
result = np.dot(M,M)
result = np.dot(result,result)
The sparse version isn't as general. It can only handle positive integer powers.
You can't count on numpy functions operating on spare matrices. The ones that do work are the ones that pass the action on to the array's own method. e.g. np.sum(A) calls A.sum().
You can also use ** notation instead of matrix_power for numpy matrix :
a=np.matrix([[1,2],[2,1]])
a**3
Out :
matrix([[13, 14],
[14, 13]])
try it with scipy sparse matrix.
Related
Let w, x, y, z be torch tensors of shape (m, n) and we wish to compute the following unbiased estimator row-wise efficiently (without for loops), where I want to compute for every row 1, ..., m:
In case of only the unbiased estimator of the square of means, i.e., for :
this is possible, e.g., using torch.einsum:
batch_outer = torch.einsum('bi, bj -> bij', x, y)
zero_diag = 1-torch.eye(batch_outer.shape[1])
return (batch_outer * zero_diag).sum(dim=2).sum(dim=1) / (n * (n-1))
However, for the case to the power of four this is not so easy doable, mostly because these are not squared tensors and in particular, because the zeroing out of the diagonals becomes very tedious.
My questions:
1.) How can this be implemented efficiently ommitting any for loops?
2.) Which time and memory complexity would that solution have in big O notation?
3.) Can this solution also be used to do it with four 3D tensors of shape (m, k, n), where again we only want to do the computations along the axes of length n (dim=2)?
4.) If I want to do it in log-space for numerical stability, i.e., to use logsumexp for summations and sums for multiplications (because log(xy)= log(x)+log(y)), any solution with einsum wouldnt work anymore. How could that computation then be done in log space?
1 This implementation seems to work if I didn't make mess with the diagonal dimensions.
import numpy as np
import torch as th
x = np.array([1,4,5,3])
y = np.array([5,2,4,5])[np.newaxis]
z = np.array([5,7,4,5])[np.newaxis][np.newaxis]
w = np.array([3,9,5,1])[np.newaxis][np.newaxis][np.newaxis]
xth = th.Tensor(x)
yth = th.Tensor(y)
zth = th.Tensor(z)
wth = th.Tensor(w)
tensor = xth*th.transpose(yth, 0, 1)*th.transpose(zth,0,2)*th.transpose(wth,0,3)
diag = th.diagonal(tensor, dim1 = -2, dim2 = -1)
result = th.sum(tensor) - th.sum(diag)
result /= np.math.factorial(len(x))
print(result)
The order is between O(n^2.37..) - O(n^3), depending on the pytorch implementation of the matrix multiplication.
I don't see why not, just choose properly the dimensions to transpose and take the diagonal.
I don't see why would this solution won't work in a log-space.
pd: my knowledge in pytorch is quite limited, but I'm sure you can define x,y,z,w in a more elegant way.
I am trying to implement the QR decomposition via householder reflectors. While attempting this on a very simple array, I am getting weird numbers. Anyone who can tell me, also, why using the # vs * operator between vec and vec.T on the last line of the function definition gets major bonus points.
This has stumped two math/comp sci phds as of this morning.
import numpy as np
def householder(vec):
vec[0] += np.sign(vec[0])*np.linalg.norm(vec)
vec = vec/vec[0]
gamma = 2/(np.linalg.norm(vec)**2)
return np.identity(len(vec)) - gamma*(vec*vec.T)
array = np.array([1, 3 ,4])
Q = householder(array)
print(Q#array)
Output:
array([-4.06557377, -7.06557377, -6.06557377])
Where it should be:
array([5.09, 0, 0])
* is elementwise multiplication, # is matrix multiplication. Both have their uses, but for matrix calculations you most likely want the matrix product.
vec.T for an array returns the same array. A simple array only has one dimension, there is nothing to transpose. vec*vec.T just returns the elementwise squared array.
You might want to use vec=vec.reshape(-1,1) to get a proper column vector, a one-column matrix. Then vec*vec.T does "by accident" the correct thing. You might want to put the matrix multiplication operator there anyway.
For example, I have an equation for projection matrix which works for 1 dimensional vectors:
where P is projection matrix and T is transpose.
We know that we can't simplify this fraction more (by cancelling terms) since denominator is a dot product (thus 0 dimensional scalar, number) and numerator is a matrix (column multiplied by row is a matrix).
I'm not sure how could I define function for this equation in numpy, considering that the current function that I'm using does not differentiate between these terms, multiplication is treated as it has commutative property. I'm using numpy.multiply method:
>>> import numpy as np
>>> a = np.array(a)
>>> a*a.T
array([1, 4, 9])
>>> a.T*a
array([1, 4, 9])
As you see, both of them output vectors.
I've also tried using numpy.matmul method:
>>> np.matmul(a, a.T)
14
>>> np.matmul(a.T, a)
14
which gives dot product for both of the function calls.
I also did try numpy.dot but it obviously doesn't work for numerator terms.
From my understanding, the first function call should output matrix (since column is multiplied by row) and the second function call should output a scalar in a proper case.
Am I mistaken? Is there any method that differentiates between a multiplied by a transpose and a transpose multiplied by a?
Thank you!
Note that 1-dimensional numpy arrays are not column vectors (and operations such as transposition do not make sense). If you want to obtain a column vector you should define your array as a 2-dimensional array (with the second dimension size equal to 1).
However, you don't need to define a column vector, as numpy offers functions to do what you want by manipulating an 1D array as follows
P = np.outer(a,a)/np.inner(a,a)
Stelios' answer is the best, no doubt but for completeness you can use the # operator with 2-d arrays:
a = np.array([1,4,9])[np.newaxis]
P = (a.T # a) / (a # a.T)
I have a linear system in which all the matrices are block diagonal. They have N blocks identical in shape.
Matrices are stored in compressed format as numpy arrays with shape (N, n, m), while the shape of the vectors is (N, m).
I currently implemented the matrix-vector product as
import numpy as np
def mvdot(m, v):
return (m * np.expand_dims(v, -2)).sum(-1)
Thanks to broadcasting rules, if all the blocks of a matrix are the same I have to store it only once in an array with shape (1, n, m): the product with a vector (N, m) still gives the correct (N, n) vector.
My questions are:
how to implement an efficient matrix-matrix product that yields the matrix with shape (N, n, m) from two matrices with shapes (N, n, p) and (N, p, m)?
is there a way to perform these operations with a numpy built-in (possibly faster) function? Functions like np.linalg.inv make me think that numpy was designed to support this compressed format for block diagonal matrices.
If I understand your question correctly, you have two arrays of shape (N,n,p) and (N,p,m), respectively, and their product should be of shape (N,n,m) where element [i,:,:] is the matrix product of M1[i,:,:] and M2[i,:,:]. This can be achieved using numpy.einsum:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = np.einsum('ijk,ikl->ijl',M1,M2)
# check if all the submatrices are what we expect
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
# True
Numpy's einsum is an incredibly versatile construction for complicated linear operations, and it's usually pretty efficient with two operands. The idea is to rewrite your operation in an indexed way: what you need is to multiply M1[i,j,k] with M2[i,k,l] for each i,j,l, and sum over k. This is exactly what the above call to einsum does: it collapses the index k, and performs the necessary products and assignments along the remaining dimensions in the given order.
The matrix-vector product can be done similarly:
M = np.random.rand(N,n,m)
v = np.random.rand(N,m)
Mvprod = np.einsum('ijk,ik->ij',M,v)
It's possible that numpy.dot can be coerced with the proper transposes and dimension tricks to directly do what you want, but I couldn't make that work.
Both of the above operations can be done in the same function call by allowing an implicit number of dimensions within einsum:
def mvdot(M1,M2):
return np.einsum('ijk,ik...->ij...',M1,M2)
Mprod = mvdot(M1,M2)
Mvprod = mvdot(M,v)
In case the input argument M2 is a block matrix, there will be a leading dimension appended to the result, creating a block matrix. In case M2 is a "block vector", the result will be a block vector.
Since Python 3.5 and above, the previous example can be simplified using the matrix multiplication operator # (numpy.matmul) which treats this case as a stack of matrices residing in the last two indexes and broadcast accordingly:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = M1 # M2 # similar to np.matmul(M1, M2)
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
#True
Assume that I have an affinity matrix A and a diagonal matrix D. How can I compute the Laplacian matrix in Python with nympy?
L = D^(-1/2) A D^(1/2)
Currently, I use L = D**(-1/2) * A * D**(1/2). Is this a right way?
Thank you.
Please note that it is recommended to use numpy's array instead of matrix: see this paragraph in the user guide. The confusion in some of the responses is an example of what can go wrong... In particular, D**0.5 and the products are elementwise if applied to numpy arrays, which would give you a wrong answer. For example:
import numpy as np
from numpy import dot, diag
D = diag([1., 2., 3.])
print D**(-0.5)
[[ 1. Inf Inf]
[ Inf 0.70710678 Inf]
[ Inf Inf 0.57735027]]
In your case, the matrix is diagonal, and so the square root of the matrix is just another diagonal matrix with the square root of the diagonal elements. Using numpy arrays, the equation becomes
D = np.array([1., 2., 3.]) # note that we define D just by its diagonal elements
A = np.cov(np.random.randn(3,100)) # a random symmetric positive definite matrix
L = dot(diag(D**(-0.5)), dot(A, diag(D**0.5)))
Numpy allows you to exponentiate a diagonal "matrix" with positive elements and a positive exponent directly:
m = diag(range(1, 11))
print m**0.5
The result is what you expect in this case because NumPy actually applies the exponentiation to each element of the NumPy array individually.
However, it indeed does not allow you to exponentiate any NumPy matrix directly:
m = matrix([[1, 1], [1, 2]])
print m**0.5
produces the TypeError that you have observed (the exception says that the exponent must be an integer–even for matrices that can be diagonalized with positive coefficients).
So, as long as your matrix D is diagonal and your exponent is positive, you should be able to directly use your formula.
Well, the only problem I see is that if you are using Python 2.6.x (without from __future__ import division), then 1/2 will be interpreted as 0 because it will be considered integer division. You can get around this by using D**(-.5) * A * D**.5 instead. You can also force float division with 1./2 instead of 1/2.
Other than that, it looks correct to me.
Edit:
I was trying to exponentiate a numpy array, not a matrix before, which works with D**.5. You can exponentiate a matrix element-wise using numpy.power. So you would just use
from numpy import power
power(D, -.5) * A * power(D, .5)
Does numpy have square root function for matrixes? Then you could do sqrt(D) instead of (D**(1/2))
Maybe the formula should realy be written
L = (D**(-1/2)) * A * (D**(1/2))
Based on previous comment this formula should work in case of D being diagonal matrix (I have not chance to prove it now).