Develop Reference

Product and summation with 3d and 1d arrays

Given a 3d array X with dimensions (K,n,m) that can be considered as a stack of K (n,m) matrices and a 1d vector b (dim n), the goal is to obtain the resulting vector r (dim n) each component of which is calculated as:
It is easy to see that the expression under the k-summation (i.e. two internal sums) is just a dot product X_k b X_k (and, therefore, can easily be calculated using numpy). So, the desired vector r is
where X_k is the k-th 2d (n,m) 'layer' of 3d array X.
I.e. the current solution is
r = 0
for k in range(K):
r += x[k,:,:] # (b # x[k, :, :])
Can r be efficiently calculated avoiding a for-loop by k?
Or maybe there is another efficient way to calculate r?
(I tried np.tensordot but since it is just pure summation by k I didn't get a correct result yet.)

This looks like a perfect usecase for einsum:
r = np.einsum('kij,l,klj->i', x, b, x)
which will vectorize the operation, e.g. it's more optimal than a for loop.

Pytorch: Efficiently compute unbiased estimator of mean to the power of four

Let w, x, y, z be torch tensors of shape (m, n) and we wish to compute the following unbiased estimator row-wise efficiently (without for loops), where I want to compute for every row 1, ..., m:
In case of only the unbiased estimator of the square of means, i.e., for :
this is possible, e.g., using torch.einsum:
batch_outer = torch.einsum('bi, bj -> bij', x, y)
zero_diag = 1-torch.eye(batch_outer.shape[1])
return (batch_outer * zero_diag).sum(dim=2).sum(dim=1) / (n * (n-1))
However, for the case to the power of four this is not so easy doable, mostly because these are not squared tensors and in particular, because the zeroing out of the diagonals becomes very tedious.
My questions:
1.) How can this be implemented efficiently ommitting any for loops?
2.) Which time and memory complexity would that solution have in big O notation?
3.) Can this solution also be used to do it with four 3D tensors of shape (m, k, n), where again we only want to do the computations along the axes of length n (dim=2)?
4.) If I want to do it in log-space for numerical stability, i.e., to use logsumexp for summations and sums for multiplications (because log(xy)= log(x)+log(y)), any solution with einsum wouldnt work anymore. How could that computation then be done in log space?

1 This implementation seems to work if I didn't make mess with the diagonal dimensions.
import numpy as np
import torch as th
x = np.array([1,4,5,3])
y = np.array([5,2,4,5])[np.newaxis]
z = np.array([5,7,4,5])[np.newaxis][np.newaxis]
w = np.array([3,9,5,1])[np.newaxis][np.newaxis][np.newaxis]
xth = th.Tensor(x)
yth = th.Tensor(y)
zth = th.Tensor(z)
wth = th.Tensor(w)
tensor = xth*th.transpose(yth, 0, 1)*th.transpose(zth,0,2)*th.transpose(wth,0,3)
diag = th.diagonal(tensor, dim1 = -2, dim2 = -1)
result = th.sum(tensor) - th.sum(diag)
result /= np.math.factorial(len(x))
print(result)
The order is between O(n^2.37..) - O(n^3), depending on the pytorch implementation of the matrix multiplication.
I don't see why not, just choose properly the dimensions to transpose and take the diagonal.
I don't see why would this solution won't work in a log-space.
pd: my knowledge in pytorch is quite limited, but I'm sure you can define x,y,z,w in a more elegant way.

Multiplying subarrays of tensor

I am trying to implement a multivariate Gaussian Mixture Model and am trying to calculate the probability distribution function using tensors. There are n data points, k clusters, and d dimensions. So far, I have two tensors. One is a (n,k,d) tensor of centered data points and the other is a kxdxd tensor of covariance matricies. I can compute an nxk matrix of probabilities by doing
centered = np.repeat(points[:,np.newaxis,:],K,axis=1) - mu[np.newaxis,:] # KxNxD
prob = np.zeros(n,k)
constant = 1/2/np.pow(np.pi, d/2)
for n in range(centered.shape[1]):
for k in range(centered.shape[0]):
p = centered[n,k,:][np.newaxis] # 1xN
power = -1/2*(p # np.linalg.inv(sigma[k,:,:]) # p.T)
prob[n,k] = constant * np.linalg.det(sigma[k,:,:]) * np.exp(power)
where sigma is the triangularized kxdxd matrix of covariances and centered are mypoints. What is a more pythonic way of doing this using numpy's tensor capabilites?

Just a couple of quick observations:
I don't see you using p in the loop; is this a mistake? Using n instead?
The T in centered[n,k,:].T does nothing; with that index the array is 1d
I'm not sure if np.linal.inv can handle batches of arrays, allowing np.linalg.inv(sigma).
# allows batches, just so long as the last 2 dim are the ones entering into the dot (with the usual last of A, 2nd to the last of B rule; einsum can also be used.
again does np.linalg.det handle batches?

Efficient linear algebra for block-diagonal matrices in compressed format

I have a linear system in which all the matrices are block diagonal. They have N blocks identical in shape.
Matrices are stored in compressed format as numpy arrays with shape (N, n, m), while the shape of the vectors is (N, m).
I currently implemented the matrix-vector product as
import numpy as np
def mvdot(m, v):
return (m * np.expand_dims(v, -2)).sum(-1)
Thanks to broadcasting rules, if all the blocks of a matrix are the same I have to store it only once in an array with shape (1, n, m): the product with a vector (N, m) still gives the correct (N, n) vector.
My questions are:
how to implement an efficient matrix-matrix product that yields the matrix with shape (N, n, m) from two matrices with shapes (N, n, p) and (N, p, m)?
is there a way to perform these operations with a numpy built-in (possibly faster) function? Functions like np.linalg.inv make me think that numpy was designed to support this compressed format for block diagonal matrices.

If I understand your question correctly, you have two arrays of shape (N,n,p) and (N,p,m), respectively, and their product should be of shape (N,n,m) where element [i,:,:] is the matrix product of M1[i,:,:] and M2[i,:,:]. This can be achieved using numpy.einsum:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = np.einsum('ijk,ikl->ijl',M1,M2)
# check if all the submatrices are what we expect
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
# True
Numpy's einsum is an incredibly versatile construction for complicated linear operations, and it's usually pretty efficient with two operands. The idea is to rewrite your operation in an indexed way: what you need is to multiply M1[i,j,k] with M2[i,k,l] for each i,j,l, and sum over k. This is exactly what the above call to einsum does: it collapses the index k, and performs the necessary products and assignments along the remaining dimensions in the given order.
The matrix-vector product can be done similarly:
M = np.random.rand(N,n,m)
v = np.random.rand(N,m)
Mvprod = np.einsum('ijk,ik->ij',M,v)
It's possible that numpy.dot can be coerced with the proper transposes and dimension tricks to directly do what you want, but I couldn't make that work.
Both of the above operations can be done in the same function call by allowing an implicit number of dimensions within einsum:
def mvdot(M1,M2):
return np.einsum('ijk,ik...->ij...',M1,M2)
Mprod = mvdot(M1,M2)
Mvprod = mvdot(M,v)
In case the input argument M2 is a block matrix, there will be a leading dimension appended to the result, creating a block matrix. In case M2 is a "block vector", the result will be a block vector.

Since Python 3.5 and above, the previous example can be simplified using the matrix multiplication operator # (numpy.matmul) which treats this case as a stack of matrices residing in the last two indexes and broadcast accordingly:
import numpy as np
N = 7
n,p,m = 3,4,5
M1 = np.random.rand(N,n,p)
M2 = np.random.rand(N,p,m)
Mprod = M1 # M2 # similar to np.matmul(M1, M2)
all([np.allclose(np.dot(M1[k,...],M2[k,...]),Mprod[k,...]) for k in range(N)])
#True

Null matrix with constant diagonal, with same shape as another matrix

I'm wondering if there is a simple way to multiply a numpy matrix by a scalar. Essentially I want all values to be multiplied by the constant 40. This would be an nxn matrix with 40's on the diagonal, but I'm wondering if there is a simpler function to use to scale this matrix. Or how would I go about making a matrix with the same shape as my other matrix and fill in its diagonal?
Sorry if this seems a bit basic, but for some reason I couldn't find this in the doc.

If you want a matrix with 40 on the diagonal and zeros everywhere else, you can use NumPy's function fill_diagonal() on a matrix of zeros. You can thus directly do:
N = 100; value = 40
b = np.zeros((N, N))
np.fill_diagonal(b, value)
This involves only setting elements to a certain value, and is therefore likely to be faster than code involving multiplying all the elements of a matrix by a constant. This approach also has the advantage of showing explicitly that you fill the diagonal with a specific value.
If you want the diagonal matrix b to be of the same size as another matrix a, you can use the following shortcut (no need for an explicit size N):
b = np.zeros_like(a)
np.fill_diagonal(b, value)

Easy:
N = 100
a = np.eye(N) # Diagonal Identity 100x100 array
b = 40*a # Multiply by a scalar
If you actually want a numpy matrix vs an array, you can do a = np.asmatrix(np.eye(N)) instead. But in general * is element-wise multiplication in numpy.