Develop Reference

Want to find the row operations when inverting a matrix

I have a sequence of matrices and I want to find the inverse of each one. I suspect that there is a pattern to the row operations when putting it into reduced echelon form/inverting. Is there a library in python that I can use to give me a sequence of row operations such as R1 = R1-aR2, R3=R1+bR4, etc etc until it is in reduced row echelon form or completely inverted.
I tried googling but it's mostly guides on how to invert a matrix.

There are libraries in Python that can help you find the sequence of row operations when inverting a matrix. One such library is SymPy, which is a Python library for symbolic mathematics. It has a built-in function called rref (reduced row echelon form) that can be used to put a matrix in reduced row echelon form, and a inv function that can be used to find the inverse of a matrix.
Here is an example of how you can use SymPy to find the row operations needed to invert a matrix:
from sympy import *
# Define the matrix
A = Matrix([[1, 2, 3], [4, 5, 6], [7, 8, 10]])
# Find the reduced row echelon form of the matrix
rref, pivots = A.rref()
# Find the inverse of the matrix
A_inv = A.inv()
# Print the row operations
for i in range(len(pivots)):
row = pivots[i]
print(f"R{row+1} = R{row+1} - {rref[i, row]}*R{i+1}")
This will print the sequence of row operations needed to find the inverse of matrix A.
You can also use SymPy's rref() function to get both the row-echelon form and the pivot columns of the matrix in one call.
It's also worth noting that using libraries like numpy and scipy have inverse function which you can use to get the inverse of a matrix without doing the operations manually.

Rotating high dimensional orthogonal vectors in python

Say I have orthogonal vectors of dimension n. I have two questions:
How to create/initialize n such orthogonal vectors in python using the existing packages (numpy, scipy, pytorch etc)? Ideally these basis vectors should be as random as possible given the constraints, that is avoiding values such as 1,0,-1 as much as possible.
How can I rotate them by an angle alpha so that they remain orthogonal in high dimensional space? Again, I would like to do this in python, preferably using existing implementation in some of the packages.

You could do a QR decomposition of a random matrix, and set the R-component to zero. This will yield a random orthogonal matrix.
Vary one of the Givens angles in the Q components and you get a random rotation.

I have an answer to your first question and some thoughts on how to approach the second.
1.
import numpy as np
#let's say we're working in 5-D space
n = 5
#set of orthogonal basis vectors
basis_vectors = []
for _ in range(n):
vector = np.random.randn(n)
for basis_vector in basis_vectors:
vector -= basis_vector.dot(vector) * vector
#uncomment following to make basis orthonormal
#vector /= np.linalg.norm(rotation_axis)
basis_vectors.append(vector)
for a_i in range(n):
for b_i (a_i + 1, n):
assert np.allclose(basis_vectors[a_i].dot(basis_vectors[b_i]), 0)
Because you want to rotate both vectors in the same manner, there must be a way to preserve information on the way each rotation is carried out (e.g. rotation matrix, rotation quaternion).
Preexisting implementations of 3D Rotation matrices include the Scipy function scipy.spatial.transform.Rotation.from_rotvec and Python's quaternion module (see henneray's answer), but these are only for 3D vectors. Unless I've overlooked something, it'd be necessary to implement ND rotation from scratch.
Here's a general outline of the steps I would take:
Find 2 linearly independent ND basis vectors of the 2D plane in which you want to rotate the two vectors. (the vectors you want to rotate, a and b, aren't necessarily on this plane)
Find the remaining (N-2)D basis vectors that are linearly independent to these first 2 vectors. Combined the N basis vectors should span the ND space.
Break up each of the two N-D orthogonal vectors you want to rotate into the sum of two vectors: 1) the vectors' projections onto the 2D plane you've constructed and 2) the "remainder" of the vector that doesn't fall on the 2D plane. Set this "remainder" aside for now.
Perform a change of basis on the projected N-D vectors so that they can be expressed as the product of a 2D vector and an Nx2 matrix, which has its columns set to each of the corresponding basis vectors calculated. Keep in mind that the 2D vector is now in a modified coordinate space, not the original.
Construct the 2D rotation matrix corresponding to the desired rotation within the 2D plane identified in the first step. Perform the rotation transformation on the 2D vectors.
Transform the rotated 2D vectors back into ND vectors in the main coordinate system by multiplying the by the Nx2 matrix.
Add the "remainder" set aside earlier back to the mapped ND vector.
The resulting two vectors have been rotated by an arbitrary angle on a particular 2D plane, but maintain orthogonality.
I hope these ideas help you. Take care.

I found a scipy function that can do 1, ortho_group, still wondering about 2.
>>> from scipy.stats import ortho_group
>>> m = ortho_group.rvs(dim=4)
>>> m
array([[-0.25952499, 0.435163 , 0.04561972, 0.86092902],
[-0.44123728, -0.38814758, -0.80217271, 0.10568846],
[ 0.16909943, -0.80707234, 0.35548632, 0.44007851],
[-0.8422362 , -0.0927839 , 0.47756387, -0.23229737]])
>>> m.dot(m.T)
array([[ 1.00000000e+00, -1.68203864e-16, 1.75471554e-16,
9.74154717e-17],
[-1.68203864e-16, 1.00000000e+00, -1.18506045e-16,
-1.81879209e-16],
[ 1.75471554e-16, -1.18506045e-16, 1.00000000e+00,
1.16692720e-16],
[ 9.74154717e-17, -1.81879209e-16, 1.16692720e-16,
1.00000000e+00]])

How to compare a search query to SVD resulting 'w' matrix

I am working on developing a search algorithm and I am struggling to understand how to actually use the results of a singular value decomposition ( u,w,vt = svd(a) ) reduction on a term-document matrix.
For example, say I have an M x N matrix as follows where each column represents a document vector (number of terms in each document)
a = [[ 0, 0, 1 ],
[ 0, 1, 2 ],
[ 1, 1, 1 ],
[ 0, 2, 3 ]]
Now, I could run a tf-idf function on this matrix to generate a score for each term/document value, but for the sake of clarity, I will ignore that.
SVD Results
Upon running SVD on this matrix, I end up with the following diagonal vector for 'w'
import svd
u,w,vt = svd.svd(a)
print w
// [4.545183973611469, 1.0343228430392626, 0.5210363733873331]
I understand more or less what this represents (thanks to a lot of reading and particularly this article https://simonpaarlberg.com/post/latent-semantic-analyses/), however I can't figure out how to relate this resulting 'approximation' matrix back to the original documents? What do these weights represent? How can I use this result in my code to find documents related to a term query?
Basically... How do I use this?

The rank-r SVD reduces a rank-R MxN matrix A into r orthogonal rank-1 MxN matrices (u_n * s_n * v_n'). If you use these singular values and vectors to reconstruct the original matrix, you will obtain the best rank-r approximation of A.
Instead of storing the full matrix A, you just store the u_n, s_n, and v_n. (A is MxN, but U is Mxr, S can be stored in one dimension as r elements, and V' is rxN).
To approximate A * x, you simply compute (U * (S * (V' * x))) [Mxr x rxr x rxN x Nx1]. You can speed this up further by storing (U * S) instead of U and S separately.
So what do the singular values represent? In a way, they represent the energy of each rank-1 matrix. The higher a singular value is, the more that its associated rank-1 matrix contributes to the original matrix and the worse your reconstruction will be if it is not included if it is truncated.
Note that this procedure is closely related to Principal Component Analysis, which is performed on covariance matrices and is commonly used in machine learning to reduce the dimensionality of measured N-dimensional variables.
Additionally, it should be noted that the SVD is useful for many other applications in signal processing.
More information is on the Wikipedia article.

Multidimensional matrix multiplication in python

I have matrix A of dimension 500x2000x30 and matrix B of dimension 30x5.
You can think that there are 500 instances of 2000x30 as matrix A is of dimension 500x2000x30.
I want to multiply each of 1x2000x30 from A with matrix B to obtain new matrix of size 1x2000x5.
i.e. A X B should give me a matrix of dimension 500x2000x5
Obviously looping 500 times through matrix A is a solution but is there an efficient way to achieve this?
Edit: Both A and B are numpy arrays

If you have numpy arrays you can use the np.dot function for this:
np.dot(A, B)
It will do exactly what you want, i.e. "contract" the last axis of A with the first axis of B:
For 2-D arrays it is equivalent to matrix multiplication, and for 1-D arrays to inner product of vectors (without complex conjugation). For N dimensions it is a sum product over the last axis of a and the second-to-last of b:
dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])

First of all any multidimensional array is a n times b split of a flatten array. The 2,3,2 dimension array, [ [ [A,B],[C,D],[E,F] ], [ [G,H],[I,J],[K,L] ] ] is simply divide A,B,C,D,E,F,G,H,I,J,K,L into 2 pieces , then 3 pieces, then 2 pieces again and there is your multidimensional array. Vectors are here two dimensional.
Secondly, two dimensional matrix multiplication is also fixed length vector multiplication with a combination. (n,m)*(m,l) dimensional matrix multiplication is actually term by term multiplication and sum of results of l different m vectors and n different m vectors. l times n combination of two different m vectors.
When you get that you simply can convert 20 dimension matrix to two dimension matrix with same vector size and multiply them and reshape it back.
Quick Example:
import numpy as np
a=np.random.random(120).reshape(2,5,3,4)
b=np.random.random(120).reshape(5,3,4,2)
br=np.rollaxis(b,3,2).reshape(30,4).T
# converted all combinations to a series of 4 dimension vectors here
test1=np.matmul(a.reshape(30,4),br).reshape(2,5,3,5,3,2)
test2=np.dot(a,b)
np.all(test1==test2)
returns
array(True)
lets do it basically for all combinations (dimensions)
sa=a.shape
sb=b.shape
res=np.ndarray([sa[0],sa[1],sb[0],sb[1],sa[2],sb[3]])
for i in range(a.shape[0]):
for j in range(a.shape[1]):
for k in range(b.shape[0]):
for n in range(b.shape[1]):
x=np.matmul(a[i,j],b[k,n])
np.append(res,x)
print(np.any(res==np.dot(a,b).reshape(2,5,5,3,3,2)))
returns
True
USE CASE:
Suppose we have a case such as for 10 different materials and 20 different geometries 3 different physical quantity in 3 dimensions vectors will be matrix multiplied for a geometry and physics processing neural network layer which will be calculated with genetic selective algorithm which has neural connection coefficient vectors of 5 different groups of population each having 100 different gene sequence (population) and 20 neural nodes. In a such case you may benefit calculating it at once or you can somehow serialize this calculation to two flat arrays and send it to your gpu or cpu according to your concurrent free ram amount. In such case you may want to understand how this calculations work.
In any case of multidimensional matrix calculation combinations of vectors are calculated term by term
You may want to multiply first term with seconds vectors last term and sum the rest. It is up to you but understanding how it works is important.
So here is a simple illustration I have used to undestand this.
[ [ [A,B],[C,D],[E,F] ], [ [G,H],[I,J],[K,L] ] ] --> ABCDEFGHIJKL
[ [ [1,2],[3,4],[5,6] ], [ [7,8],[9,0],[€,$] ] ] --> 1234567890€$
use operator(multiply) term by term shifting first array by amount of vector size (2)
ABCDEFGHIJKL CDEFGHIJKLAB FGHIJKLABCDE ...
1234567890€$ 1234567890€$ 1234567890€$ ...
Here comes all combinations
append all of them and reshape and use another operator (+)
[A+2*B],[C*3+D*4],[E*5,F*6] ...... [I*€+J*$]
Hope this helps and saves time to grasp this.

Null matrix with constant diagonal, with same shape as another matrix

I'm wondering if there is a simple way to multiply a numpy matrix by a scalar. Essentially I want all values to be multiplied by the constant 40. This would be an nxn matrix with 40's on the diagonal, but I'm wondering if there is a simpler function to use to scale this matrix. Or how would I go about making a matrix with the same shape as my other matrix and fill in its diagonal?
Sorry if this seems a bit basic, but for some reason I couldn't find this in the doc.

If you want a matrix with 40 on the diagonal and zeros everywhere else, you can use NumPy's function fill_diagonal() on a matrix of zeros. You can thus directly do:
N = 100; value = 40
b = np.zeros((N, N))
np.fill_diagonal(b, value)
This involves only setting elements to a certain value, and is therefore likely to be faster than code involving multiplying all the elements of a matrix by a constant. This approach also has the advantage of showing explicitly that you fill the diagonal with a specific value.
If you want the diagonal matrix b to be of the same size as another matrix a, you can use the following shortcut (no need for an explicit size N):
b = np.zeros_like(a)
np.fill_diagonal(b, value)

Easy:
N = 100
a = np.eye(N) # Diagonal Identity 100x100 array
b = 40*a # Multiply by a scalar
If you actually want a numpy matrix vs an array, you can do a = np.asmatrix(np.eye(N)) instead. But in general * is element-wise multiplication in numpy.