I am working with a large (complex) Hermitian matrix and I am trying to diagonalize it efficiently using Python/Scipy.
Using the eigh function from scipy.linalgit takes about 3s to generate and diagonalize a roughly 800x800 matrix and compute all the eigenvalues and eigenvectors.
The eigenvalues in my problem are symmetrically distributed around 0 and range from roughly -4 to 4. I only need the eigenvectors corresponding to the negative eigenvalues, though, which turns the range I am looking to calculate into [-4,0).
My matrix is sparse, so it's natural to use the scipy.sparsepackage and its functions to calculate the eigenvectors via eigsh, since it uses much less memory to store the matrix.
Also I can tell the program to only calculate the negative eigenvalues via which='SA'. The problem with this method is, that it takes now roughly 40s to compute half the eigenvalues/eigenvectors. I know, that the ARPACK algorithm is very inefficient when computing small eigenvalues, but I can't think of any other way to compute all the eigenvectors that I need.
Is there any way, to speed up the calculation? Maybe with using the shift-invert mode? I will have to do many, many diagonalizations and eventually increase the size of the matrix as well, so I am a bit lost at the moment.
I would really appreciate any help!
This question is probably better to ask on http://scicomp.stackexchange.com as it's more of a general math question, rather than specific to Scipy or related to programming.
If you need all eigenvectors, it does not make very much sense to use ARPACK. Since you need N/2 eigenvectors, your memory requirement is at least N*N/2 floats; and probably in practice more. Using eigh requires N*N+3*N floats. eigh is then within a factor of 2 from the minimum requirement, so the easiest solution is to stick with it.
If you can process the eigenvectors "on-line" so that you can throw the previous one away before processing the next, there are other approaches; look at the answers to similar questions on scicomp.
Related
I'm currently working with a least-square algorithm on Python, regarding some geodetic calculations.
I chose Python (which is not the fastest) and it works pretty well. However, in my code, I have inversions of large sparse symmetric (non-positive definite, so can't use Cholesky) matrix to execute (image below). I currenty use np.linalg.inv() which is using the LU decomposition method.
I pretty sure there would be some optimization to do in terms of rapidity.
I thought about Cuthill-McKee algotihm to rearange the matrix and take its inverse. Do you have any ideas or advice ?
Thank you very much for your answers !
Good news is that if you're using any of the popular python libraries for linear algebra (namely, numpy), the speed of python really doesn't matter for the math – it's all done natively inside the library.
For example, when you write matrix_prod = matrix_a # matrix_b, that's not triggering a bunch of Python loops to multiply the two matrices, but using numpy's internal implementation (which I think uses the FORTRAN LAPACK library).
The scipy.sparse.linalg module has your back covered when it comes to solving sparsely stored matrices specifying sparse systems of equations. (which is what you do with the inverse of a matrix). If you want to use sparse matrices, that's your way to go – notice that there's matrices that are sparse in mathematical terms (i.e., most entries are 0), and matrices which are stored as sparse matrix, which means you avoid storing millions of zeros. Numpy itself doesn't have sparsely stored matrices, but scipy does.
If your matrix is densely stored, but mathematically sparse, i.e. you're using standard numpy ndarrays to store it, then you won't get any more rapid by implementing anything in Python. The theoretical complexity gains will be outweighed by the practical slowness of Python compared to highly optimized inversion.
Inverting a sparse matrix usually loses the sparsity. Also, you never invert a matrix if you can avoid it at all! For a sparse matrix, solving the linear equation system Ax = b, with A your matrix and b a known vector, for x, is so much faster done forward than computing A⁻¹! So,
I'm currently working with a least-square algorithm on Python, regarding some geodetic calculations.
since LS says you don't need the inverse matrix, simply don't calculate it, ever. The point of LS is finding a solution that's as close as it gets, even if your matrix isn't invertible. Which can very well be the case for sparse matrices!
I have a huge distance matrix.
Example: (10000 * 10000)..
Is there an effective way to find a inverse matrix?
I've tried numpy's Inv() but it's too slow.
Is there a more effective way?
You can try using Singular Value Decomposition https://numpy.org/doc/stable/reference/generated/numpy.linalg.svd.html
Inverting the decomposed form might take less time.
You probably don't actually need the inverse matrix.
There are a lot of numeric techniques that let people solve matrix problems without computing the inverse. Unfortunately, you have not described what your problem is, so there is no way to know which of those techniques might be useful to you.
For such a large matrix (10k x 10k), you probably want to look for some kind of iterative technique. Alternately, it might be better to look for some way to avoid constructing such a large matrix in the first place -- e.g., try using the source data in some other way.
Assume that I have a square matrix M. Assume that I would like to invert the matrix M.
I am trying to use the the fractions mpq class within gmpy2 as members of my matrix M. If you are not familiar with these fractions, they are functionally similar to python's built-in package fractions. The only problem is, there are no packages that will invert my matrix unless I take them out of fraction form. I require the numbers and the answers in fraction form. So I will have to write my own function to invert M.
There are known algorithms that I could program, such as gaussian elimination. However, performance is an issue, so my question is as follows:
Is there a computationally fast algorithm that I could use to calculate the inverse of a matrix M?
Is there anything else you know about these matrices? For example, for symmetric positive definite matrices, Cholesky decomposition allows you to invert faster than the standard Gauss-Jordan method you mentioned.
For general matrix inversions, the Strassen algorithm will give you a faster result than Gauss-Jordan but slower than Cholesky.
It seems like you want exact results, but if you're fine with approximate inversions, then there are algorithms which approximate the inverse much faster than the previously mentioned algorithms.
However, you might want to ask yourself if you need the entire matrix inverse for your specific application. Depending on what you are doing it might be faster to use another matrix property. In my experience computing the matrix inverse is an unnecessary step.
I hope that helps!
Say I have to solve for a large system of equations where
A_i = f(B_i)
B_i = g(A_i)
for many different i. Now, this is a system of equations which are only pair-wise dependent. The lm algorythm has proven most stable to solve this.
Now, I could solve these either independently (i.e. loop over i many scipy.optimize.root, or stack them all together and solve at the same time). I'm unsure which will be the fastest, and it's difficult to know generally. I'm having the following arguments for and against:
The algorythm initially numerically approximates the Jacobian at the provided guess, increasing dimensionality exponentially increases the time it takes to find the Jacobian (speaks against stacking)
Once the Jacobian is found, most of the updating is linear matrix algebra, and therefore should be faster if stacked.
Does that make sense? My conclusion would in that case be "if solving it takes a long time (bad guess or irregular function), stack them, if it's quick, do not stack".
I am not sure I understand correctly; when you say that they are pairwise dependent do you mean that the full system can be decomposed in a collection of small 2x2 systems? If so, you should definitely opt for solving the smaller systems. If not, can you provide some equations?
I am writing code to compute Classical Multidimensional Scaling (abbreviated to MDS) of a very large n by n matrix, n = 500,000 in my example.
In one step of MDS, I need to compute the highest three eigenvalues and their corresponding eigenvectors of a n by n matrix. This matrix is called the B matrix. I only need these three eigenvectors and eigenvalues. Common methods of calculating eigenvectors and eigenvalues of a large matrix take a long time, and I do not require a very accurate answer, so I am seeking an estimation of the eigenvectors and eigenvalues.
Some parameters:
The B matrix is symmetric, real, and quite dense
The eigenvalue decomposition of B in theory should always produce real numbers.
I do not require an entirely precise estimation, just a fast one. I would need it to complete in several hours.
I write in python and C++
My question: Are there fast methods of estimating the three highest eigenvectors and eigenvalues of such a large B matrix?
My progress: I have found a method of approximating the highest eigenvalue of a matrix, but I do not know if I can generalize it to the highest three. I have also found this paper written in 1996, but it is extremely technical and hard for me to read.
G. Golub and C.F Van Loan Matrix Computations 2nd in chapter 9 state that Lanczos algorithms are one choice for this (except that the matrix should ideally be sparse - it clearly works for non-sparse ones as well)
https://en.wikipedia.org/wiki/Lanczos_algorithm
You can get the highest eigenvector of B and then, transform the data into B' using that eigenvector. Then pop the first column of B' and get B'' so you can get the highest eigenvector of B'': it is enough information to compose a plausible second highest eigenvector for B. And then for the third.
About speed: you can randomly sample that huge dataset to be only a dataset of N items. If you are getting only three dimensions, I hope you can also get rid of most of the data to get an overview of the eigenvectors. You can call it: 'electoral poll'. I cannot help you in measuring the error rate, but I will try sampling 1k items, several times, and seeing if results are more or less the same.
Now you can get the mean of several 'polls' to build a 'prediction'.
Have a look at suggestions in this thread
Largest eigenvalues (and corresponding eigenvectors) in C++
As suggested there you can use ARPACK package which has a C++ interface.