numpy: Possible for zero determinant matrix to be inverted? - python
By definition, a square matrix that has a zero determinant should not be invertible. However, for some reason, after generating a covariance matrix, I take the inverse of it successfully, but taking the determinant of the covariance matrix ends up with an output of 0.0.
What could be potentially going wrong? Should I not trust the determinant output, or should I not trust the inverse covariance matrix? Or both?
Snippet of my code:
cov_matrix = np.cov(data)
adjusted_cov = cov_matrix + weight*np.identity(cov_matrix.shape[0]) # add small weight to ensure cov_matrix is non-singular
inv_cov = np.linalg.inv(adjusted_cov) # runs with no error, outputs a matrix
det = np.linalg.det(adjusted_cov) # ends up being 0.0
The numerical inversion of matrices does not involve computing the determinant. (Cramer's formula for the inverse is not practical for large matrices.) So, the fact that determinant evaluates to 0 (due to insufficient precision of floats) is not an obstacle for the matrix inversion routine.
Following up on the comments by BobChao87, here is a simplified test case (Python 3.4 console, numpy imported as np)
A = 0.2*np.identity(500)
np.linalg.inv(A)
Output: a matrix with 5 on the main diagonal, which is the correct inverse of A.
np.linalg.det(A)
Output: 0.0, because the determinant (0.2^500) is too small to be represented in double precision.
A possible solution is a kind of pre-conditioning (here, just rescaling): before computing the determinant, multiply the matrix by a factor that will make its entries closer to 1 on average. In my example, np.linalg.det(5*A) returns 1.
Of course, using the factor of 5 here is cheating, but np.linalg.det(3*A) also returns a nonzero value (about 1.19e-111). If you try np.linalg.det(2**k*A) for k running through modest positive integers, you will likely hit one that will return nonzero. Then you will know that the determinant of the original matrix was approximately 2**(-k*n) times the output, where n is the matrix size.
Related
Matrix Mutliplication should converge, but it doesn't
I'm trying to calculate the eigenvalues of multiplication between matrices of this kind: import numpy as np μ=1.5 σ=0.5 m=np.random.normal(μ,σ) P=[[-1, m, 0.1, 0.1],[1,0,0,0],[0,1,0,0],[0,0,1,0]] #matrix n=200 for i in range(n): m=np.random.normal(μ,σ) T=[[-1, m, 0.1, 0.1],[1,0,0,0],[0,1,0,0],[0,0,1,0]] P=np.dot(T,P) l,v=np.linalg.eig(T) λ,w=np.linalg.eig(P) print(l) print(λ) Being three of the eigenvalues of the matrix T lower than 1 (in module), I expect something similar from the eigenvalues of the matrix product P, in particular that the three eigenvalues lower than 1 will correspond to eigenvalues of P which decreases and converge to 0 with n increasing. In fact, this is true until n=40. Then it doesn't work anymore. I'm sure there is something wrong in the algorithm and not in the math because for σ=0 P would be the product of n identical matrices with eigenvalues lower than 1. But the eigenvalues of P diverges.
Your T[i] is not guaranteed to have eigenvalues less than 1 Add an assertion and you will see when it violates the assumption import numpy as np μ=1.5 σ=0.5 m=np.random.normal(μ,σ) P=[[-1, m, 0.1, 0.1],[1,0,0,0],[0,1,0,0],[0,0,1,0]] #matrix n=200 for i in range(n): m=np.random.normal(μ,σ) T=[[-1, m, 0.1, 0.1],[1,0,0,0],[0,1,0,0],[0,0,1,0]] assert np.all(abs(np.linalg.eigvals(T)) < 1) P=np.dot(T,P) l,v=np.linalg.eig(T) λ,w=np.linalg.eig(P) print(abs(l)) print(λ) Being a stochastic system even violating the constraint sometimes it can converge, but it requires some careful analysis of the expected values of the sequence. The matrix structure resembles the controlable canonical form in linar dynamic systems [1]. But since the matrix T sometimes have eigenvalues larger than 1, the reasoning you suggested does not apply in this case.
The factorization of B could not be completed and no eigenvalues or eigenvectors were computed
I'm trying to solve the eigenvalue equation, A x = λ B x with A and B as 16×16 square Hermitian matrices. Using the linalg library on python (Spyder4) and I got an error saying:
LinAlgError: The leading minor of order 12 of B is not positive definite.
The factorization of B could not be completed and no eigenvalues or eigenvectors were computed.
here is the matrix and the command I used:
H = np.array([[a11,0,0,0,0,0,0,0,a19,a110,a111,a112,a113,a114,a115,a116]
[0,a22,0,0,0,0,0,0,a29,a210,a211,a212,a213,a214,a215,a216],
[0,0,a33,0,0,0,0,0,a39,a310,a311,a312,a313,a314,a315,a316],
[0,0,0,a44,0,0,0,0,a49,a410,a411,a412,a413,a414,a415,a416],
[0,0,0,0,a55,0,0,0,a59,a510,a511,a512,a513,a514,a515,a516],
[0,0,0,0,0,a66,0,0,a69,a610,a611,a612,a613,a614,a615,a616],
[0,0,0,0,0,0,a77,0,a79,a710,a711,a712,a713,a714,a715,a716],
[0,0,0,0,0,0,0,a88,a89,a810,a811,a812,a813,a814,a815,a816],
[0,0,0,0,0,0,0,0,a99,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,a1010,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,a1111,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,a1212,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,a1313,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,a1414,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,a1515,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,a1616]])
S = np.array([[1,0,0,0,0,0,0,0,b19,b110,b111,b112,b113,b114,b115,b116],
[0,1,0,0,0,0,0,0,b29,b210,b211,b212,b213,b214,b215,b216],
[0,0,1,0,0,0,0,0,b39,b310,b311,b312,b313,b314,b315,b316],
[0,0,0,1,0,0,0,0,b49,b410,b411,b412,b413,b414,b415,b416],
[0,0,0,0,1,0,0,0,b59,b510,b511,b512,b513,b514,b515,b516],
[0,0,0,0,0,1,0,0,b69,b610,b611,b612,b613,b614,b615,b616],
[0,0,0,0,0,0,1,0,b79,b710,b711,b712,b713,b714,b715,b716],
[0,0,0,0,0,0,0,1,b89,b810,b811,b812,b813,b814,b815,b816],
[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1]])
lamda, x = lg.eigh(H, S, lower=False, eigvals_only=False)
print("Eigenvalues")
print(lamda)
The variables in the matrix are user inputs (Some -ve values, and complex numbers).
The eigenvalue is computed when I use "linalg.eig" command but since my actual matrix is symmetric, I'm trying to use eigh command.
Has anybody faced this problem and/or suggest what the error is about?
Thanks
The documentation says that the second argument must be a complex Hermitian or a real symmetric definite positive matrix. If your b are complex, that it appears that this argument is neither. The documentation also says that this method throws a LinAlgError: If eigenvalue computation does not converge, an error occurred, or b matrix is not definite positive. Note that if input matrices are not symmetric or Hermitian, no error will be reported but results will be wrong.
Find eigenvectors with specific eigenvalue of sparse matrix in python
I have a large sparse matrix and I want to find its eigenvectors with specific eigenvalue. In scipy.sparse.linalg.eigs, it says the required argument k: "k is the number of eigenvalues and eigenvectors desired. k must be smaller than N-1. It is not possible to compute all eigenvectors of a matrix". The problem is that I don't know how many eigenvectors corresponding to the eigenvalue I want. What should I do in this case?
I'd suggest using Singular Value Decomposition (SVD) instead. There is a function from scipy where you can use SVD from scipy.sparse.linalg import svds and it can handle sparse matrix. You can find eigenvalues (in this case will be singular value) and eigenvectors by the following: U, Sigma, VT = svds(X, k=n_components, tol=tol) where X can be sparse CSR matrix, U and VT is set of left eigenvectors and right eigenvectors corresponded to singular values in Sigma. Here, you can control number of components. I'd say start with small n_components first and then increase it. You can rank your Sigma and see the distribution of singular value you have. There will be some large number and drop quickly. You can make threshold on how many eigenvectors you want to keep from singular values. If you want to use scikit-learn, there is a class sklearn.decomposition.TruncatedSVD that let you do what I explained.
Numpy covariance matrix nonrmalisation
I know that numpy.cov calculates the covariance given a N dimensional array. I can see from the documentation on GitHub that the normalisation is done by (N-1). But for my specific case, the covariance matrix is given by: where xi is the quantity. i and j are the bins. As you can see from the above equation, this covariance matrix is normalised by (N-1)/N. TO GET THE ABOVE NORMALISATION Can I simply multiply the covariance matrix obtained from numpy.cov by (N-1)**2 / N to get the above normalisation? Is that correct? Or Should I use the bias parameter inside numpy.cov? If so how?
There are two ways of doing this. We can call np.cov with bias=1 and then multiply the result by N-1 or We can multiply the overall covariance matrix obtained by (N-1)**2/N
Numpy Cholesky decomposition LinAlgError
In my attempt to perform cholesky decomposition on a variance-covariance matrix for a 2D array of periodic boundary condition, under certain parameter combinations, I always get LinAlgError: Matrix is not positive definite - Cholesky decomposition cannot be computed. Not sure if it's a numpy.linalg or implementation issue, as the script is straightforward: sigma = 3. U = 4 def FromListToGrid(l_): i = np.floor(l_/U) j = l_ - i*U return np.array((i,j)) Ulist = range(U**2) Cov = [] for l in Ulist: di = np.array([np.abs(FromListToGrid(l)[0]-FromListToGrid(i)[0]) for i, x in enumerate(Ulist)]) di = np.minimum(di, U-di) dj = np.array([np.abs(FromListToGrid(l)[1]-FromListToGrid(i)[1]) for i, x in enumerate(Ulist)]) dj = np.minimum(dj, U-dj) d = np.sqrt(di**2+dj**2) Cov.append(np.exp(-d/sigma)) Cov = np.vstack(Cov) W = np.linalg.cholesky(Cov) Attempts to remove potential singularies also failed to resolve the problem. Any help is much appreciated.
Digging a bit deeper in problem, I tried printing the Eigenvalues of the Cov matrix. print np.linalg.eigvalsh(Cov) And the answer turns out to be this [-0.0801339 -0.0801339 0.12653595 0.12653595 0.12653595 0.12653595 0.14847999 0.36269785 0.36269785 0.36269785 0.36269785 1.09439988 1.09439988 1.09439988 1.09439988 9.6772531 ] Aha! Notice the first two negative eigenvalues? Now, a matrix is positive definite if and only if all its eigenvalues are positive. So, the problem with the matrix is not that it's close to 'zero', but that it's 'negative'. To extend #duffymo analogy, this is linear algebra equivalent of trying to take square root of negative number. Now, let's try to perform same operation, but this time with scipy. scipy.linalg.cholesky(Cov, lower=True) And that fails saying something more numpy.linalg.linalg.LinAlgError: 12-th leading minor not positive definite That's telling something more, (though I couldn't really understand why it's complaining about 12-th minor). Bottom line, the matrix is not quite close to 'zero' but is more like 'negative'
The problem is the data you're feeding to it. The matrix is singular, according to the solver. That means a zero or near-zero diagonal element, so inversion is impossible. It'd be easier to diagnose if you could provide a small version of the matrix. Zero diagonals aren't the only way to create a singularity. If two rows are proportional to each other then you don't need both in the solution; they're redundant. It's more complex than just looking for zeroes on the diagonal. If your matrix is correct, you have a non-empty null space. You'll need to change algorithms to something like SVD. See my comment below.