What method should i use?
a,b are vectors, or arrays n dimensionals, and X is (nxn) dimensional. Im using numpy for this.
I have a
X^T X a=X^T b
matrix vector equation. X,X^T,b is known and the question is a.
I have tried X^T X as X^T#X=z and doing z^-1, then
z^-1#X^T =g and doing np.linalg.solve(g,b). Is there some basic linear algebra i'm doing wrong here?
Is there a specific python code for these types of equations?
"Is there a specific python code for these types of equations?"
Yes. The problem that you are solving is ordinary least squares (see also linear least squares).
NumPy has the function numpy.linalg.lstsq for solving such problems. In your case, to compute a given X and b, you would use
a, residuals, rank, singvals = np.linalg.lstsq(X, b)
residuals, rank and singvals are additional information returned by lstsq, as explained in the docstring.
Related
Scipy.optimize.root enables you to minimize a vector function while Scipy.optimize.root_scalar enables you to minimize a scalar function. What I need to solve is somewhat in between. I have a bunch of complex functions f_i depending on index i and x_i, I want to solve f_1(x_1)=0,f_2(x_2)=0,...,f_n(x_n)=0. But instead of solving them using for loop, I want to solve in a vectorized style. The reason for that is because using for loop to query the value of f_1,...,f_n is expensive. But querying them in batch (f_1,...f_n) is relatively cheaper.
Let f=(f_1,...,f_n) and x=(x_1,...x_n). We want to solve f(x)=(f_1(x_1),f_2(x_2),...f_n(x_n))=0. By directly calling scipy.optimize.root is not ideal since the solver has no idea that each dimension is independent.
A toy example:
from scipy import optimize
import numpy as np
coef = np.arange(10)
def f(x):
return x ** 2 + 2 * coef * x + coef ** 2
optimize.root(f, np.zeros(10))
How can we let the solver know each dimension is independent to speed it up?
The above is just a toy example to illustrate my problem. In the real case, the function f is like a black box and there is no analytical derivative for each component f_1, f_2, ...f_n. So I couldn't just input a diagonal Jacobian to the solver. I tried to look at if we can let the solver know that Jacobian matrix should be diagonal but I have no luck towards this path. Any suggestions?
Consider a case where, given an MxM matrix A and a vector b, I want to solve something of the form inv(A # A.T) # b (where I know A is invertible).
As far as I know, it is always faster to use solve_* rather than inv. There are also variants for more efficient solving for PSD matrices (which A # A.T must be), using Cholesky factorization.
My question - since I'm constructing the matrix A # A.T just to immediately throw it away - is there a more specialized procedure for solving linear equations with the gram matrix of A without having to construct it?
You can compute the factorization of A and then use that to solve your system.
Assume we want to solve
A A^T x = b
for x.
Compute the factorization of A=LU.
Then solve Ay=b for y.
Then solve A^T x = y for x.
This way you dont have to compute the matrix A^T A.
Note that if one has a factorization of A=LU then one can solve Ax=b as well as A^T x=b efficiently for x.
This is because A^T=U^T L^T which is again a factorization of a lower times an upper triangular matrix.
Solve_ivp is an initial value problem solver function from Scipy.
In a few words
scipy.integrate.solve_ivp(fun, t_span, y0, method=’RK45’, t_eval=None, dense_output=False, events=None, vectorized=False, args=None, **options)
Solve an initial value problem for a system of ODEs.
This function numerically integrates a system of ordinary differential equations given an initial value.
In the solve_ivp function documentation (Scipy reference guide 1.4.1 page 695) we have the following
Parameters
fun [callable] Right-hand side of the system. The calling signature is fun(t, y). Here t is a scalar, and there are two options for the ndarray y: It can either have the shape (n,); then fun must return array_like with shape (n,). Alternatively, it can have the shape (n, k); then fun must return an array_like with shape (n, k), i.e. each column corresponds to a single column in y. The choice between the two options is determined by the vectorized argument (see below). The vectorized implementation allows a faster approximation of the Jacobian by finite differences (required for stiff solvers).
Here n stands for the no of dimensions in y.
What does k stand for? This might seem a very naive question for those who know the answer. But please, believe me, I really could not find it (at least not in the documentation).
The great hpaulj's answer to this question seems to shed some light. But well, IMHO it still is too dark to move around.
Well, y kan depend on other variables, say x.
y generally stands for n fields.
If n = 3 then y = numpy.array([u, v, w]) and this means that u, v and w can all depend on x.
k is the number of values that x can have.
I have been trying to find an answer to this problem for a couple of hours now, but i can't find anything so far...
So I have two vectors let's call them b and x, of which i know all values. They add up to be the same amount, so sum(b) = sum(x).
I also have a Matrix, let's call it A, of which i know what values are 0, all the other values are unknown (but are different from 0).
Furthermore, the the elements of each column of A has the sum of 1 (I think that's called it's a left stochastic matrix)
Generally the Equation can be written in the form A*x = b.
Now I'm trying to find the missing values of A.
I have found one answer to the general problem here: https://math.stackexchange.com/questions/1170843/solving-ax-b-when-x-and-b-are-given
Furthermore i looked at the documentation of numpy.linalg
:https://docs.scipy.org/doc/numpy/reference/routines.linalg.html, but i just can't figure out how to do it.
It looks similar to a multi linear regression problem, but also on sklearn, i couldn't find anything: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression
Not a complete answer, but a bit of a more formal statement of the problem.
I think this can be solved as just a system of linear equations. Let
NZ = {(i,j)|a(i,j) is not fixed to zero}
Then write:
sum( j | (i,j) ∈ NZ, a(i,j) * x(j) ) = b(i) ∀i
sum( i | (i,j) ∈ NZ, a(i,j)) = 1 ∀j
This is just a system of linear equations in a(i,j). It may be under- (or over-) determined and it may be sparse. I think it depends a bit on this how to solve it. It may possible to think about these as constraints in a linear (or quadratic) programming problem. That would allow you to add an objective (in case of an underdetermined system or overdetermined -- in that case minimize sum of squared deviations, or 1-norm of deviations). In addition we can add bounds on a(i,j) (e.g. lower bounds of zero and upper bounds of one). So a linear programming approach may be what you are looking for.
This problem looks a bit like matrix balancing. This is used a lot for economic data sets that come from different sources and where we want to reconcile the data to get a consistent data set usable for subsequent modeling.
I'm trying to solve an overdetermined system with QR decomposition and linalg.solve but the error I get is
LinAlgError: Last 2 dimensions of the array must be square.
This happens when the R array is not square, right? The code looks like this
import numpy as np
import math as ma
A = np.random.rand(2,3)
b = np.random.rand(2,1)
Q, R = np.linalg.qr(A)
Qb = np.matmul(Q.T,b)
x_qr = np.linalg.solve(R,Qb)
Is there a way to write this in a more efficient way for arbitrary A dimensions? If not, how do I make this code snippet work?
The reason is indeed that the matrix R is not square, probably because the system is overdetermined. You can try np.linalg.lstsq instead, finding the solution which minimizes the squared error (which should yield the exact solution if it exists).
import numpy as np
A = np.random.rand(2, 3)
b = np.random.rand(2, 1)
x_qr = np.linalg.lstsq(A, b)[0]
You need to call QR with the flag mode='reduced'. The default Q R matrices are returned as M x M and M x N, so if M is greater than N then your matrix R will be nonsquare. If you choose reduced (economic) mode your matrices will be M x N and N x N, in which case the solve routine will work fine.
However, you also have equations/unknowns backwards for an overdetermined system. Your code snippet should be
import numpy as np
A = np.random.rand(3,2)
b = np.random.rand(3,1)
Q, R = np.linalg.qr(A, mode='reduced')
#print(Q.shape, R.shape)
Qb = np.matmul(Q.T,b)
x_qr = np.linalg.solve(R,Qb)
As noted by other contributors, you could also call lstsq directly, but sometimes it is more convenient to have Q and R directly (e.g. if you are also planning on computing projection matrix).
As shown in the documentation of numpy.linalg.solve:
Computes the “exact” solution, x, of the well-determined, i.e., full rank, linear matrix equation ax = b.
Your system of equations is underdetermined not overdetermined. Notice that you have 3 variables in it and 2 equations, thus fewer equations than unknowns.
Also notice how it also mentions that in numpy.linalg.solve(a,b), a must be an MxM matrix. The reason behind this is that solving the system of equations Ax=b involves computing the inverse of A, and only square matrices are invertible.
In these cases a common approach is to take the Moore-Penrose pseudoinverse, which will compute a best fit (least squares) solution of the system. So instead of trying to solve for the exact solution use numpy.linalg.lstsq:
x_qr = np.linalg.lstsq(R,Qb)