I need to solve a large number of 3x3 symmetric, postive-definite systems with Python. So far, I did
res = numpy.zeros(n)
for k, obj in enumerate(data_array):
# construct A, rhs, idx from obj
res[idx] += numpy.linalg.solve(A, rhs)
This produces the correct result, however is also quite slow if n is large. (Well... Yeah.) Perhaps 3x3 isn't a problem size where calling solve() makes much sense.
Any hints?
In NumPy 1.8 and later, numpy.linalg.solve actually broadcasts. For numpy.linalg.solve(a, b), if b.ndim == a.ndim - 1, it will perform a broadcasted matrix-vector solve; otherwise, it'll do a broadcasted matrix-matrix solve. (This decision criterion isn't documented; I had to look at the source.)
If you can efficiently construct a stack of As and rhss, you can call solve once and avoid a Python loop.
Related
I have several thousands of 3x3 matrices, which are stored as a numpy array Nx9, where N is a number of matrices, stored in rows. I want to calculate the eigenvalues & eigenvectors for each of them (i.e. e, v = np.linalg.eig(matrices[row_idx, :].reshape((3, 3)), applied to each row_idx in range(matrices.shape[0])).
So, the function would look as:
def compute_eigen(matrices):
e_list, v_list = [], []
for i in range(matrices.shape[0]):
e, v = np.linalg.eig(matrices[row_idx, :].reshape((3, 3))
v = -v
e_list.append(e)
v_list.append(v)
return e_list, v_list
Now, how to make this run really fast? np.apply_along_axis is just a syntactic sugar for the for-loops, apparently, so it did not help much.
I tried to look into Numba, and run the same function with #jit(nopython=True) decorator, which, indeed, dropped the execution time of this function from ~20s to ~2s for ~6000 matrices. However, is it possible to drop it under 1s? Is there some smart way to do it (assuming no usage of some very powerful hardware, GPUs etc)? Thanks.
I have a very big matrix. buildmatrix is an N by N matrix. So sumHamiltonian is a N^2 by N^2 matrix. Then transform(N) is an N^2 by N(N-1)/2 matrix. (So the resultant shortHamiltonian(N) is an N(N-1)/2 by N(N-1)/2 matrix). The matrix entries are also complex numbers.
If N=200, I get a memory error. Is there a way to rewrite:
def sumHamiltonian(N):
return 0.5*(np.kron(buildmatrix(N),np.identity(N))+np.kron(np.identity(N),buildmatrix(N)))
def shortHamiltonian(N):
return np.matmul(np.transpose(transform(N)),np.matmul(sumHamiltonian(N),transform(N)))
to reduce memory?
I have seen some ways to reduce memory for matrix multiplication (Python/Numpy MemoryError) which is helpful but my memory error comes up at the kronecker product. Is there a way to rewrite this, or better yet, all of the matrix operations to avoid a memory error?
One suggestion is to use sparse matrices. Also, if possible, make your buildmatrix to return a matrix of dtype np.complex64 instead of np.complex128. See the example below:
from scipy import sparse
def buildmatrix(N):
return (np.random.rand(N, N) + np.random.rand(N, N) * 1j).astype(np.complex64)
N = 100
m = buildmatrix(N)
I = np.identity(N)
sumHamiltonian = 0.5 * (np.kron(m, I) + np.kron(I, m))
print(f'{sumHamiltonian.nbytes * 1e-9} GB')
#1.6 GB
m_s = sparse.csr_matrix(m)
I_s = sparse.identity(N)
sumHamiltonian_s = 0.5 * (sparse.kron(m_s, I_s) + sparse.kron(I_s, m_s))
print(f'{sumHamiltonian_s.data.nbytes * 1e-6} MB')
#31.84 MB
#np.all(sumHamiltonian == sumHamiltonian_s)
#True
Similarly try to convert your shortHamiltonian to "sparse version".
Depending on how much memory excess you have, you may be able to decrease the size of your matrices just by using something like .astype(np.float32) (e.g. if the current dtype is np.float64).
buildmatrix(N) is called twice in the same line (same with transform). If the result is deterministic (you just want to use the same result) then try assigning it to a variable, so that it only needs to be called once.
These tips might slightly reduce memory profile but might or might not be enough. I'm not sure if the computation can also be simplified by some identity.
I have been using lpsolve in Python for a very long time, with generally good results. I have even written my own Cython wrapper to it to overcome the mess that the original Python wrapper is.
My Cython wrapper is much faster in the setup of the problem, but of course the solution time only depends on the lpsolve C code and has got nothing to do with Python.
I am only solving real-valued linear problems, no MIPs. The latest (and one of the biggest) LP I had to solve had a constraint matrix of size about 5,000 x 3,000, and the setup plus solve takes about 150 ms. The problem is, I have to solve a slightly modified version of the same problem many times in my simulation (constraints, RHS, bounds and so on are time-dependent for a simulation with many timesteps). The constraint matrix is usually very sparse, with about 0.1% - 0.5% of NNZ or less.
Using lpsolve, the setup and solve of the problem is as easy as the following:
import numpy
from lp_solve.lp_solve import PRESOLVE_COLS, PRESOLVE_ROWS, PRESOLVE_LINDEP, PRESOLVE_NONE, SCALE_DYNUPDATE, lpsolve
# The constraint_matrix id a 2D NumPy array and it contains
# both equality and inequality constraints
# And I need it to be single precision floating point, like all
# other NumPy arrays from here onwards
m, n = constraint_matrix.shape
n_le = len(inequality_constraints)
n_e = len(equality_constraints)
# Setup RHS vector
b = numpy.zeros((m, ), dtype=numpy.float32)
# Assign equality and inequality constraints (= and <=)
b[0:n_e] = equality_constraints
b[-n_le:] = inequality_constraints
# Tell lpsolve which rows are equalities and which are inequalities
e = numpy.asarray(['LE']*m)
e[0:-n_le] = 'EQ'
# Make the LP
lp = lpsolve('make_lp', m, n)
# Some options for scaling the problem
lpsolve('set_scaling', lp, SCALE_DYNUPDATE)
lpsolve('set_verbose', lp, 'IMPORTANT')
# Use presolve as it is much faster
lpsolve('set_presolve', lp, PRESOLVE_COLS | PRESOLVE_ROWS | PRESOLVE_LINDEP)
# I only care about maximization
lpsolve('set_sense', lp, True)
# Set the objective function of the problem
lpsolve('set_obj_fn', lp, objective_function)
lpsolve('set_mat', lp, constraint_matrix)
# Tell lpsolve about the RHS
lpsolve('set_rh_vec', lp, b)
# Set the constraint type (equality or inequality)
lpsolve('set_constr_type', lp, e)
# Set upper bounds for variables - lower bounds are automatically 0
lpsolve('set_upbo', lp, ub_values)
# Solve the problem
out = lpsolve('solve', lp)
# Retrieve the solution for all the variables
vars_sol = numpy.asarray([lpsolve('get_var_primalresult', lp, i) for i in xrange(m + 1, m + n + 1)], dtype=numpy.float32)
# Delete the problem, timestep done
lpsolve('delete_lp', lp)
For reasons that are too long to explain, my NumPy arrays are all single precision floating point arrays, and I'd like them to stay that way.
Now, after all this painful introduction, I would like to ask: does anyone know of another library (with reasonable Python wrappers) that allows me to setup and solve a problem of this size as fast (or potentially faster) than lpsolve? Most of the libraries I have looked at (PuLP, CyLP, PyGLPK and so on) do not seem to have a straightforward way to say "this is my entire constraint matrix, set it in one go". They mostly seem to be oriented towards being "modelling languages" that allow fancy things like this (CyLP example):
# Add variables
x = s.addVariable('x', 3)
y = s.addVariable('y', 2)
# Create coefficients and bounds
A = np.matrix([[1., 2., 0],[1., 0, 1.]])
B = np.matrix([[1., 0, 0], [0, 0, 1.]])
D = np.matrix([[1., 2.],[0, 1]])
a = CyLPArray([5, 2.5])
b = CyLPArray([4.2, 3])
x_u= CyLPArray([2., 3.5])
# Add constraints
s += A * x <= a
s += 2 <= B * x + D * y <= b
s += y >= 0
s += 1.1 <= x[1:3] <= x_u
I honestly don't care about the flexibility, I just need raw speed in the problem setup and solve. Creating NumPy matrices plus doing all those fancy operations above is definitely going to be a performance killer.
I'd rather stay on Open Source solvers if possible, but any suggestion is most welcome. My apologies for the long post.
Andrea.
#infinity77,
I am looking at using lpsolve now. It is fairly straight forward to generate a .lp file for input and it did solve several smaller problems. BUT... I am now attempting to solve a node coloring problem. This is a MIP. My first attempt at a 500 node problem ran about 15 minutes as an lp relaxation. lpsolve has been grinding on the true MIP since last night. Still grinding. These coloring problems are difficult but 14 hours with no end in sight is tooooo much.
The best alternative I am aware of is from coin-or.org;[Coin-OR Solvers] 1 Try their clp and cbc solvers depending on which type of problem you are solving. Benchmarks I have seen say they are the best choice outside of CPLEX and Gurobi. It is free but you will need to be sure it is "legal" for your purposes. A good benchmark paper by Bernhard Meindl and Matthias Templ at Open Source Solver Benchmarks
I have two boolean sparse square matrices of c. 80,000 x 80,000 generated from 12BM of data (and am likely to have orders of magnitude larger matrices when I use GBs of data).
I want to multiply them (which produces a triangular matrix - however I dont get this since I don't limit the dot product to yield a triangular matrix).
I am wondering what the best way of multiplying them is (memory-wise and speed-wise) - I am going to do the computation on a m2.4xlarge AWS instance which has >60GB of RAM. I would prefer to keep the calc in RAM for speed reasons.
I appreciate that SciPy has sparse matrices and so does h5py, but have no experience in either.
Whats the best option to go for?
Thanks in advance
UPDATE: sparsity of the boolean matrices is <0.6%
If your matrices are relatively empty it might be worthwhile encoding them as a data structure of the non-False values. Say a list of tuples describing the location of the non-False values. Or a dictionary with the tuples as the keys.
If you use e.g. a list of tuples you could use a list comprehension to find the items in the second list that can be multiplied with an element from the first list.
a = [(0,0), (3,7), (5,2)] # et cetera
b = ... # idem
for r, c in a:
res = [(r, k) for j, k in b if k == j]
-- EDITED TO SATISFY BELOW COMMENT / DOWNVOTER --
You're asking how to multiply matrices fast and easy.
SOLUTION 1: This is a solved problem: use numpy. All these operations are easy in numpy, and since they are implemented in C, are rather blazingly fast.
http://www.numpy.org/
http://www.scipy.org
also see:
Very large matrices using Python and NumPy
http://docs.scipy.org/doc/scipy/reference/sparse.html
SciPy and Numpy have sparse matrices and matrix multiplication. It doesn't use much memory since (at least if I wrote it in C) it probably uses linked lists, and thus will only use the memory required for the sum of the datapoints, plus some overhead. And, it will almost certainly be blazingly fast compared to pure python solution.
SOLUTION 2
Another answer here suggests storing values as tuples of (x, y), presuming value is False unless it exists, then it's true. Alternate to this is a numeric matrix with (x, y, value) tuples.
REGARDLESS: Multiplying these would be Nasty time-wise: find element one, decide which other array element to multiply by, then search the entire dataset for that specific tuple, and if it exists, multiply and insert the result into the result matrix.
SOLUTION 3 ( PREFERRED vs. Solution 2, IMHO )
I would prefer this because it's simpler / faster.
Represent your sparse matrix with a set of dictionaries. Matrix one is a dict with the element at (x, y) and value v being (with x1,y1, x2,y2, etc.):
matrixDictOne = { 'x1:y1' : v1, 'x2:y2': v2, ... }
matrixDictTwo = { 'x1:y1' : v1, 'x2:y2': v2, ... }
Since a Python dict lookup is O(1) (okay, not really, probably closer to log(n)), it's fast. This does not require searching the entire second matrix's data for element presence before multiplication. So, it's fast. It's easy to write the multiply and easy to understand the representations.
SOLUTION 4 (if you are a glutton for punishment)
Code this solution by using a memory-mapped file of the required size. Initialize a file with null values of the required size. Compute the offsets yourself and write to the appropriate locations in the file as you do the multiplication. Linux has a VMM which will page in and out for you with little overhead or work on your part. This is a solution for very, very large matrices that are NOT SPARSE and thus won't fit in memory.
Note this solves the complaint of the below complainer that it won't fit in memory. However, the OP did say sparse, which implies very few actual datapoints spread out in giant arrays, and Numpy / SciPy handle this natively and thus nicely (lots of people at Fermilab use Numpy / SciPy regularly, I'm confident the sparse matrix code is well tested).
I have this line of code in MATLAB, written by someone else:
c=a.'/b
I need to translate it into Python. a, b, and c are all arrays. The dimensions that I am currently using to test the code are:
a: 18x1,
b: 25x18,
which gives me c with dimensions 1x25.
The arrays are not square, but I would not want the code to fail if they were. Can someone explain exactly what this line is doing (mathematically), and how to do it in Python? (i.e., the equivalent for the built-in mrdivide function in MATLAB if it exists in Python?)
The line
c = a.' / b
computes the solution of the equation c b = aT for c. Numpy does not have an operator that does this directly. Instead you should solve bT cT = a for cT and transpose the result:
c = numpy.linalg.lstsq(b.T, a.T)[0].T
The symbol / is the matrix right division operator in MATLAB, which calls the mrdivide function. From the documentation, matrix right division is related to matrix left division in the following way:
B/A = (A'\B')'
If A is a square matrix, B/A is roughly equal to B*inv(A) (although it's computed in a different, more robust way). Otherwise, x = B/A is the solution in the least squares sense to the under- or over-determined system of equations x*A = B. More detail about the algorithms used for solving the system of equations is given here. Typically packages like LAPACK or BLAS are used under the hood.
The NumPy package for Python contains a routine lstsq for computing the least-squares solution to a system of equations. This routine will likely give you comparable results to using the mrdivide function in MATLAB, but it is unlikely to be exact. Any differences in the underlying algorithms used by each function will likely result in answers that differ slightly from one another (i.e. one may return a value of 1.0, whereas the other may return a value of 0.999). The relative size of this error could end up being larger, depending heavily on the specific system of equations you are solving.
To use lstsq, you may have to adjust your problem slightly. It appears that you want to solve an equation of the form cB = a, where B is 25-by-18, a is 1-by-18, and c is 1-by-25. Applying a transpose to both sides gives you the equation BTcT = aT, which is a more standard form (i.e. Ax = b). The arguments to lstsq should be (in this order) BT (an 18-by-25 array) and aT (an 18-element array). lstsq should return a 25-element array (cT).
Note: while NumPy doesn't make any distinction between a 1-by-N or N-by-1 array, MATLAB certainly does, and will yell at you if you don't use the proper one.
In Matlab, A.' means transposing the A matrix. So mathematically, what is achieved in the code is AT/B.
How to go about implementing matrix division in Python (or any language) (Note: Let's go over a simple division of the form A/B; for your example you would need to do AT first and then AT/B next, and it's pretty easy to do the transpose operation in Python |left-as-an-exercise :)|)
You have a matrix equation
C*B=A (You want to find C as A/B)
RIGHT DIVISION (/) is as follows:
C*(B*BT)=A*BT
You then isolate C by inverting (B*BT)
i.e.,
C = A*BT*(B*BT)' ----- [1]
Therefore, to implement matrix division in Python (or any language), get the following three methods.
Matrix multiplication
Matrix transpose
Matrix inverse
Then apply them iteratively to achieve division as in [1].
Only, you need to do AT/B, therefore your final operation after implementing the three basic methods should be:
AT*BT*(B*BT)'
Note: Don't forget the basic rules of operator precedence :)
You can also approach this using the pseudo-inverse of B then post multiplying that result with A. Try using numpy.linalg.pinv then combine this with matrix multiplication via numpy.dot:
c = numpy.dot(a, numpy.linalg.pinv(b))
[edited] As Suvesh pointed out, i was completely wrong before. however, numpy can still easily do the procedure he gives in his post:
A = numpy.matrix(numpy.random.random((18, 1))) # as noted by others, your dimensions are off
B = numpy.matrix(numpy.random.random((25, 18)))
C = A.T * B.T * (B * B.T).I