I have been using lpsolve in Python for a very long time, with generally good results. I have even written my own Cython wrapper to it to overcome the mess that the original Python wrapper is.
My Cython wrapper is much faster in the setup of the problem, but of course the solution time only depends on the lpsolve C code and has got nothing to do with Python.
I am only solving real-valued linear problems, no MIPs. The latest (and one of the biggest) LP I had to solve had a constraint matrix of size about 5,000 x 3,000, and the setup plus solve takes about 150 ms. The problem is, I have to solve a slightly modified version of the same problem many times in my simulation (constraints, RHS, bounds and so on are time-dependent for a simulation with many timesteps). The constraint matrix is usually very sparse, with about 0.1% - 0.5% of NNZ or less.
Using lpsolve, the setup and solve of the problem is as easy as the following:
import numpy
from lp_solve.lp_solve import PRESOLVE_COLS, PRESOLVE_ROWS, PRESOLVE_LINDEP, PRESOLVE_NONE, SCALE_DYNUPDATE, lpsolve
# The constraint_matrix id a 2D NumPy array and it contains
# both equality and inequality constraints
# And I need it to be single precision floating point, like all
# other NumPy arrays from here onwards
m, n = constraint_matrix.shape
n_le = len(inequality_constraints)
n_e = len(equality_constraints)
# Setup RHS vector
b = numpy.zeros((m, ), dtype=numpy.float32)
# Assign equality and inequality constraints (= and <=)
b[0:n_e] = equality_constraints
b[-n_le:] = inequality_constraints
# Tell lpsolve which rows are equalities and which are inequalities
e = numpy.asarray(['LE']*m)
e[0:-n_le] = 'EQ'
# Make the LP
lp = lpsolve('make_lp', m, n)
# Some options for scaling the problem
lpsolve('set_scaling', lp, SCALE_DYNUPDATE)
lpsolve('set_verbose', lp, 'IMPORTANT')
# Use presolve as it is much faster
lpsolve('set_presolve', lp, PRESOLVE_COLS | PRESOLVE_ROWS | PRESOLVE_LINDEP)
# I only care about maximization
lpsolve('set_sense', lp, True)
# Set the objective function of the problem
lpsolve('set_obj_fn', lp, objective_function)
lpsolve('set_mat', lp, constraint_matrix)
# Tell lpsolve about the RHS
lpsolve('set_rh_vec', lp, b)
# Set the constraint type (equality or inequality)
lpsolve('set_constr_type', lp, e)
# Set upper bounds for variables - lower bounds are automatically 0
lpsolve('set_upbo', lp, ub_values)
# Solve the problem
out = lpsolve('solve', lp)
# Retrieve the solution for all the variables
vars_sol = numpy.asarray([lpsolve('get_var_primalresult', lp, i) for i in xrange(m + 1, m + n + 1)], dtype=numpy.float32)
# Delete the problem, timestep done
lpsolve('delete_lp', lp)
For reasons that are too long to explain, my NumPy arrays are all single precision floating point arrays, and I'd like them to stay that way.
Now, after all this painful introduction, I would like to ask: does anyone know of another library (with reasonable Python wrappers) that allows me to setup and solve a problem of this size as fast (or potentially faster) than lpsolve? Most of the libraries I have looked at (PuLP, CyLP, PyGLPK and so on) do not seem to have a straightforward way to say "this is my entire constraint matrix, set it in one go". They mostly seem to be oriented towards being "modelling languages" that allow fancy things like this (CyLP example):
# Add variables
x = s.addVariable('x', 3)
y = s.addVariable('y', 2)
# Create coefficients and bounds
A = np.matrix([[1., 2., 0],[1., 0, 1.]])
B = np.matrix([[1., 0, 0], [0, 0, 1.]])
D = np.matrix([[1., 2.],[0, 1]])
a = CyLPArray([5, 2.5])
b = CyLPArray([4.2, 3])
x_u= CyLPArray([2., 3.5])
# Add constraints
s += A * x <= a
s += 2 <= B * x + D * y <= b
s += y >= 0
s += 1.1 <= x[1:3] <= x_u
I honestly don't care about the flexibility, I just need raw speed in the problem setup and solve. Creating NumPy matrices plus doing all those fancy operations above is definitely going to be a performance killer.
I'd rather stay on Open Source solvers if possible, but any suggestion is most welcome. My apologies for the long post.
Andrea.
#infinity77,
I am looking at using lpsolve now. It is fairly straight forward to generate a .lp file for input and it did solve several smaller problems. BUT... I am now attempting to solve a node coloring problem. This is a MIP. My first attempt at a 500 node problem ran about 15 minutes as an lp relaxation. lpsolve has been grinding on the true MIP since last night. Still grinding. These coloring problems are difficult but 14 hours with no end in sight is tooooo much.
The best alternative I am aware of is from coin-or.org;[Coin-OR Solvers] 1 Try their clp and cbc solvers depending on which type of problem you are solving. Benchmarks I have seen say they are the best choice outside of CPLEX and Gurobi. It is free but you will need to be sure it is "legal" for your purposes. A good benchmark paper by Bernhard Meindl and Matthias Templ at Open Source Solver Benchmarks
Related
I'm dealing with a mathematical optimization problem, in more detail it is a semi-definite program (see code-snipped below), which is used to solve another problem iteratively.
It is required that the equality-constraints are met up to ~10^(-10) or better. Even if I start my optimization with a matrix M that meets the constraints up to 10^(-12) or better, the optimization result X doesn't meet the requirements for X+M very close (at least two or three of them are only met up to 10**(-7)).
Is there a way to improve the accuracy of how close cvx (mosek) meets the constraints?
Sidenote: I got the initial value of my optimization as solution of exactly the same problem, so it seems to be possible to yield a higher accuracy, but I guess this was only lucky. Unfortunatly, this matrix isn't close to minimum, so I need to do another iteration.
# defining variable
X = cp.Variable((m,m), hermitian=True)
#pos. semi-definite constraint
constraints = [M+X >> 0]
# all the other constraints
for i in range(0,len(b)):
constraints += [ cp.trace(A[i]#(M+X)) == b[i]]
#problem formulation
prob = cp.Problem(cp.Minimize(cp.real(cp.trace(C#X))), constraints)
Result = prob.solve(solver=cp.MOSEK, verbose = False, parallel = True)
Here M and C are known matrices, A and b is a list of matrices and scalars respectively.
I've already tried to find an answer in the documentation and on the internet, but I couldn't find a solution. Therefore, I'd be grateful for any help!
Thank's in advance!
I have a very large multiply and sum operation that I need to implement as efficiently as possible. The best method I've found so far is bsxfun in MATLAB, where I formulate the problem as:
L = 10000;
x = rand(4,1,L+1);
A_k = rand(4,4,L);
tic
for k = 2:L
i = 2:k;
x(:,1,k+1) = x(:,1,k+1)+sum(sum(bsxfun(#times,A_k(:,:,2:k),x(:,1,k+1-i)),2),3);
end
toc
Note that L will be larger in practice. Is there a faster method? It's strange that I need to first add the singleton dimension to x and then sum over it, but I can't get it to work otherwise.
It's still much faster than any other method I've tried, but not enough for our application. I've heard rumors that the Python function numpy.einsum may be more efficient, but I wanted to ask here first before I consider porting my code.
I'm using MATLAB R2017b.
I believe both of your summations can be removed, but I only removed the easier one for the time being. The summation over the second dimension is trivial, since it only affects the A_k array:
B_k = sum(A_k,2);
for k = 2:L
i = 2:k;
x(:,1,k+1) = x(:,1,k+1) + sum(bsxfun(#times,B_k(:,1,2:k),x(:,1,k+1-i)),3);
end
With this single change the runtime is reduced from ~8 seconds to ~2.5 seconds on my laptop.
The second summation could also be removed, by transforming times+sum into a matrix-vector product. It needs some singleton fiddling to get the dimensions right, but if you define an auxiliary array that is B_k with the second dimension reversed, you can generate the remaining sum as ~x*C_k with this auxiliary array C_k, give or take a few calls to reshape.
So after a closer look I realized that my original assessment was overly optimistic: you have multiplications in both dimensions in your remaining term, so it's not a simple matrix product. Anyway, we can rewrite that term to be the diagonal of a matrix product. This implies that we're computing a bunch of unnecessary matrix elements, but this still seems to be slightly faster than the bsxfun approach, and we can get rid of your pesky singleton dimension too:
L = 10000;
x = rand(4,L+1);
A_k = rand(4,4,L);
B_k = squeeze(sum(A_k,2)).';
tic
for k = 2:L
ii = 1:k-1;
x(:,k+1) = x(:,k+1) + diag(x(:,ii)*B_k(k+1-ii,:));
end
toc
This runs in ~2.2 seconds on my laptop, somewhat faster than the ~2.5 seconds obtained previously.
Since you're using an new version of Matlab you might try broadcasting / implicit expansion instead of bsxfun:
x(:,1,k+1) = x(:,1,k+1)+sum(sum(A_k(:,:,2:k).*x(:,1,k-1:-1:1),3),2);
I also changed the order of summation and removed the i variable for further improvement. On my machine, and with Matlab R2017b, this was about 25% faster for L = 10000.
I need to solve a large number of 3x3 symmetric, postive-definite systems with Python. So far, I did
res = numpy.zeros(n)
for k, obj in enumerate(data_array):
# construct A, rhs, idx from obj
res[idx] += numpy.linalg.solve(A, rhs)
This produces the correct result, however is also quite slow if n is large. (Well... Yeah.) Perhaps 3x3 isn't a problem size where calling solve() makes much sense.
Any hints?
In NumPy 1.8 and later, numpy.linalg.solve actually broadcasts. For numpy.linalg.solve(a, b), if b.ndim == a.ndim - 1, it will perform a broadcasted matrix-vector solve; otherwise, it'll do a broadcasted matrix-matrix solve. (This decision criterion isn't documented; I had to look at the source.)
If you can efficiently construct a stack of As and rhss, you can call solve once and avoid a Python loop.
I have a large number of small linear equation systems that I'd like to solve efficiently using numpy. Basically, given A[:,:,:] and b[:,:], I wish to find x[:,:] given by A[i,:,:].dot(x[i,:]) = b[i,:]. So if I didn't care about speed, I could solve this as
for i in range(n):
x[i,:] = np.linalg.solve(A[i,:,:],b[i,:])
But since this involved explicit looping in python, and since A typically has a shape like (1000000,3,3), such a solution would be quite slow. If numpy isn't up to this, I could do this loop in fortran (i.e. using f2py), but I'd prefer to stay in python if possible.
For those coming back to read this question now, I thought I'd save others time and mention that numpy handles this using broadcasting now.
So, in numpy 1.8.0 and higher, the following can be used to solve N linear equations.
x = np.linalg.solve(A,b)
I guess answering yourself is a bit of a faux pas, but this is the fortran solution I have a the moment, i.e. what the other solutions are effectively competing against, both in speed and brevity.
function pixsolve(A, b) result(x)
implicit none
real*8 :: A(:,:,:), b(:,:), x(size(b,1),size(b,2))
integer*4 :: i, n, m, piv(size(b,1)), err
n = size(A,3); m = size(A,1)
x = b
do i = 1, n
call dgesv(m, 1, A(:,:,i), m, piv, x(:,i), m, err)
end do
end function
This would be compiled as:
f2py -c -m foo{,.f90} -llapack -lblas
And called from python as
x = foo.pixsolve(A.T, b.T).T
(The .Ts are needed due to a poor design choice in f2py, which both causes unnecessary copying, inefficient memory access patterns and unnatural looking fortran indexing if the .Ts are left out.)
This also avoids a setup.py etc. I have no bone to pick with fortran (as long as strings aren't involved), but I was hoping that numpy might have something short and elegant which could do the same thing.
I think you're wrong about the explicit looping being a problem. Usually it's only the innermost loop it's worth optimizing, and I think that holds true here. For example, we can measure the code of the overhead vs the cost of the actual computation:
import numpy as np
n = 10**6
A = np.random.random(size=(n, 3, 3))
b = np.random.random(size=(n, 3))
x = b*0
def f():
for i in xrange(n):
x[i,:] = np.linalg.solve(A[i,:,:],b[i,:])
np.linalg.pseudosolve = lambda a,b: b
def g():
for i in xrange(n):
x[i,:] = np.linalg.pseudosolve(A[i,:,:],b[i,:])
which gives me
In [66]: time f()
CPU times: user 54.83 s, sys: 0.12 s, total: 54.94 s
Wall time: 55.62 s
In [67]: time g()
CPU times: user 5.37 s, sys: 0.01 s, total: 5.38 s
Wall time: 5.40 s
IOW, it's only spending 10% of its time doing anything other than actually solving your problem. Now, I could totally believe that np.linalg.solve itself is too slow for you relative to what you could get out of Fortran, and so you want to do something else. That's probably especially true on small problems, come to think of it: IIRC I once found it faster to unroll certain small solutions by hand, although that was a while back.
But by itself, it's not true that using an explicit loop on the first index will make the overall solution quite slow. If np.linalg.solve is fast enough, the loop won't change it much here.
I think you can do it in one go, with a (3x100000,3x100000) matrix composed of 3x3 blocks around the diagonal.
Not tested :
b_new = np.vstack([ b[i,:] for i in range(len(i)) ])
x_new = np.zeros(shape=(3x10000,3) )
A_new = np.zeros(shape=(3x10000,3x10000) )
n,m = A.shape
for i in range(n):
A_new[3*i:3*(i+1),3*i:3*(i+1)] = A[i,:,:]
x = np.linalg.solve(A_new,b_new)
I have this line of code in MATLAB, written by someone else:
c=a.'/b
I need to translate it into Python. a, b, and c are all arrays. The dimensions that I am currently using to test the code are:
a: 18x1,
b: 25x18,
which gives me c with dimensions 1x25.
The arrays are not square, but I would not want the code to fail if they were. Can someone explain exactly what this line is doing (mathematically), and how to do it in Python? (i.e., the equivalent for the built-in mrdivide function in MATLAB if it exists in Python?)
The line
c = a.' / b
computes the solution of the equation c b = aT for c. Numpy does not have an operator that does this directly. Instead you should solve bT cT = a for cT and transpose the result:
c = numpy.linalg.lstsq(b.T, a.T)[0].T
The symbol / is the matrix right division operator in MATLAB, which calls the mrdivide function. From the documentation, matrix right division is related to matrix left division in the following way:
B/A = (A'\B')'
If A is a square matrix, B/A is roughly equal to B*inv(A) (although it's computed in a different, more robust way). Otherwise, x = B/A is the solution in the least squares sense to the under- or over-determined system of equations x*A = B. More detail about the algorithms used for solving the system of equations is given here. Typically packages like LAPACK or BLAS are used under the hood.
The NumPy package for Python contains a routine lstsq for computing the least-squares solution to a system of equations. This routine will likely give you comparable results to using the mrdivide function in MATLAB, but it is unlikely to be exact. Any differences in the underlying algorithms used by each function will likely result in answers that differ slightly from one another (i.e. one may return a value of 1.0, whereas the other may return a value of 0.999). The relative size of this error could end up being larger, depending heavily on the specific system of equations you are solving.
To use lstsq, you may have to adjust your problem slightly. It appears that you want to solve an equation of the form cB = a, where B is 25-by-18, a is 1-by-18, and c is 1-by-25. Applying a transpose to both sides gives you the equation BTcT = aT, which is a more standard form (i.e. Ax = b). The arguments to lstsq should be (in this order) BT (an 18-by-25 array) and aT (an 18-element array). lstsq should return a 25-element array (cT).
Note: while NumPy doesn't make any distinction between a 1-by-N or N-by-1 array, MATLAB certainly does, and will yell at you if you don't use the proper one.
In Matlab, A.' means transposing the A matrix. So mathematically, what is achieved in the code is AT/B.
How to go about implementing matrix division in Python (or any language) (Note: Let's go over a simple division of the form A/B; for your example you would need to do AT first and then AT/B next, and it's pretty easy to do the transpose operation in Python |left-as-an-exercise :)|)
You have a matrix equation
C*B=A (You want to find C as A/B)
RIGHT DIVISION (/) is as follows:
C*(B*BT)=A*BT
You then isolate C by inverting (B*BT)
i.e.,
C = A*BT*(B*BT)' ----- [1]
Therefore, to implement matrix division in Python (or any language), get the following three methods.
Matrix multiplication
Matrix transpose
Matrix inverse
Then apply them iteratively to achieve division as in [1].
Only, you need to do AT/B, therefore your final operation after implementing the three basic methods should be:
AT*BT*(B*BT)'
Note: Don't forget the basic rules of operator precedence :)
You can also approach this using the pseudo-inverse of B then post multiplying that result with A. Try using numpy.linalg.pinv then combine this with matrix multiplication via numpy.dot:
c = numpy.dot(a, numpy.linalg.pinv(b))
[edited] As Suvesh pointed out, i was completely wrong before. however, numpy can still easily do the procedure he gives in his post:
A = numpy.matrix(numpy.random.random((18, 1))) # as noted by others, your dimensions are off
B = numpy.matrix(numpy.random.random((25, 18)))
C = A.T * B.T * (B * B.T).I