I am trying to solve an LP problem represented using sparse matrices in Gurobi / python.
max c′ x, subject to A x = b, L ≤ x ≤ U
where A is a SciPy linked list sparse matrix of size ~10002. Using the code
model = gurobipy.Model()
rows, cols = len(b), len(c)
for j in range(cols):
model.addVar(lb=L[j], ub=U[j], obj=c[j])
model.update()
vars = model.getVars()
S = scipy.sparse.coo_matrix(A)
expr, used = [], []
for i in range(rows):
expr.append(gurobipy.LinExpr())
used.append(False)
for i, j, s in zip(S.row, S.col, S.data):
expr[i] += s*vars[j]
used[i] = True
for i in range(rows):
if used[i]:
model.addConstr(lhs=expr[i], sense=gurobipy.GRB.EQUAL, rhs=b[i])
model.update()
model.ModelSense = -1
model.optimize()
the problem is built and solved in ~1s, which is ~10-100 times slower than the same task in Gurobi / Matlab. Do you have any suggestions for improving the efficiency of the problem definition, or suggestions for avoiding translation to sparse coordinate format?
MATLAB will always be more efficient than scipy when dealing with sparse matrices. However, there are a few things you might try to speed things up.
Gurobi's Python interface takes individual sparse constraints. This means that you want to access your matrix in compressed sparse row format (rather than coordinate format).
Try doing:
S = S.tocsr()
or directly constructing your matrix in compressed sparse row format.
This page indicates that you can access the raw data, indicies, and row pointers from a scipy sparse matrix in CSR format. So you should then be able to iterate over these as follows:
model = gurobipy.Model()
row, cols = len(b), len(c)
x = []
for j in xrange(cols):
x.append(model.addVar(lb=L[j], ub=U[j], obj=c[j])
model.update()
# iterate over the rows of S adding each row into the model
for i in xrange(rows):
start = S.indptr[i]
end = S.indptr[i+1]
variables = [x[j] for j in S.indices[start:end]]
coeff = S.data[start:end]
expr = gurobipy.LinExpr(coeff, variables)
model.addConstr(lhs=expr, sense=gurobipy.GRB.EQUAL, rhs=b[i])
model.update()
model.ModelSense = -1
model.optimize()
Note that I added all the terms at once into the expression using the LinExpr() constructor.
Related
I want to create a matrix with 1 column and n rows, to use in a calculation for a PageRank algorithm. If I make it like this, and then use the matrix in a calculation, it gives this result:
A = [[0.0375,0.4625,0.0375,0.32083333],
[0.0375, 0.0375,0.0375,0.32083333],
[0.8875, 0.0375, 0.0375, 0.32083333],
[0.0375, 0.4625, 0.8875, 0.0375]]
my_dp = 1/4
r = np.matrix([my_dp, my_dp, my_dp, my_dp])
r = np.transpose(r)
print(r)
for i in range(1,50):
r = A*r
print("Final:", print(r))
[[0.25]
[0.25]
[0.25]
[0.25]]
[[0.3570795 ]
[0.19760835]
[0.30663962]
[0.13867253]]
Final: None
But if I create it automatically, using np.empty and .fill, I get this result:
r = np.empty(n)
r.fill(my_dp)
r = r[None].T
print(r)
for i in range(1,50):
r = A*r
print("Final:", print(r))
[[0.25]
[0.25]
[0.25]
[0.25]]
[[3.35329783e-71 3.35329783e-71 7.21422583e-04 9.73677480e-18]
[1.60559016e-25 3.35329783e-71 3.35329783e-71 9.73677480e-18]
[1.60559016e-25 7.21422583e-04 3.35329783e-71 3.35329783e-71]
[1.60559016e-25 3.35329783e-71 3.35329783e-71 3.35329783e-71]]
Final: None
A is an nxn adjacency matrix.
Why is this? As you can see, if I print the matrices, they look identical, and they should be.
I tried creating a new matrix and filling it with .fill, I tried creating a matrix with .full, but everything resulted in the second outcome. The only time it works properly is when I create the matrix manually, which is not very possible, since to continue, I will need to have hundreds of elements in the matrix.
It isn't clearly visible, but there IS a difference between the two examples.
The difference is that the first r is an np.matrix, and the second r is an np.array. One of the few differences between the two is the multiply operator. Using * on a matrix does a matrix multiply. Using * on an array does an element-wise multiply, where r gets broadcast to fit the shape of A.
If you want a matrix multiply, use the # operator:
r = A#r
What are the differences between numpy arrays and matrices? Which one should I use?
I have a scipy sparse csc_matrix J with J.shape = (n, k).
Suppose d is some k-length array with no zeros.
I want to construct a LinearOperator, call it linop, where
from scipy.sparse.linalg import LinearOperator,
J = # (n,k) csc_matrix
d = # some k-array
D = # assume I make a sparse diagonal matrix here of 1/d
linop = LinearOperator((n,k),
matvec=lambda v: J.dot(v/d),
rmatvec=lambda v: D.dot(J.T.dot(v))
)
My question is, under what conditions does this preserve "sparsity"? Not of the result, but of the intermediate steps. (I am unsure in general what happens "under the hood" when you multiply sparse times dense.)
For example, if (v/d) is dense, is J converted to dense before the multiplication? This would be very bad for my use case. Do I need to explicitly convert the input arguments in the lambda methods to sparse before the multiplication?
Thank you in advance.
Edit: pre-computing "J / d" is not an option as I need J later, and don't have the memory to store J and J / d.
In the following function, if I use
np.linalg.inv when Nx, Nt get large the function seems to take forever. In my mind I know I should instead use sparse matrices, which are in scipy (which I've never used before), but I'm getting really stuck how to convert M to a sparse matrix, find its inverse, and then convert it back to a numpy array for the for loop.
If anyone could help I'd be really grateful! Thanks!
def BTCS(phiOld, c, Nx, Nt):
#Initiate phi for the for loop
phi = phiOld.copy()
#Crate the matrix M for the BTCS scheme
M = np.zeros((Nx, Nx))
for i in range(Nx):
M[i,(i-1)%Nx] = -c/2
M[i,i] = 1
M[i,(i+1)%Nx] = c/2
#Take the inverse of M so as to have phi(n+1) = M^(-1) * phi(n)
M_inv = np.linalg.inv(M)
#Loop over all time steps
for it in range(Nt):
#Loop over space (excluding end points)
for ix in range(1,Nx-1):
phi[ix] = M_inv.dot(phiOld)[ix]
#Compute boundary values using periodic boundary conditions
phi[0] = M_inv.dot(phiOld)[0]
phi[Nx-1] = phi[0]
#Update old time value
phiOld = phi.copy()
return phi
I have a number of indices and values that make up a scipy.coo_matrix. The indices/values are generated from different subroutines and are concatenated together before handed over to the matrix constructor:
import numpy
from scipy import sparse
n = 100000
I0 = range(n)
J0 = range(n)
V0 = numpy.random.rand(n)
I1 = range(n)
J1 = range(n)
V1 = numpy.random.rand(n)
# [...]
I = numpy.concatenate([I0, I1])
J = numpy.concatenate([J0, J1])
V = numpy.concatenate([V0, V1])
matrix = sparse.coo_matrix((V, (I, J)), shape=(n, n))
Now, the components of (I, J, V) can be quite large such that the concatenate operations become significant. (In the above example it takes over 20% of the runtime on my machine.) I'm reading that it's not possible to concatenate without a copy.
Is there a way for handing over indices and values without copying the input data around first?
If you look at the code for coo_matrix.__init__ you'll see that it's pretty simple. In fact if the (V, (I,J)) inputs are right it will simply assign those 3 arrays to its .data, row, col attributes. You can even check that after creation by comparing those attributes with your variables.
If they aren't 1d arrays of the right dtype, it will massage them - make the arrays, etc. So without getting into details, processing that you do before hand might save time in the coo call.
self.row = np.array(row, copy=copy, dtype=idx_dtype)
self.col = np.array(col, copy=copy, dtype=idx_dtype)
self.data = np.array(obj, copy=copy)
One way or other those attributes will have to each be a single array, not a loose list of arrays or lists of lists.
sparse.bmat makes a coo matrix from other ones. It collected their coo attributes, joins them in the fill an empty array styles, and calls coo_matrix. Look at its code.
Almost all numpy operations that return a new array do so by allocating an empty and filling it. Letting numpy do that in compiled code (with np.concatentate) should be a be a little faster, but details like the size and number of inputs will make a difference.
A non_connonical coo matrix is just the start. Many operations require a conversion to one of the other formats.
Efficiently construct FEM/FVM matrix
This is about sparse matrix constrution where there are many duplicate points that need to be summed - and using using the csr format for calculations.
You can try pre-allocating the arrays. It'll spare you the copy at least. I didn't see any speedup for the example, but you might see a change.
import numpy
from scipy import sparse
n = 100000
I = np.empty(2*n, np.double)
J = np.empty_like(I)
V = np.empty_like(I)
I[:n] = range(n)
J[:n] = range(n)
V[:n] = numpy.random.rand(n)
I[n:] = range(n)
J[n:] = range(n)
V[n:] = numpy.random.rand(n)
matrix = sparse.coo_matrix((V, (I, J)), shape=(n, n))
i'm currently implementing a small finite element sim. using Python/Numpy, and i am looking for an efficient way to create the global stiffness matrix:
1) I think that the creation of a sparse matrix from smaller element stiffness matrices should be done using coo_matrix(). However, can i extend an existing coo_matrix, or should i create it from the final i,j and v lists?
2) Currently, i am creating the i and j lists from the smaller element stiffness matrix using list comprehensions and concatenating them. Is there a better way to create these lists?
3) Creation of the data vector: Same question, are python lists preferred over numpy vectors due to the easy extension possibilities?
4) Of course i am open for any advices :). Thank You!
Here is a small example of my current plan to do the global assembly to make clear what i intend:
import numpy as np
from scipy.sparse import coo_matrix
#2 nodes, 3 dof per node
locations = [0, 6]
nNodes = 2
dof =3
totSize = nNodes * dof
Ke = np.array([[1,1,1, 2,2,2],
[1,1,1, 2,2,2],
[1,1,1, 2,2,2],
[2,2,2, 3,3,3],
[2,2,2, 3,3,3],
[2,2,2, 3,3,3]])
I = []
J = []
#generate rowwise i and j lists:
i = [ idx + u for i in range(totSize) for idx in locations for u in range(dof) ]
j = [ idx + u for idx in locations for u in range(dof) for i in range(totSize) ]
I += i
J += J
Data = Ke.flatten()
cMatrix = coo_matrix( (Data, (i,j)), )
In this post, I would try to focus on performance issue specific to the creation of lists i, j and finally matrix cMatrix.
Under those loop/list comprehensions, you are basically performing element-wise additions of locations and range(dof). Porting over to NumPy, we could leverage broadcasting there. Finally, to simulate for range(totSize) again in those comprehensions, we could tile the final addition result with np.tile. We will use it as its flattened version for indexing into columns of the sparse matrix and its transposed flattened version for rows.
Thus, the implementation would look something like this -
idx0 = (np.asarray(locations)[:,None] + np.arange(dof)).ravel()
J = np.tile(idx0[:,None],totSize)
cMatrix = coo_matrix( (Data, (J.ravel('F'),J.ravel())), )