Develop Reference

PYOMO Constraints - setting constraints over indexed variables

I have been trying to get into python optimization, and I have found that pyomo is probably the way to go; I had some experience with GUROBI as a student, but of course that is no longer possible, so I have to look into the open source options.
I basically want to perform an non-linear mixed integer problem in which I will minimized a certain ratio. The problem itself is setting up a power purchase agreement (PPA) in a renewable energy scenario. Depending on the electricity generated, you will have to either buy or sell electricity acording to the PPA.
The only starting data is the generation; the PPA is the main decision variable, but I will need others. "buy", "sell", "b1" and "b2" are unknown without the PPA value. These are the equations:
Equations that rule the problem (by hand).
Using pyomo, I was trying to set up the problem as:
# Dataframe with my Generation information:
January = Data['Full_Data'][(Data['Full_Data']['Month'] == 1) & (Data['Full_Data']['Year'] == 2011)]
Gen = January['Producible (MWh)']
Time = len(Generacion)
M=100
# Model variables and definition:
m = ConcreteModel()
m.IDX = range(time)
m.PPA = Var(initialize = 2.0, bounds =(1,7))
m.compra = Var(m.IDX, bounds = (0, None))
m.venta = Var(m.IDX, bounds = (0, None))
m.b1 = Var(m.IDX, within = Binary)
m.b2 = Var(m.IDX, within = Binary)
And then, the constraint; only the first one, as I was already getting errors:
m.b1_rule = Constraint(
expr = (((Gen[i] - PPA)/M for i in m.IDX) <= m.b1[i])
)
which gives me the error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-5d5f5584ebca> in <module>
1 m.b1_rule = Constraint(
----> 2 expr = (((Generacion[i] - PPA)/M for i in m.IDX) <= m.b1[i])
3 )
pyomo\core\expr\numvalue.pyx in pyomo.core.expr.numvalue.NumericValue.__ge__()
pyomo\core\expr\logical_expr.pyx in pyomo.core.expr.logical_expr._generate_relational_expression()
AttributeError: 'generator' object has no attribute 'is_expression_type'
I honestly have no idea what this means. I feel like this should be a simple problem, but I am strugling with the syntax. I basically have to apply a constraint to each individual data from "Generation", there is no sum involved; all constraints are 1-to-1 contraints set so that the physical energy requirements make sense.
How do I set up the constraints like this?
Thank you very much

You have a couple things to fix. First, the error you are getting is because you have "extra parenthesis" around an expression that python is trying to convert to a generator. So, step 1 is to remove the outer parenthesis, but that will not solve your issue.
You said you want to generate this constraint "for each" value of your index. Any time you want to generate copies of a constraint "for each" you will need to either do that by making a constraint list and adding to it with some kind of loop, or use a function-rule combination. There are examples of each in the pyomo documentation and plenty on this site (I have posted a ton if you look at some of my posts.) I would suggest the function-rule combo and you should end up with something like:
def my_constr(m, i):
return m.Gen[i] - m.PPA <= m.b1[i] * M
m.C1 = Constraint(m.IDX, rule=my_constr)

GSVD for python Generalized Singular Value Decomposition

MATLAB has a gsvd function to perform the generalised SVD. Since 2013 I think there has been a lot of discussion on the github pages regarding putting it in scipy and some pages have code that I can use such as here which is super complicated for a novice like me(to get it running).
I also found LJWilliams github page with an implementation. This is of no good as has lot of bugs when transferred to python 3. Attempted correcting the simple ones such as assert and print. It quickly gets complicated.
Can someone help me with a gsvd code for python or show me how to use the ones that are online?
Also, This is what I get with the LJWilliams implementation, once the print and assert statements are corrected. The code looks complicated and I am not sure spending time on it is the best thing to do! Also some people have reported issues on the same github page which I am not sure are fixed or connected.
n = 10
m = 6
p = 6
A = np.random.rand(m,n)
B = np.random.rand(p,n)
gsvd(A,B)
File "/home/eghx/agent18/master_thesis/AMfe/amfe/gsvd.py", line 260,
in gsvd
U, V, Z, C, S = csd(Q[0:m,:],Q[m:m+n,:])
File "/home/eghx/agent18/master_thesis/AMfe/amfe/gsvd.py", line 107,
in csd
Q,R = scipy.linalg.qr(S[q:n,m:p])
File
"/home/eghx/anaconda3/lib/python3.5/site-packages/scipy/linalg/decomp_qr.py",
line 141, in qr
overwrite_a=overwrite_a)
File
"/home/eghx/anaconda3/lib/python3.5/site-packages/scipy/linalg/decomp_qr.py",
line 19, in safecall
ret = f(*args, **kwargs)
ValueError: failed to create intent(cache|hide)|optional array-- must
have defined dimensions but got (0,)

If you want to work from the LJWillams implementation on github, there are a couple of bugs. However, to understand the technique fully, I'd probably recommend having a go at implementing it yourself. I looked up what Octave (MATLAB free software equivalent) do and their "code is a wrapper to the corresponding Lapack dggsvd and zggsvd routines.", which is what scipy should do IMHO.
I'll post up the bugs I found, but I'm not going to post the code in full working order, because I'm not sure how that stands with regard to copyright, given the copyrighted MATLAB implementation from which it is translated.
Caveat : I am not an expert on the Generalised SVD and have approached this only from the perspective of debugging, not whether the underlying algorithm is correct. I have had this working on your original random arrays and the test case already present in the Python file.
Bugs
Setting k
Around line 63, the conditions for setting k and a misunderstanding of numpy.argparse (particularly in comparison to MATLAB's find) seem to set k wrong in some circumstances. Change that code to
if q == 1:
k = 0
elif m < p:
k = n;
else:
k = max([0,sum((np.diag(C) <= 1/np.sqrt(2)))])
line 79
S[1,1] should be S[0,0], I think (Python 0-indexed arrays)
lines 83 onwards
The numpy matrix slicing around here seems wrong. I got the code working by changing lines 83-95 to read:
UT, ST, VT = scipy.linalg.svd(slice_matrix(S,i,j))
ST = add_zeros(ST,np.zeros([n-k,r-k]))
if k > 0:
print('Zeroing elements of S in row indices > r, to be replaced by ST')
S[0:k,k:r] = 0
S[k:n,k:r] = ST
C[:,j] = np.dot(C[:,j],VT)
V[:,i] = np.dot(V[:,i],UT)
Z[:,j] = np.dot(Z[:,j],VT)
i = np.arange(k,q)
Q,R = scipy.linalg.qr(C[k:q,k:r])
C[i,j] = np.diag(diagf(R))
U[:,k:q] = np.dot(U[:,k:q],Q)
in diagp()
There are two matrix multiplications using X*Y that should be np.dot(X,Y) instead (note * is element-wise multiplication in numpy, not matrix multiplication.)

How to prevent infeasible error with pulp and python?

I have an optimization problem and I write a python program to solve it. I used Pulp with the CPLEX solver:
import pulp
prob = LpProblem("myProblem", LpMinimize)
x = pulp.LpVariable.dicts("p", range( K ), 0, 1, pulp.LpContinuous)
prob += pulp.lpSum( x[k] for k in range( K ) )
...
# Rest of the constraints
status = prob.solve( pulp.CPLEX( msg = 0 ) )
I get the error:
File "C:\Anaconda\lib\site-packages\pulp\solvers.py", line 468, in readsol
raise PulpSolverError, "Unknown status returned by CPLEX: "+statusString
pulp.solvers.PulpSolverError: Unknown status returned by CPLEX: infeasible
My question is : How can I test if the problem is infeasible or not? I want to prevent this event like if problem is infeasible then return 0.
I tried :
if prob.status == 'infeasible':
...
and I tried
if pulp.LpStatusInfeasible == 'infeasible':
...

Is your 'problem' finding whether a given problem instance is feasible or not, or are you actually interested in the solution if it is feasible. Rather than just trap the error when the model is infeasible, I would examine your problem and try to add some slack variables and penalty costs to give you some more information when the problem would otherwise be infeasible.
So rather than add a hard constraint like
sum(x) <= K
you could try something like
sum(x) <= K + penaltyVar
and add a term into your objective like 1000000 * penaltyVar so that the solver really doesn't want to use that penalty variable as non-zero.
Adding these slack/penalty variables in various places in your model can help identify where the constraints are too tight and making your model infeasible.
Don't just ignore the answer above though, as it is still valuable to trap the error.

I think you can solve this by caging the statement inside a try-exception clause.
for example:
# ...
try:
status = prob.solve(pulp.CPLEX(msg = 0))
except PulpSolverError:
# infeasible
return 0
return status

Scipy Sparse Eigensolver: MemoryError after multiple passes through loop without anything new being written during loop

I'm using Python + Scipy to diagonalize sparse matrices with random entries on the diagonal; in particular, I need eigenvalues in the middle of the spectrum. The code I've written has worked fine for months, but now I'm looking at bigger matrices and am running into "MemoryError"s. What's confusing/driving me insane is that the error only shows up after a few iterations (namely 9) of constructing a random matrix and diagonalizing it, but I don't see any way in which my code stores anything extra in memory from one iteration to the next, and so can't see how my code could fail during the 9th iteration but not the 1st.
Here are the details (and I apologize in advance if I've left anything out, I'm new to posting on this site):
Each matrix I construct is 16000x16000, with 15x16000 non-zero entries. Everything ran fine when I was looking at 4000x4000-size matrices. The bulk of my code is
#Initialization
#...
for i in range(dim):
for n in range(N):
digit = (i % 2**(n+1)) / 2**n
index = (i % 2**n) + ((digit + 1) % 2)*(2**n) + (i / 2**(n+1))*(2**(n+1))
row[dim + N*i + n] = index
col[dim + N*i + n] = i
dat[dim + N*i + n] = -G
e_list = open(e_list_name + "_%03dk_%010ds" % (num_states, int(start_time)), "w")
e_log = open(e_log_name + "_%03dk_%010ds" % (num_states, int(start_time)), "w")
for t in range(num_itr): #Begin iterations
dat[0:dim] = math.sqrt(N/2.0)*np.random.randn(dim) #Get new diagonal elements
H = sparse.csr_matrix((dat, (row, col))) #Construct new matrix
vals = sparse.linalg.eigsh(H, k = num_states + 2, sigma = target_energy, which = 'LM', return_eigenvectors = False) #Get new eigenvalues
vals = np.sort(vals)
vals.tofile(e_list)
e_log.write("Iter %d complete\n" % (t+1))
e_list.flush()
e_log.flush()
e_list.close()
e_log.close()
I've been setting num_itr to 100. During the 9th pass through the num_itr loop (as indicated by 8 lines having been written to e_log), the program crashes with the error message
Can't expand MemType 0: jcol 7438
Traceback (most recent call last):
File "/usr/lusers/clb37/QREM_Energy_Gatherer.py", line 55, in <module>
vals = sparse.linalg.eigsh(H, k = num_states + 2, sigma = target_energy, which = 'LM', return_eigenvectors = False)
File "/usr/lusers/clb37/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 1524, in eigsh
symmetric=True, tol=tol)
File "/usr/lusers/clb37/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 1030, in get_OPinv_matvec
return SpLuInv(A.tocsc()).matvec
File "/usr/lusers/clb37/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 898, in __init__
self.M_lu = splu(M)
File "/usr/lusers/clb37/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 242, in splu
ilu=False, options=_options)
MemoryError
Sure enough, the program will fail during the 9th pass through that loop every time I run it on my machine, and when I try running this code on machines with more memory the program makes it through more iterations before crashing, so it looks like the computer really is running out of memory. If that's all there is to it then fine, but what I can't understand is why the program doesn't crash during the 1st iteration. I don't see any point in the 8 lines of the num_itr loop at which something gets written to memory without just being overwritten during the following iteration. I've used Heapy's heap() function to look at my memory usage, and it just prints out "Total size = 11715240 bytes" during every pass.
I feel like there's something fundamental that I just don't know about going on here, either some bug in my writing that I don't know to look for or some detail about how memory is handled. Can anyone explain to me why this code fails during the 9th pass through the num_itr loop but not the 1st?

Ok, this seems to be reproducible on Scipy 0.14.0.
It can apparently be worked around the issue by adding
import gc; gc.collect()
inside the loop to force Pythons cyclic garbage collector to run.
The issue appears that somewhere inside scipy.sparse.eigh there is a cyclic reference loop, in the vein of:
class Foo(object):
pass
a = Foo()
b = Foo()
a.spam = b
b.spam = a
del a, b # <- but a, b still refer to each other and are not dead
This is still perfectly OK in principle: although Python's reference counting doesn't detect such cyclic garbage, a collection is run periodically to gather such objects. However, if each object is very large in memory (eg. big Numpy arrays) the periodic runs are too infrequent, and you run out of memory before the next cyclic garbage collection run is done.
So a workaround is to force the GC to run when you know there's big garbage to collect.
A better workaround would be to change scipy.sparse.eigh so that such cyclic garbage is not generated in the first place.

printSoln module problem

Hi I found in book:Numerical Methods in engineering with Python the module run_kut5, but for that module I need module printSoln, all provided in the book. Now I cp the code, made necessary line adjustments and so. The code looks like:
# -*- coding: cp1250 -*-
## module printSoln
''' printSoln(X,Y,freq).
Prints X and Y returned from the differential
equation solvers using printput frequency ’freq’.
freq = n prints every nth step.
freq = 0 prints initial and final values only.
'''
def printSoln(X,Y,freq):
def printHead(n):
print "\n x ",
for i in range (n):
print " y[",i,"] ",
print
def printLine(x,y,n):
print "%13.4e"% x,f
for i in range (n):
print "%13.4e"% y[i],
print
m = len(Y)
try: n = len(Y[0])
except TypeError: n = 1
if freq == 0: freq = m
printHead(n)
for i in range(0,m,freq):
printLine(X[i],Y[i],n)
if i != m - 1: printLine(X[m - 1],Y[m - 1],n)
Now, when I run the program it says:
line 24, in <module>
m = len(Y)
NameError: name 'Y' is not defined
But I cp'd from the book :\ So now when I call the run_kut module I get the same error, no Y defined in printSoln...
I'm trying to figure this out but I suck :(
Help, please...

I would guess it's a tabs/spaces problem - check that you don't have mixed tabs and spaces in the indentation.
EDIT: If it's not indentation, and since your error message contains "<module>" and not a filename, I'm guessing you're pasting this into an interactive interpreter.
Instead, you should paste the code into a file called printsoln.py, and then run this in the interactive interpreter:
from printsoln import printSoln
If you still want to paste it all in the interpreter then you'll probably need to remove the blank lines - those after def printSoln and after each internal function. The interactive interpreter uses blank lines to figure out when you're done with a multi-line definition, and tries to evaluate m = len(Y) outside the context of function printSoln. In this context, variable Y doesn't exist.