Related
I have been trying to get into python optimization, and I have found that pyomo is probably the way to go; I had some experience with GUROBI as a student, but of course that is no longer possible, so I have to look into the open source options.
I basically want to perform an non-linear mixed integer problem in which I will minimized a certain ratio. The problem itself is setting up a power purchase agreement (PPA) in a renewable energy scenario. Depending on the electricity generated, you will have to either buy or sell electricity acording to the PPA.
The only starting data is the generation; the PPA is the main decision variable, but I will need others. "buy", "sell", "b1" and "b2" are unknown without the PPA value. These are the equations:
Equations that rule the problem (by hand).
Using pyomo, I was trying to set up the problem as:
# Dataframe with my Generation information:
January = Data['Full_Data'][(Data['Full_Data']['Month'] == 1) & (Data['Full_Data']['Year'] == 2011)]
Gen = January['Producible (MWh)']
Time = len(Generacion)
M=100
# Model variables and definition:
m = ConcreteModel()
m.IDX = range(time)
m.PPA = Var(initialize = 2.0, bounds =(1,7))
m.compra = Var(m.IDX, bounds = (0, None))
m.venta = Var(m.IDX, bounds = (0, None))
m.b1 = Var(m.IDX, within = Binary)
m.b2 = Var(m.IDX, within = Binary)
And then, the constraint; only the first one, as I was already getting errors:
m.b1_rule = Constraint(
expr = (((Gen[i] - PPA)/M for i in m.IDX) <= m.b1[i])
)
which gives me the error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-5d5f5584ebca> in <module>
1 m.b1_rule = Constraint(
----> 2 expr = (((Generacion[i] - PPA)/M for i in m.IDX) <= m.b1[i])
3 )
pyomo\core\expr\numvalue.pyx in pyomo.core.expr.numvalue.NumericValue.__ge__()
pyomo\core\expr\logical_expr.pyx in pyomo.core.expr.logical_expr._generate_relational_expression()
AttributeError: 'generator' object has no attribute 'is_expression_type'
I honestly have no idea what this means. I feel like this should be a simple problem, but I am strugling with the syntax. I basically have to apply a constraint to each individual data from "Generation", there is no sum involved; all constraints are 1-to-1 contraints set so that the physical energy requirements make sense.
How do I set up the constraints like this?
Thank you very much
You have a couple things to fix. First, the error you are getting is because you have "extra parenthesis" around an expression that python is trying to convert to a generator. So, step 1 is to remove the outer parenthesis, but that will not solve your issue.
You said you want to generate this constraint "for each" value of your index. Any time you want to generate copies of a constraint "for each" you will need to either do that by making a constraint list and adding to it with some kind of loop, or use a function-rule combination. There are examples of each in the pyomo documentation and plenty on this site (I have posted a ton if you look at some of my posts.) I would suggest the function-rule combo and you should end up with something like:
def my_constr(m, i):
return m.Gen[i] - m.PPA <= m.b1[i] * M
m.C1 = Constraint(m.IDX, rule=my_constr)
I have been struggling with solving the following problem:
I have some values stored in variable returns, and I want to multiply them by a weight (x_vars)in order to make the sum of the products and obtain the value of the variable target_return. As the following equation:
I am struggling with the answer returned, as it saying x_13 = 1. But is not correct. I have tried to do it with normal constrains and eslastic but I get no correct answer. I have a tolerance of let's say 0.01% of error.
Any guess?
Thanks
import pulp as plp
# Name model
# set variables
target_return = 0.0204185791833761
returns = [0.025865338474480914,
0.031617753987556174,
0.017530329404997325,
-0.0008543358059154293,
0.010510143115372461,
0.012048338516174262,
0.04959181591738604,
0.06059545751936519,
0.05926623356137273,
0.058971753714631814,
0.03442022388647947,
0.013688974565667422,
0.02104075216985901,
-0.0021299262578251543,
0.017882182143459602,
0.018964184657020766,
0.005792320704504306,
0.018823216764509265,
-0.0015860431556348198,
0.008352716776521163,
0.030728674721250515,
0.016529301951210496,
0.0184734317514465,
-0.0008822232596910062,
0.010912806711330658,
0.023530497410194418,
0.0378090116601979,
0.009456335242604919,
0.005556382185357922,
0.020013334218681678,
0.05852489326632937,
0.047988175193893645,
0.016134386609760742,
0.014350880108888964,
0.006756782462879585]
# Initialize model
prob=plp.LpProblem("Find Weights Model", plp.LpMinimize)
x_vars=plp.LpVariable.dicts("x", range(0, len(returns)), 0, 1)
for index in range(len(returns)):
x_vars[index].setInitialValue(0.5)
#Set problem
prob += plp.lpSum([x_vars[i] for i in x_vars]) == 1
#Set constrain
constrain=plp.LpConstraint(plp.lpSum(
[returns[i]*x_vars[i] for i in range(0, len(returns))]), rhs=target_return)
# To use standard constrain uncomment following line
#prob.addConstraint(constrain)
# To use elastic constrain uncomment following line
elastic_constrain=constrain.makeElasticSubProblem(penalty = 500000,proportionFreeBoundList =[0.001,0.0001])
prob.extend(elastic_constrain)
prob.solve(plp.PULP_CBC_CMD(msg=True, warmStart=True))
final_weights={}
for v in prob.variables():
final_weights[v.name]=v.varValue
print("Status:", plp.LpStatus[prob.status])
final_weights
Possible solutions:
solution1 = [0.00962135141677065,
0.00962135141677065,
0.0277264834687149,
0.0297957188693739,
0.0324961558793471,
0.030826881062523,
0.0306009427392119,
0.0250863577897657,
0.0234700849350161,
0.0236653284777789,
0.0237085827820698,
0.0273148420035226,
0.0303599573647406,
0.0292800893464517,
0.0326835213770196,
0.0297440370126145,
0.0295851066643113,
0.0315198597719625,
0.0296058126214705,
0.0326036329340179,
0.0311437752841399,
0.0278570759221869,
0.0299427552142207,
0.0296571910611703,
0.0325002521378998,
0.0307677356659276,
0.0289143823239777,
0.0268170787933395,
0.0309816698637205,
0.0315545156635921,
0.0294310019878726,
0.023774220455596,
0.0253219090166822,
0.0300007623180508,
0.0302627332639599]
solution2 = [0.0291039150753347,
0.0291039150753347,
0.028739103831572,
0.0296325128619877,
0.0307984495156388,
0.0300777243941537,
0.0299801757193628,
0.027599206714805,
8.34695965451632E-05,
0.0269856659961344,
0.0270043421537945,
0.0285613748092319,
0.0298761263170002,
0.0294098868961038,
0.0308793458966854,
0.0296102001652331,
0.0295415791679557,
0.0303769265588311,
0.0295505206943934,
0.0308448536374279,
0.0302145485146196,
0.0287954885757358,
0.0296959977222549,
0.0295727013961937,
0.030800217918514,
0.0300521885203749,
0.0292519883685474,
0.0283464607865791,
0.0301445556616913,
0.030391887092054,
0.0294750434895131,
0.0270326804365011,
0.0277009088398652,
0.0297210421051448,
0.0298341506415484]
In excel, I've got the solutions using the following configuration:
I think the solution you got from pulp (CBC) is correct, it's just not the same as the one from Excel. To confirm the constraints are respected you can do the following:
solution3 = [x.value() for x in x_vars.values()]
# objective: 0.0204185791833761
sum(val*returns[i] for i, val in enumerate(solution1))
# 0.02053629325573023
sum(val*returns[i] for i, val in enumerate(solution3))
# 0.02039816058427895
# constraint:
sum(val for i, val in enumerate(solution1))
# 0.9782431569057903
sum(val for i, val in enumerate(solution3))
# 0.999999999
In fact, the solution from pulp is closer to the objective and respects a lot better the constraint of sum()=1.
I was trying to solve certain set of constraints using z3 in python. My code:
import math
from z3 import *
### declaration
n_co2 = []
c_co2 = []
alpha = []
beta = []
m_dot_air = []
n_pir = []
pir_sensor = []
for i in range(2):
c_co2.append(Real('c_co2_'+str(i)))
n_pir.append(Real('n_pir_'+str(i)))
n_co2.append(Real('n_co2_'+str(0)))
alpha.append(Real('alpha_'+str(0)))
beta.append(Real('beta_'+str(0)))
m_dot_air.append(Real('m_dot_air_'+str(0)))
pir_sensor.append(Real('pir_sensor_'+str(0)))
s = Solver()
s.add(n_co2[0]>0)
s.add(c_co2[0]>0)
s.add(c_co2[1]>=0.95*c_co2[0])
s.add(c_co2[1]<=1.05*c_co2[0])
s.add(n_co2[0]>=0.95*n_pir[1])
s.add(n_co2[0]<=1.05*n_pir[1])
s.add(c_co2[1]>0)
s.add(alpha[0]<=-1)
s.add(beta[0]>0)
s.add(m_dot_air[0]>0)
s.add(alpha[0]==-1*(1+ m_dot_air[0] + (m_dot_air[0]**2)/2.0 + (m_dot_air[0]**3)/6.0 ))
s.add(beta[0]== (1-alpha[0])/m_dot_air[0])
s.add(n_co2[0]== (c_co2[1]-alpha[0]*c_co2[0])/(beta[0]*19.6)-(m_dot_air[0]*339)/19.6)
s.add(n_pir[1]>=0)
s.add(pir_sensor[0]>=-1)
s.add(pir_sensor[0]<=1)
s.add(Not(pir_sensor[0]==0))
s.add(n_pir[1]==(n_pir[0]+pir_sensor[0]))
#### testing
s.add(pir_sensor[0]==1)
s.add(n_pir[1]==1)
s.add(n_co2[0]==1)
print(s.check())
print(s.reason_unknown())
print(s.model())
The output of the code:
sat
[c_co2_0 = 355,
c_co2_1 = 1841/5,
m_dot_air_0 = 1,
n_co2_0 = 1,
n_pir_1 = 1,
pir_sensor_0 = 1,
n_pir_0 = 0,
beta_0 = 11/3,
alpha_0 = -8/3,
/0 = [(19723/15, 1078/15) -> 1793/98,
(11/3, 1) -> 11/3,
else -> 0]]
What is the significance "/0 = ..." part of the output model.
But when I change the type of n_pir from Real to Int, z3 cannot solve it. Although we saw that we have an Int solution for n_pir. Reason of unknown:
smt tactic failed to show goal to be sat/unsat (incomplete (theory arithmetic))
How this problem can be solved? Could anyone please provide reasoning about this problem?
For the "/0" part: It's an internally generated constraint from converting real to int solutions. You can totally ignore that. In fact, you shouldn't really even look at the value of that, it's an artifact of the z3py bindings and should probably be hidden from the user.
For your question regarding why you cannot make 'Real' to 'Int'. That's because you have a non-linear set of equations (where you multiply or divide two variables), and non-linear integer arithmetic is undecidable in general. (Whereas non-linear real arithmetic is decidable.) So, when you use 'Int', solver simply uses some heuristics, and in this case fails and says unknown. This is totally expected. Read this answer for more details: How does Z3 handle non-linear integer arithmetic?
Z3 does come with an NRA solver, you can give that a try. Declare your solver as:
s = SolverFor("NRA")
But again you're at the mercy of the heuristics and you may or may not get a solution. Also, watch out for z3py bindings coercing constants to when you mix and match arithmetic like that. A good way is to write:
print s.sexpr()
before you call s.check() and take a look at the output and convince yourself that the translation has been done correctly. For details on that, see this question: Python and Z3: integers and floating, how to manage them in the correct way?
I am trying to solve MILP in puLP (Python), and I keep getting the following error:
Traceback (most recent call last):
File "main_lp.py", line 63, in <module>
ans = solve_lp(C)
File "/home/ashwin/Documents/Williams/f2014/math317_or/project/solve_lp.py", line 36, in solve_lp
prob.solve()
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/pulp.py", line 1619, in solve
status = solver.actualSolve(self, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solvers.py", line 1283, in actualSolve
return self.solve_CBC(lp, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solvers.py", line 1346, in solve_CBC
raise PulpSolverError("Pulp: Error while executing "+self.path)
pulp.solvers.PulpSolverError: Pulp: Error while executing /usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solverdir/cbc-32
For my linear programming problem, I am attempting to take sums of different vectors as the constraint, and I think I must have done that wrong somehow, because a much simpler problem works with no hitches. I have attached the code (C is an N by N numpy array).
def solve_lp(C):
N = len(C)
prob=LpProblem('Scheduling',LpMinimize)
X = [[LpVariable('X' + str(i+1) + str(j+1), 0, C[i,j],LpBinary)
for j in range(N)] for i in range(N)]
X = np.array(X)
X_o = [LpVariable('X0' + str(i), 0, None, LpBinary) for i in range(N)]
X_t = [LpVariable('X' + str(i) + 't', 0, None, LpBinary) for i in range(N)]
# Objective Function
ones_vec = list(np.ones(len(X_o)))
prob += lpDot(ones_vec,X_o), 'Minimize Buses'
# Constraints
for i in range(N):
row = list(X[i,:]) + [X_t[i]]
ones_vec = list(np.ones(len(row)))
prob += lpDot(ones_vec, row) == 1, 'Only one destination for ' + str(i)
for j in range(N):
col = list(X[:,j]) + [X_o[j]]
ones_vec = list(np.ones(len(col)))
prob += lpDot(ones_vec,col) == 1, 'Only one source for ' + str(j)
prob.solve()
return X, value(prob.objective)
Make sure you don't have duplicate LpVariable names, and watch out for LpVariable names with the unsupported characters -+[] ->/ since all of those characters are silently converted to underscores _
Setting LpSolverDefault.msg = 1 before calling prob.solve() may help by printing the solvers output to the console.
I recently had a similar problem due to Nan inputs in the model. I had the data in a DataFrame where some the cells should not bee converted to variables to improve performance. However, upon creating the objective function and the constraints, I noticed the presence of Nan and when I changed them it worked perfectly.
I think that you have duplicate LpVariable names. I just had the same problem and saw it thanks to levis501's answer. Here:
X = [[LpVariable('X' + str(i+1) + str(j+1), 0, C[i,j],LpBinary)
for j in range(N)] for i in range(N)]
X contains some variables with the same name. For example for i = 0 and j = 10 you get 'X111', and for i = 10 and j = 0 you also get 'X111'.
I had a similar problem, and indeed with duplicate LpVariable names just like levis501's answer and aleon answered.
my code was:
var = [[pulp.LpVariable(f'x{i}{j}', lowBound=0, cat=pulp.LpInteger) for j in range(col)] for i in range(row)]
when i=1 j=11, x would be x111, and when i=11 j=1, x also would be x111
I changed to:
var = [[pulp.LpVariable(f'x{i}_{j}', lowBound=0, cat=pulp.LpInteger) for j in range(col)] for i in range(row)]
After changing, it runs successfully.
I recently had the same error raised aswell. The reason this error code was raised in my case was that my data frame was not correctly populated. i incorrectly had NaN on the RHS on some of my constraints
What is had was something like:
Matrix = df.pivot(1st_dimension, 2nd_dimension, value)
This operation automatically puts NaN for the instances that are not in the original dataframe.
in my case the NaN'sare replaced with 0 which is what i was expecting it to be:
Matrix = Matrix.fillna(0)
I experienced the same issue when launching multiple instances of the LPSolver class. As fmars stated, the problem is that the path 'tmpSol' does not exist, which is defined in the following lines of code within the solvers.py file of pulp:
pid = os.getpid()
tmpLp = os.path.join(self.tmpDir, "%d-pulp.lp" % pid)
tmpMps = os.path.join(self.tmpDir, "%d-pulp.mps" % pid)
tmpSol = os.path.join(self.tmpDir, "%d-pulp.sol" % pid)
This bit of code appears in every solver. The problem is that these paths are deleted later on, but may coincide for different instances of the LPSolver class (as the variable pid is not unique).
The solution is to get a unique path for each instance of LPSolver, using, for example, the current time. Replacing the above lines by the following four will do the trick.
currentTime = time()
tmpLp = os.path.join(self.tmpDir, "%f3-pulp.lp" % currentTime)
tmpMps = os.path.join(self.tmpDir, "%f3-pulp.mps" % currentTime)
tmpSol = os.path.join(self.tmpDir, "%f3-pulp.sol" % currentTime)
Don't forget to
from time import time
Cheers,
Tim
I have met some similar problem which because of some PuLP's bug. When some problem is infeasible and the solver failed to solve the problem, PuLP raise an exception rather than return status equals to infeasible. Below is the reason.
(You may first want to check out latest codebase of PuLP because the line number you paste doesn't match the latest one. I'll explain based on latest one but you can look at yours very simply.)
https://github.com/coin-or/pulp/blob/master/src/pulp/solvers.py#L1405-L1406
This is where exception prompt.
if not os.path.exists(tmpSol):
raise PulpSolverError("Pulp: Error while executing "+self.path)
tmpSol is the temporary file where stores the solution. If PuLP cannot find such solution file, it will throw the exception you saw. And the bug I mentioned above is, if the problem itself is infeasible, then PuLP won't be able to generate such temporary file. So it will always throw such exception.
One thing you can do is, send a pull request to PuLP repo and fix it.
And a simple workaround is, rather than directly invoke
prob.solve()
in your code, you should always do
try:
prob.solve()
except Exception:
logger.debug('Problem infeasible')
I had this problem today, and it was because the temporary files for CBC were trying to be written in a location whose path had spaces in it, and the command that pulp passes to subprocess.Popen() to run CBC does not use quotes, and thus the command was misinterpreted and CBC couldn't find the location to create temp files.
For this I found two solutions:
(1) Explicitly set a temporary file directory that doesn't have spaces in it,
pulp.LpSolverDefault.tmpDir = self.tmp_file_dir # tmp_file_dir can't have spaces!
prob.solve()
or
(2) don't use CBC (my problem is small)
prob.solve(pulp.GLPK_CMD())
I have a work-related constraint that a lot of my work is stuck in directories that have spaces.
I am working on some very large scale linear programming problems. (Matrices are currently roughly 1000x1000 and these are the 'mini' ones.)
I thought that I had the program running successfully, only I have realized that I am getting some very unintuitive answers. For example, let's say I were to maximize x+y+z subject to a set of constraints x+y<10 and y+z <5. I run this and get an optimal solution. Then, I run the same equation but with different constraints: x+y<20 and y+z<5. Yet in the second iteration, my maximization decreases!
I have painstakingly gone through and assured myself that the constraints are loading correctly.
Does anyone know what the problem might be?
I found something in the documentation about lpx_check_kkt which seems to tell you when your solution is likely to be correct or high confidence (or low confidence for that matter), but I don't know how to use it.
I made an attempt and got the error message lpx_check_kkt not defined.
I am adding some code as an addendum in hopes that someone can find an error.
The result of this is that it claims an optimal solution has been found. And yet every time I raise an upper bound, it gets less optimal.
I have confirmed that my bounds are going up and not down.
size = 10000000+1
ia = intArray(size)
ja = intArray(size)
ar = doubleArray(size)
prob = glp_create_prob()
glp_set_prob_name(prob, "sample")
glp_set_obj_dir(prob, GLP_MAX)
glp_add_rows(prob, Num_constraints)
for x in range(Num_constraints):
Variables.add_variables(Constraints_for_simplex)
glp_set_row_name(prob, x+1, Variables.variers[x])
glp_set_row_bnds(prob, x+1, GLP_UP, 0, Constraints_for_simplex[x][1])
print 'we set the row_bnd for', x+1,' to ',Constraints_for_simplex[x][1]
glp_add_cols(prob, len(All_Loops))
for x in range(len(All_Loops)):
glp_set_col_name(prob, x+1, "".join(["x",str(x)]))
glp_set_col_bnds(prob,x+1,GLP_LO,0,0)
glp_set_obj_coef(prob,x+1,1)
for x in range(1,len(All_Loops)+1):
z=Constraints_for_simplex[0][0][x-1]
ia[x] = 1; ja[x] = x; ar[x] = z
x=len(All_Loops)+1
while x<Num_constraints + len(All_Loops):
for y in range(2, Num_constraints+1):
z=Constraints_for_simplex[y-1][0][0]
ia[x] = y; ja[x] =1 ; ar[x] = z
x+=1
x=Num_constraints+len(All_Loops)
while x <len(All_Loops)*(Num_constraints-1):
for z in range(2,len(All_Loops)+1):
for y in range(2,Num_constraints+1):
if x<len(All_Loops)*Num_constraints+1:
q = Constraints_for_simplex[y-1][0][z-1]
ia[x] = y ; ja[x]=z; ar[x] = q
x+=1
glp_load_matrix(prob, len(All_Loops)*Num_constraints, ia, ja, ar)
glp_exact(prob,None)
Z = glp_get_obj_val(prob)
Start by solving your problematic instances with different solvers and checking the objective function value. If you can export your model to .mps format (I don't know how to do this with GLPK, sorry), you can upload the mps file to http://www.neos-server.org/neos/solvers/index.html and solve it with several different LP solvers.