Develop Reference

How to iterate over external input list in pyomo objective function?

I am trying to run a simple LP pyomo Concrete model with Gurobisolver :
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
model = pyo.ConcreteModel()
nb_years = 3
nb_mins = 2
step = 8760*1.5
delta = 10000
#Range of hour
model.h = pyo.RangeSet(0,8760*nb_years-1)
#Individual minimums
model.min = pyo.RangeSet(0, nb_mins-1)
model.mins = pyo.Var(model.min, within=model.h, initialize=[i for i in model.min])
def maximal_step_between_mins_constraint_rule(model, min):
next_min = min + 1 if min < nb_mins-1 else 0
if next_min == 0: # We need to take circularity into account
return 8760*nb_years - model.mins[min] + model.mins[next_min] <= step + delta
return model.mins[next_min] - model.mins[min] <= step + delta
def minimal_step_between_mins_constraint_rule(model, min):
next_min = min + 1 if min < nb_mins-1 else 0
if next_min == 0: # We need to take circularity into account
return 8760*nb_years - model.mins[min] + model.mins[next_min] >= step - delta
return model.mins[next_min] - model.mins[min] >= step - delta
model.input_list = pyo.Param(model.h, initialize=my_input_list, within=pyo.Reals, mutable=False)
def objective_rule(model):
return sum([model.input_list[model.mins[min]] for min in model.min])
model.maximal_step_between_mins_constraint= pyo.Constraint(model.min, rule=maximal_step_between_mins_constraint_rule)
model.minimal_step_between_mins_constraint= pyo.Constraint(model.min, rule=minimal_step_between_mins_constraint_rule)
model.objective = pyo.Objective(rule=objective_rule, sense=pyo.minimize)
opt = SolverFactory('gurobi')
results = opt.solve(model, options={'Presolve':2})
Basically I am trying to find two hours in my input list (which looks like this) spanning over 3 years of data, with constraints on the distance separating them, and where the sum of both value is minimized by the model.
I implemented my list as a parameter of fixed value, however even if mutable is set to False running my model produces this error :
ERROR: Rule failed when generating expression for Objective objective with
index None: RuntimeError: Error retrieving the value of an indexed item
input_list: index 0 is not a constant value. This is likely not what you
meant to do, as if you later change the fixed value of the object this
lookup will not change. If you understand the implications of using non-
constant values, you can get the current value of the object using the
value() function.
ERROR: Constructing component 'objective' from data=None failed: RuntimeError:
Error retrieving the value of an indexed item input_list: index 0 is not a
constant value. This is likely not what you meant to do, as if you later
change the fixed value of the object this lookup will not change. If you
understand the implications of using non-constant values, you can get the
current value of the object using the value() function.
Any idea why I get this error and how to fix it ?
Obviously, changing the objective function to sum([pyo.value(model.input_list[model.mins[min]]) for min in model.min]) is not a solution to my problem.
I also tried not to use pyomo parameters (with something like sum([input_list[model.mins[min]] for min in model.min]), but pyomo can't iterate over it and raises the following error :
ERROR: Constructing component 'objective' from data=None failed: TypeError:
list indices must be integers or slices, not _GeneralVarData

You have a couple serious syntax and structure problems in your model. Not all of the elements are included in the code you provide, but you (minimally) need to fix these:
In this snippet, you are initializing the value of each variable to a list, which is invalid. Start with no variable initializations:
model.mins = pyo.Var(model.min, within=model.h, initialize=[i for i in model.min])
In this summation, you appear to be using a variable as the index for some data. This is an invalid construct. The value of the variable is unkown when the model is built. You need to reformulate:
return sum([model.input_list[model.mins[min]] for min in model.min])
My suggestion: Start with a very small chunk of your data and pprint() your model and read it carefully for quality before you attempt to solve.
model.pprint()

cplex changes the fixed variable after solving the problem in pyomo

I am trying to solve a MIP, I use pyomo, and Cplex(Interactive Optimizer 20.1.0.0) is solver.
The problem is that I want to fix some binary integer variables then solve the problem, and I used:
model.y[1,4].fix(1)
model.y[2,3].fix(0)
, but I have noticed that after solving the problem those fixed variables have changed to another values.
How can I say cplex to not change that fixed variables?

let me use the bus example for fixed start with pyomo
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
opt = pyo.SolverFactory("cplex")
model = pyo.ConcreteModel()
model.nbBus = pyo.Var([40,30], domain=pyo.PositiveIntegers)
#fixed start
model.nbBus[40].fix(3)
# end of fixed start
model.OBJ = pyo.Objective(expr = 500*model.nbBus[40] + 400*model.nbBus[30])
model.Constraint1 = pyo.Constraint(expr = 40*model.nbBus[40] + 30*model.nbBus[30] >= 300)
results = opt.solve(model)
print("nbBus40=",int(model.nbBus[40].value))
print("nbBus30=",int(model.nbBus[30].value))
gives
nbBus40= 3
nbBus30= 6
whereas if you remove the fixed start you get
nbBus40= 6
nbBus30= 2

Since fixing variable didn't work for me, I tackle my problem with adding a new constraint to the model, and it works.
def yvar_fix(model, i, j ):
if (i,j) in y_set_init:
constraint = (model.y[i,j] == 1)
else:
constraint = (model.y[i,j] == 0)
return constraint
model.yvar_fix = pe.Constraint(model.edges, rule=yvar_fix)

PYOMO Constraints - setting constraints over indexed variables

I have been trying to get into python optimization, and I have found that pyomo is probably the way to go; I had some experience with GUROBI as a student, but of course that is no longer possible, so I have to look into the open source options.
I basically want to perform an non-linear mixed integer problem in which I will minimized a certain ratio. The problem itself is setting up a power purchase agreement (PPA) in a renewable energy scenario. Depending on the electricity generated, you will have to either buy or sell electricity acording to the PPA.
The only starting data is the generation; the PPA is the main decision variable, but I will need others. "buy", "sell", "b1" and "b2" are unknown without the PPA value. These are the equations:
Equations that rule the problem (by hand).
Using pyomo, I was trying to set up the problem as:
# Dataframe with my Generation information:
January = Data['Full_Data'][(Data['Full_Data']['Month'] == 1) & (Data['Full_Data']['Year'] == 2011)]
Gen = January['Producible (MWh)']
Time = len(Generacion)
M=100
# Model variables and definition:
m = ConcreteModel()
m.IDX = range(time)
m.PPA = Var(initialize = 2.0, bounds =(1,7))
m.compra = Var(m.IDX, bounds = (0, None))
m.venta = Var(m.IDX, bounds = (0, None))
m.b1 = Var(m.IDX, within = Binary)
m.b2 = Var(m.IDX, within = Binary)
And then, the constraint; only the first one, as I was already getting errors:
m.b1_rule = Constraint(
expr = (((Gen[i] - PPA)/M for i in m.IDX) <= m.b1[i])
)
which gives me the error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-5d5f5584ebca> in <module>
1 m.b1_rule = Constraint(
----> 2 expr = (((Generacion[i] - PPA)/M for i in m.IDX) <= m.b1[i])
3 )
pyomo\core\expr\numvalue.pyx in pyomo.core.expr.numvalue.NumericValue.__ge__()
pyomo\core\expr\logical_expr.pyx in pyomo.core.expr.logical_expr._generate_relational_expression()
AttributeError: 'generator' object has no attribute 'is_expression_type'
I honestly have no idea what this means. I feel like this should be a simple problem, but I am strugling with the syntax. I basically have to apply a constraint to each individual data from "Generation", there is no sum involved; all constraints are 1-to-1 contraints set so that the physical energy requirements make sense.
How do I set up the constraints like this?
Thank you very much

You have a couple things to fix. First, the error you are getting is because you have "extra parenthesis" around an expression that python is trying to convert to a generator. So, step 1 is to remove the outer parenthesis, but that will not solve your issue.
You said you want to generate this constraint "for each" value of your index. Any time you want to generate copies of a constraint "for each" you will need to either do that by making a constraint list and adding to it with some kind of loop, or use a function-rule combination. There are examples of each in the pyomo documentation and plenty on this site (I have posted a ton if you look at some of my posts.) I would suggest the function-rule combo and you should end up with something like:
def my_constr(m, i):
return m.Gen[i] - m.PPA <= m.b1[i] * M
m.C1 = Constraint(m.IDX, rule=my_constr)

How do I setup an objective function in CPLEX Python containing indicator functions?

The following is the objective function:
The idea is that a mean-variance optimization has already been done on a universe of securities. This gives us the weights for a target portfolio. Now suppose the investor already is holding a portfolio and does not want to change their entire portfolio to the target one.
Let w_0 = [w_0(1),w_0(2),...,w_0(N)] be the initial portfolio, where w_0(i) is the fraction of the portfolio invested in
stock i = 1,...,N. Let w_t = [w_t(1), w_t(2),...,w_t(N)] be the target portfolio, i.e., the portfolio
that it is desirable to own after rebalancing. This target portfolio may be constructed using quadratic optimization techniques such as variance minimization.
The objective is to decide the final portfolio w_f = [w_f (1), w_f (2),..., w_f(N)] that satisfies the
following characteristics:
(1) The final portfolio is close to our target portfolio
(2) The number of transactions from our initial portfolio is sufficiently small
(3) The return of the final portfolio is high
(4) The final portfolio does not hold many more securities that our initial portfolio
An objective function which is to be minimized is created by summing together the characteristic terms 1 through 4.
The first term is captured by summing the absolute difference in weights from the final and the target portfolio.
The second term is captured by the sum of an indicator function multiplied by a user specified penalty. The indicator function is y_{transactions}(i) where it is 1 if the weight of security i is different in the initial portfolio and the final portfolio, and 0 otherwise.
The third term is captured by the total final portfolio return multiplied by a negative user specified penalty since the objective is minimization.
The final term is the count of assets in the final portfolio (ie. sum of an indicator function counting the number of positive weights in the final portfolio), multiplied by a user specified penalty.
Assuming that we already have the target weights as target_w how do I setup this optimization problem in docplex python library? Or if anyone is familiar with mixed integer programming in NAG it would be helpful to know how to setup such a problem there as well.
`
final_w = [0.]*n
final_w = np.array(final_w)
obj1 = np.sum(np.absolute(final_w - target_w))
pen_trans = 1.2
def ind_trans(final,inital):
list_trans = []
for i in range(len(final)):
if abs(final[i]-inital[i]) == 0:
list_trans.append(0)
else:
list_trans.append(1)
return list_trans
obj2 = pen_trans*sum(ind_trans(final_w,initial_w))
pen_returns = 0.6
returns_np = np.array(df_secs['Return'])
obj3 = (-1)*np.dot(returns_np,final_w)
pen_count = 1.
def ind_count(final):
list_count = []
for i in range(len(final)):
if final[i] == 0:
list_count.append(0)
else:
list_count.append(1)
return list_count
obj4 = sum(ind_count(final_w))
objective = obj1 + obj2 + obj3 + obj4

The main issue in your code is that final_w is not a an array of variables but an array of data. So there will be nothing to optimize. To create an array of variables in docplex you have to do something like this:
from docplex.mp.model import Model
with Model() as m:
final = m.continuous_var_list(n, 0.0, 1.0)
That creates n variables that can take values between 0 and 1. With that in hand you can start things. For example:
obj1 = m.sum(m.abs(initial[i] - final[i]) for i in range(n))
For the next objective things become harder since you need indicator constraints. To simplify definition of these constraints first define a helper variable delta that gives the absolute difference between stocks:
delta = m.continuous_var_list(n, 0.0, 1.0)
m.add_constraints(delta[i] == m.abs(initial[i] - final[i]) for i in range(n))
Next you need an indicator variable that is 1 if a transaction is required to adjust stock i:
needtrans = m.binary_var_list(n)
for i in range(n):
# If needtrans[i] is 0 then delta[i] must be 0.
# Since needtrans[i] is penalized in the objective, the solver will
# try hard to set it to 0. It will only set it to 1 if delta[i] != 0.
# That is exactly what we want
m.add_indicator(needtrans[i], delta[i] == 0, 0)
With that you can define the second objective:
obj2 = pen_trans * m.sum(needtrans)
once all objectives have been defined, you can add their sum to the model:
m.minimize(obj1 + obj2 + obj3 + obj4)
and then solve the model and display its solution:
m.solve()
print(m.solution.get_values(final))
If any of the above is not (yet) clear to you then I suggest you take a look at the many examples that ship with docplex and also at the (reference) documentation.

Finding minimum value of a function wit 11,390,625 variable combinations

I am working on a code to solve for the optimum combination of diameter size of number of pipelines. The objective function is to find the least sum of pressure drops in six pipelines.
As I have 15 choices of discrete diameter sizes which are [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80] that can be used for any of the six pipelines that I have in the system, the list of possible solutions becomes 15^6 which is equal to 11,390,625
To solve the problem, I am using Mixed-Integer Linear Programming using Pulp package. I am able to find the solution for the combination of same diameters (e.g. [2,2,2,2,2,2] or [4,4,4,4,4,4]) but what I need is to go through all combinations (e.g. [2,4,2,2,4,2] or [4,2,4,2,4,2] to find the minimum. I attempted to do this but the process is taking a very long time to go through all combinations. Is there a faster way to do this ?
Note that I cannot calculate the pressure drop for each pipeline as the choice of diameter will affect the total pressure drop in the system. Therefore, at anytime, I need to calculate the pressure drop of each combination in the system.
I also need to constraint the problem such that the rate/cross section of pipeline area > 2.
Your help is much appreciated.
The first attempt for my code is the following:
from pulp import *
import random
import itertools
import numpy
rate = 5000
numberOfPipelines = 15
def pressure(diameter):
diameterList = numpy.tile(diameter,numberOfPipelines)
pressure = 0.0
for pipeline in range(numberOfPipelines):
pressure += rate/diameterList[pipeline]
return pressure
diameterList = [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80]
pipelineIds = range(0,numberOfPipelines)
pipelinePressures = {}
for diameter in diameterList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(diameter))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[diameter] = pressureList
print 'pipepressure', pipelinePressures
prob = LpProblem("Warehouse Allocation",LpMinimize)
use_diameter = LpVariable.dicts("UseDiameter", diameterList, cat=LpBinary)
use_pipeline = LpVariable.dicts("UsePipeline", [(i,j) for i in pipelineIds for j in diameterList], cat = LpBinary)
## Objective Function:
prob += lpSum(pipelinePressures[j][i] * use_pipeline[(i,j)] for i in pipelineIds for j in diameterList)
## At least each pipeline must be connected to a diameter:
for i in pipelineIds:
prob += lpSum(use_pipeline[(i,j)] for j in diameterList) ==1
## The diameter is activiated if at least one pipelines is assigned to it:
for j in diameterList:
for i in pipelineIds:
prob += use_diameter[j] >= lpSum(use_pipeline[(i,j)])
## run the solution
prob.solve()
print("Status:", LpStatus[prob.status])
for i in diameterList:
if use_diameter[i].varValue> pressureTest:
print("Diameter Size",i)
for v in prob.variables():
print(v.name,"=",v.varValue)
This what I did for the combination part which took really long time.
xList = np.array(list(itertools.product(diameterList,repeat = numberOfPipelines)))
print len(xList)
for combination in xList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(combination))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[combination] = pressureList
print 'pipelinePressures',pipelinePressures

I would iterate through all combinations, I think you would run into memory problems otherwise trying to model ALL combinations in a MIP.
If you iterate through the problems perhaps using the multiprocessing library to use all cores, it shouldn't take long just remember only to hold information on the best combination so far, and not to try and generate all combinations at once and then evaluate them.
If the problem gets bigger you should consider Dynamic Programming Algorithms or use pulp with column generation.