I recently started using CPLEX integrated in python for my master project and I have a hard time with one of my variables. I am modelling the charge and discharge of a battery in function of the wind and solar power as well as electricity market prices. All my variables for charge, discharge and production are well defined but my battery state of charge ends up being null at all times after solving. When calling get values of this variable i get a list of zeros (with sol the solution of the optimization and Ebes the name of the State of charge):
sol.get_values(Ebes[t]for t in time)
It is even unfeasible that this variable would be null as i also have the constraints in the model :
for t in time:
mdl.add_constraint(Ebes[t]>=Ebmin)
mdl.add_constraint(Ebes[t]<=Ebmax)
When I display the model before solving with print(mdl.export_to_string()) it shows that Ebes is constrained to be higher than Ebmin(=20) for all the time steps. The only hint I get is that the name of the variables in there is slightly different from the others. In here the variables of Ebes are named _Ebes_date whereas the other variables are named for example Pdischarge_date and not _Pdischarge_date. I guess this "_" before the name shows that there is a problem but i can't manage to find what to change.
My variables are defined as:
Ebes=mdl.continuous_var_dict(time,name='Ebes')
for i in range(len(time)):
if i==0:
mdl.add_constraint(Ebes[time[0]]==Ebes[time[len(time)-1]]*(1-etaleak)+Pcha[time[0]]*etacha*dt-(Pdis[time[0]]/etadis*dt))
else:
t=time[i]
tm=time[i-1]
mdl.add_constraint(Ebes[t]==Ebes[tm]*(1-etaleak)+Pcha[t]*etacha*dt-(Pdis[t]/etadis*dt))
Thank you if you take time to answer me :)
The whole example:
import pandas as pd
from docplex.mp.model import Model
ind=['01_09_2016 00','01_09_2016 01','01_09_2016 02','01_09_2016 03','01_09_2016 04']#[1,2,3,4,5]
M=pd.Series(data=[10,30,30,15,30],index=ind)
P=pd.DataFrame(data={'Time':ind,'DK2_wind':[0.3,0.24,0.14,0.18,0.22],'DK2_solar':[0,0,0,0,0.01]}).set_index('Time',drop=True)
mdl=Model('dispatch')
time=P.index
Psolar=300 #MW
Pwind= 400 #MW
P.DK2_solar=P.DK2_solar*Psolar
P.DK2_wind=P.DK2_wind*Pwind
Pres=P.sum(axis=1) #MW
Pmax=800 #MW
#Battery parameters:
Pbmax= 50 #MW
Ebmax= 100 #MWh
Ebmin= 20 #MWh
Pbal =mdl.continuous_var_dict(time,name='Pbal')
Pcha =mdl.continuous_var_dict(time,name='Pcharge')
Pdis =mdl.continuous_var_dict(time,name='Pdischarge')
Ebes=mdl.continuous_var_dict(time,name='Ebes')
switch=mdl.binary_var_dict(time,name='switch')
for t in time:
mdl.add_constraint(Pbal[t]==Pres[t]+Pdis[t]-Pcha[t])
for t in time:
mdl.add_constraint(Pdis[t]<=Pbmax*(1-switch[t]))
for t in time:
mdl.add_constraint(Pdis[t]>=0)
for t in time:
mdl.add_constraint(Pcha[t]<=Pbmax*switch[t])
for t in time:
mdl.add_constraint(Pcha[t]>=0)
for t in time:
mdl.add_constraint(Ebes[t]>=Ebmin)
for t in time:
mdl.add_constraint(Ebes[t]<=Ebmax)
for i in range(len(time)):
if i==0:
mdl.add_constraint(Ebes[time[0]]==Ebes[time[len(time)-1]]+Pcha[time[0]]-Pdis[time[0]])
else:
t=time[i]
tm=time[i-1]
mdl.add_constraint(Ebes[t]==Ebes[tm]+Pcha[t]-Pdis[t])
mdl.maximize(mdl.sum((Pbal[t]*M[t]) for t in time))
sol=mdl.solve(url=URLmt,key=Mykey,log_output=True)
sol_Ebess=sol.get_values(Ebes[t]for t in time)
sol_Ebess
sol.solve_details.status
So here the sol_Ebess is null for all indices. if I change ind to be numbers instead it works and Ebess is equal to the real value.
Related
I'm trying to find the best allocation for a portfolio based on backtesting data. As a general rule, I've divided stocks into large caps and small/mid caps and growth/value and want no more than 80% of my portfolio in large caps or 70% of my portfolio in value. I need an algorithm that will be flexible enough to use for more than two stocks. So far, what I have is (including a random class called Ticker):
randomBoolean=True
listOfTickers=[]
listOfLargeCaps=[]
listOfSmallMidCaps=[]
largeCapAllocation=0
listOfValue=[]
listOfGrowthBlend=[]
valueAllocation=0
while randomBoolean:
tickerName=input("What is the name of the ticker?")
tickerCap=input("What is the cap of the ticker?")
tickerAllocation=int(input("Around how much do you want to allocate in this ticker?"))
tickerValue=input("Is this ticker a Value, Growth, or Blend stock?")
tickerName=Ticker(tickerCap,tickerValue,tickerAllocation,tickerName)
listOfTickers.append(tickerName)
closer=input("Type DONE if you are finished. Type ENTER to continue entering tickers")
if closer=="DONE":
randomBoolean=False
for ticker in listOfTickers:
if ticker.cap==("Large" or "large"):
listOfLargeCaps.append(ticker)
else:
listOfSmallMidCaps.append(ticker)
if ticker.value==("Value" or "value"):
listOfValue.append(ticker)
else:
listOfGrowthBlend.append(ticker)
for largeCap in listOfLargeCaps:
largeCapAllocation +=largeCap.allocation
if largeCapAllocation>80:
#run a function that will readjust ticker stuff and decrease allocation to large cap stocks
for value in listOfValue:
valueAllocation+=value.allocation
if valueAllocation>70:
#run a function that will readjust ticker stuff and decrease allocation to value stocks
The "function" I have so far just iterates through -5 to 6 in a sort of
for i in range (-5,6):
ticker1AllocationPercent + i
ticker2AllocationPercent - i
#update the bestBalance if the new allocation is better
How would I modify this algorithm to work for 3, 4, 5, etc. stocks, and how would I go about changing the allocations for the large/small-mid cap stocks and such?
As mentioned in the above answer, typically Quadratic solver is used in such problems. You can use Quadratic solver available in Pyportfolio. See this link for more details.
Need help in solving a demand-optimiztion planning for factories.
Factories have Machines which can make one or more Products in it.
Each Product takes time to make 1 unit which is known as 'Cycle-Time'. So, to make 10 units of product/component on a machine with cycle-time of 5, it will take 5*10=50 seconds in total.
Not all products can be made in all machines.
So, we need to make products on valid machines and in the most effective manner.
Also, each machine has an availability limit (in seconds) and we can't go over it.
What we need to do is :
apply a "cost" of running a machine to make one or more products.
apply a "cost" in case the demand of a product is NOT met.
Objective is to Minimize this cost.
I'm also happy if we are able to solve this using equations as constraints (like model += ( x1 * 0.055555555555556 <= 10000, "material_300005551211-2" )). but unable to do so at the moment.
Sample data :
I tried optimizing PULP, but this approach isn't working correctly - for example, in case of demand being too high, it doens't max upto the limit of machine's availability but unsure where i'm going wrong.
import pandas as pd
import pulp
factories = pd.read_csv('factory_machines_small.csv', index_col=['Component', 'Machine'])
print(factories)
demand = pd.read_csv('component_demand_small.csv', index_col=['Component'])
print(demand)
production = pulp.LpVariable.dicts("production",
((component, machine) for component, machine in factories.index),
lowBound=0,
#upBound=1,
cat='Integer')
factory_status = pulp.LpVariable.dicts("factory_status",
((component, machine) for component, machine in factories.index),
cat='Binary')
model = pulp.LpProblem("Cost minimising scheduling problem", pulp.LpMinimize)
model += pulp.lpSum(
[production[component, machine] * factories.loc[(component, machine), 'Cycle_Time'] for component, machine in factories.index]
)
# Production in any month must be equal to demand
components = demand.index
for component in components :
model += production[(component, 'IP01')] + production[(component, 'IP02')] + production[(component, 'IP03')] \
+ production[(component, 'IP04')] + production[(component, 'IP05')] == demand.loc[component, 'Demand']
# Production in any month must be between minimum and maximum capacity, or zero.
for component, machine in factories.index:
min_production = factories.loc[(component, machine), 'Min_Capacity']
max_production = factories.loc[(component, machine), 'Max_Capacity']
model += production[(component, machine)] >= min_production * factory_status[component, machine]
model += production[(component, machine)] <= max_production * factory_status[component, machine]
model.solve()
print(pulp.LpStatus[model.status])
output = []
for component, machine in production:
var_output = {
'Component': component,
'Machine': machine,
'Production': production[(component, machine)].varValue,
'Machine Status': factory_status[(component, machine)].varValue
}
output.append(var_output)
#print(output)
output_df = pd.DataFrame.from_records(output).sort_values(['Component', 'Machine'])
output_df.set_index(['Component', 'Machine'], inplace=True)
print(output_df)
output_df.to_csv('OUTPUT.csv')
# Print our objective function value (Total Costs)
print (pulp.value(model.objective))
The first thing to do is to get rid of production[(component, 'IP01')]+production[(component, 'IP02')]+production[(component, 'IP03')]+production[(component, 'IP04')]+production[(component, 'IP05')]. I hope you realize how bad this type of hardcoding is. This only works if you only have machines IP01..IP05. Indeed for the example data, this is not the case. You need to make this data-driven. We have a sum construct for that.
To model shortage, the rhs of the supply=demand constraint needs to become
production == demand - shortage
where shortage is an additional nonnegative variable. You also need to add a cost term to the objective for this variable.
I currently have an array of desired position vs. time of an object in my plant. I am using an inverse dynamics controller in order to drive the object to this desired position but I'm experiencing some difficulties. Here is how I am doing this:
I created the controller system
ID_cont = InverseDynamicsController(robot=controller_plant, kp=np.array([0.5]), ki=np.array([0.3]), kd=np.array([0.4]), has_reference_acceleration=False)
ID_controller = builder.AddSystem(ID_cont)
I got the controller input and output ports
control_estimated_state_input_port = ID_controller.get_input_port(0)
control_desired_state_input_port = ID_controller.get_input_port(1)
control_output_port = ID_controller.get_output_port(0)
I added a constant state source (likely wrong to do) and a state interpolator
constant_state_source = ConstantVectorSource(np.array([0.0]))
builder.AddSystem(constant_state_source)
position_to_state = StateInterpolatorWithDiscreteDerivative(controller_plant.num_positions(),
controller_plant.time_step())
builder.AddSystem(position_to_state)
I wired the controller to the plant
builder.Connect(constant_state_source.get_output_port(), position_to_state.get_input_port())
builder.Connect(position_to_state.get_output_port(), control_desired_state_input_port)
builder.Connect(plant.get_state_output_port(model_instance_1), control_estimated_state_input_port)
builder.Connect(control_output_port, plant.get_actuation_input_port(model_instance_1))
Next, I am trying to create a while loop that advances the simulation and changes the 'constant vector source' so I can feed in my position vs. time values but I'm unsure if the reason this isn't working out is because this is the complete wrong approach or if this is the right approach but I just have a few things wrong
diagram_context = diagram.CreateDefaultContext()
sim_time_temp = diagram_context.get_time()
time_step = 0.1
while sim_time_temp < duration:
ID_controller_context = diagram.GetMutableSubsystemContext(ID_controller, diagram_context)
simulator.AdvanceTo(sim_time_temp)
sim_time_temp = sim_time_temp + time_step
I added a constant state source (likely wrong to do) and a state interpolator
As you suspected, this is not the best way to go if you already have a desired sequence of positions and times that you want the system to track. Instead, you should use a TrajectorySource. Since you have a set of positions samples, positions (num_times x num_positions array), that you'd like the system to hit at specified times (num_times x 1 array), PiecewisePolynomial.CubicShapePreserving is a reasonable choice for building the trajectory.
desired_position_trajectory = PiecewisePolynomial.CubicShapePreserving(times, positions)
desired_state_source = TrajectorySource(desired_position_trajectory,
output_derivative_order=1)
builder.AddSystem(desired_state_source)
The output_derivative_order=1 argument makes desired_state_source output a [position, velocity] vector rather than just a position vector. You can connect desired_state_source directly to the controller, without an interpolator.
With this setup, you can advance the simulation all the way to duration without the need for a while loop.
I am working on a code to solve for the optimum combination of diameter size of number of pipelines. The objective function is to find the least sum of pressure drops in six pipelines.
As I have 15 choices of discrete diameter sizes which are [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80] that can be used for any of the six pipelines that I have in the system, the list of possible solutions becomes 15^6 which is equal to 11,390,625
To solve the problem, I am using Mixed-Integer Linear Programming using Pulp package. I am able to find the solution for the combination of same diameters (e.g. [2,2,2,2,2,2] or [4,4,4,4,4,4]) but what I need is to go through all combinations (e.g. [2,4,2,2,4,2] or [4,2,4,2,4,2] to find the minimum. I attempted to do this but the process is taking a very long time to go through all combinations. Is there a faster way to do this ?
Note that I cannot calculate the pressure drop for each pipeline as the choice of diameter will affect the total pressure drop in the system. Therefore, at anytime, I need to calculate the pressure drop of each combination in the system.
I also need to constraint the problem such that the rate/cross section of pipeline area > 2.
Your help is much appreciated.
The first attempt for my code is the following:
from pulp import *
import random
import itertools
import numpy
rate = 5000
numberOfPipelines = 15
def pressure(diameter):
diameterList = numpy.tile(diameter,numberOfPipelines)
pressure = 0.0
for pipeline in range(numberOfPipelines):
pressure += rate/diameterList[pipeline]
return pressure
diameterList = [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80]
pipelineIds = range(0,numberOfPipelines)
pipelinePressures = {}
for diameter in diameterList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(diameter))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[diameter] = pressureList
print 'pipepressure', pipelinePressures
prob = LpProblem("Warehouse Allocation",LpMinimize)
use_diameter = LpVariable.dicts("UseDiameter", diameterList, cat=LpBinary)
use_pipeline = LpVariable.dicts("UsePipeline", [(i,j) for i in pipelineIds for j in diameterList], cat = LpBinary)
## Objective Function:
prob += lpSum(pipelinePressures[j][i] * use_pipeline[(i,j)] for i in pipelineIds for j in diameterList)
## At least each pipeline must be connected to a diameter:
for i in pipelineIds:
prob += lpSum(use_pipeline[(i,j)] for j in diameterList) ==1
## The diameter is activiated if at least one pipelines is assigned to it:
for j in diameterList:
for i in pipelineIds:
prob += use_diameter[j] >= lpSum(use_pipeline[(i,j)])
## run the solution
prob.solve()
print("Status:", LpStatus[prob.status])
for i in diameterList:
if use_diameter[i].varValue> pressureTest:
print("Diameter Size",i)
for v in prob.variables():
print(v.name,"=",v.varValue)
This what I did for the combination part which took really long time.
xList = np.array(list(itertools.product(diameterList,repeat = numberOfPipelines)))
print len(xList)
for combination in xList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(combination))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[combination] = pressureList
print 'pipelinePressures',pipelinePressures
I would iterate through all combinations, I think you would run into memory problems otherwise trying to model ALL combinations in a MIP.
If you iterate through the problems perhaps using the multiprocessing library to use all cores, it shouldn't take long just remember only to hold information on the best combination so far, and not to try and generate all combinations at once and then evaluate them.
If the problem gets bigger you should consider Dynamic Programming Algorithms or use pulp with column generation.
Sorry, a little finance related on the topic, but also a scipy/python question. For context of what I am trying to do, it's literally the same as these two blog posts.
https://quantdare.com/risk-parity-in-python/
https://thequantmba.wordpress.com/2016/12/14/risk-parityrisk-budgeting-portfolio-in-python/
So I have a bunch of returns on stocks, and I want to equalize the risk contributions of each stock. To do this I will need to solve for the weights that will give me an equal risk contribution for each using the scipy minimize optimizer.
So I will pass in my target risk contributions, and my initial guess into the optimizer. For example, 6 stocks. My initial guess is merely 1/6 of the total 100% weight in the portfolio.
initial_weight = [0.16666666666667, 0.16666666666667, 0.16666666666667,
0.16666666666667, 0.16666666666667, 0.16666666666667]
risk_contrib_target =[0.16666666666667, 0.16666666666667, 0.16666666666667,
0.16666666666667, 0.16666666666667, 0.16666666666667]
This was taken from the quantmba link, so all credit to that guy. It looks right to me.
# risk budgeting optimization
def calculate_portfolio_var(w,V):
# function that calculates portfolio risk
w = np.matrix(w)
return (w*V*w.T)[0,0]
def calculate_risk_contribution(w,V):
# function that calculates asset contribution to total risk
w = np.matrix(w, dtype=object)
sigma = np.sqrt(calculate_portfolio_var(w,V))
# Marginal Risk Contribution
MRC = V*w.T
# Risk Contribution
RC = np.multiply(MRC,w.T)/sigma
RC = RC / sum(RC)
return RC
def risk_budget_objective(x,pars):
# calculate portfolio risk
V = pars[0]# covariance table
x_t = pars[1] # risk target in percent of portfolio risk
sig_p = np.sqrt(calculate_portfolio_var(x,V)) # portfolio sigma
risk_target = np.asmatrix(x_t, dtype=object)
asset_RC = calculate_risk_contribution(x,V)
J = sum(np.square(asset_RC-risk_target.T))[0,0] * 1000 # sum of squared error
return J
I also have a list of dates that I am running through to solve this many times over a time period.
rebalance_dates = my_list_of_dates
I noticed that sometimes, it doesn't solve this correctly. This is easy to check because the way it is set up, the function should have a 0 solution. Also I can check the risk contribution afterwards to see that they reached my target. To get around this, I kick it to basin hopping if it does not find this 0 solution. I think it is solving a local minimum and not a global minimum and I read this is one solution to that problem.
The get_returns_matrix function is just getting the data that I want from one of my files. This part is not important.
returns_matrix = get_returns_matrix(asset_returns, 60, date, components)
This is the optimization.
for date in rebalance_dates:
print(date)
returns_matrix = get_returns_matrix(asset_returns, 60, date, components)
covariance = np.cov(returns_matrix)
annual_covar = [map(lambda x:x * 260, group) for group in covariance]
annual_covar = [list(x) for x in annual_covar]
cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1.0},
{'type': 'ineq', 'fun': lambda x: x})
res= minimize(risk_budget_objective, initial_weight, args=[annual_covar, risk_contrib_target], method='SLSQP',constraints=cons,
options={'disp': False, 'ftol': .00000000001, 'eps' : .0000000000000005, 'maxiter':1000})
if res.fun > .00000000001:
print("Kick to basin hopping")
minimizer_kwargs = dict(method="SLSQP", constraints=cons, args=[annual_covar, risk_contrib_target], options={'ftol': .000000000000000000001, 'eps' : .0000000000000005, 'maxiter':100})
res = basinhopping(risk_budget_objective, initial_weight, niter=50, minimizer_kwargs=minimizer_kwargs)
I have two constraints, one being the sum of weights needs to equal 100% and the other being all weights should be positive.
This solves correctly about 75% of the time, the other times it gets stuck at a local minimum I believe. So a correct result from this would look like:
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Category|Stock 1 |Stock 2 |Stock 3 |Stock 4 |Stock 5 |Stock 6 |
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Weights |0.121465654|0.17829418 |0.091558469|0.105659033|0.156959021|0.346063642|
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Risk Con|0.166666667|0.166666667|0.166666667|0.166666667|0.166666667|0.166666667|
Function return val 0.0000000000
But occasionally (25% of the times) I will get a result that does not solve the function, like this:
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Category|Stock 1 |Stock 2 |Stock 3 |Stock 4 |Stock 5 |Stock 6 |
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Weights |0.159442825|0.166949713|0.235404372|0.175430619|0.262772472|0.000000000|
|--------|-----------|-----------|-----------|-----------|-----------|-----------|
|Risk Con|0.199661774|0.199803048|0.200448716|0.199943667|0.200142796|0.000000000|
Function return val 33.33371143
The times that it is wrong, it seems to completely disregard stock 6. Giving it both a 0 weight and a 0 risk contribution.
Is there any parameter I am not using correctly in the solver? Sorry, this might be a little difficult to solve without the data that I'm using. But just wondering if there is anything obviously wrong with my approach.
I also happen to know there is a solution to the ones scipy doesn't solve correctly because I can do the same thing correctly in an excel spreadsheet with the GRG-nonlinear constraint solver.
Thanks so much!
Basinhopping is a stochastic global optimizer. There is no guarantee that it will find the global optimum within the specified number of iterations.
It sounds like from your description that you have a way of checking whether a solution is the global optimum. In that case you can use the callback parameter to optimize your search
callback : callable, callback(x, f, accept), optional
A callback function which will be called for all minima found. x and f are the coordinates and function value of the trial minimum, and accept is whether or not that minimum was accepted. This can be used, for example, to save the lowest N minima found. Also, callback can be used to specify a user defined stop criterion by optionally returning True to stop the basinhopping routine.
def my_callback(x, f, accept):
return minimum_is_global_minimum(x)
Then you can set niter to some large number and it will stop as soon as it finds the gobal minimum.