I have a Python script (Pulp library) for allocating funds among a number of clients depending on their current level of funding (gap/requirements) and their membership to priority groups. However, I am not receiving the expected results.
In particular, I want:
All allocations must be positive and their sum should be equal to the total available money I have.
I want to minimize the target funding gap for the most vulnerable group (group A) and then I want that the target gap % in the less vulnerable group increase of 10%: (for group B = funding gap A1.1, for group C = funding gap B1.1...).
I have tried this:
"""
DECISION VARIABLES
"""
# Create a continuous Decision Variable and Affine Expression for the amount of additional funding received by each
# project
allocation = {}
allocation_expr = LpAffineExpression()
for z in range(n):
if priority[z] == 'X' or (requirements[z] == 0 and skip_zero_requirements):
# Projects in Priority Group 'X' don't get any allocation
allocation[project_names[z]] = pulp.LpVariable(f'allocation_{project_names[z]}', lowBound=0, upBound=0)
else:
# allocation is non negative and cannot be greater than the initial gap
allocation[project_names[z]] = pulp.LpVariable(f'allocation_{project_names[z]}', lowBound=0, upBound=(gap[z]))
allocation_expr += allocation[project_names[z]]
# Create a continuous Decision Variable and Affine Expression for the maximum GAP% within each priority group
target_group_A_expr = LpAffineExpression()
target_group_A = pulp.LpVariable(f'allocation', lowBound=0 )
target_group_A_expr += target_group_A
"""
LINEAR PROGRAMMING PROBLEM
"""
# Create the linear programming problem object
lp_prob = pulp.LpProblem('Multi-Objective Optimization', pulp.LpMaximize)
"""
OBJECTIVE FUNCTIONS
"""
# Define the objective function as an LpAffineExpression
obj = LpAffineExpression()
# MAXIMIZE the portion of additional funding allocated to projects
obj += allocation_expr
# MINIMIZE the Max GAP% within each group [actually Maximizing the -(Max GAP%)]
obj += -target_group_A_expr
# Set the Objective Function
lp_prob += obj
"""
CONSTRAINTS
"""
# Additional funding allocations to individual projects must be non-negative and not greater than the project's gap
#for v in range(n):
# lp_prob += allocation[project_names[v]] <= gap[v]
# lp_prob += allocation[project_names[v]] >= 0
# The sum of allocations to individual projects cannot be greater than the additional funding
lp_prob += pulp.lpSum([allocation[project_names[u]] for u in range(n)]) <= additional_funding
# The Max GAP % within each group >= of the GAP % of all projects in the group (proxy for dynamic max calculation)
for i, (p, group) in enumerate(priority_groups.items()):
# Get the indices of the projects in the group
group_indices = priority_groups[p] #selects the indices matching with the rows of the projects belonging to that group
# Iterate over the indices of the projects in the group
for index in group:
# Create an LpAffineExpression for the GAP% of the project
project_gap_percentage = LpAffineExpression()
if requirements[index] == 0:
project_gap_percentage += 0
else:
project_gap_percentage += (gap[index] - allocation[project_names[index]]) / requirements[index]
# Add constraint to the model
lp_prob += target_group_A == (project_gap_percentage/pow(delta_gap, i))
"""
PROGRAMMING MODEL SOLVER
"""
# Solve the linear programming problem
lp_prob.solve()
delta_gap and the additional_funding are external parameters.
I receive even negative allocations and the constrains is not always meet, e.g. in group B, C I reach level of funding gap much lower than the level of group A- sometimes they randomly go to zero. How can this be possible?
I am considering to use another library, any suggestions?
Related
I am trying to solve a capacitated routing problem where I have a set of nodes which require different amounts and different types of items.
In addition I want to allow node drops, because all nodes with one type of item might still exceed the vehicle capacity and thus would lead to no solution.
However eventually all nodes should be served so I use an iterative approach where I was treating each item type as individual routing problem.
But I was wondering if one could use disjunctions or something similar to solve the 'global' routing problem. Any help on whether this is possible is appreciated.
Example:
Node 1 - item A - demand 10
Node 2 - item A - demand 10
Node 3 - item A - demand 12
Node 4 - item B - demand 10
Node 5 - item B - demand 10
vehicle I - capacity 20
vehicle II - capacity 10
My approach:
First solve for item A: vehicle I serves node 1 & 2, node 3 is dropped, save dropped nodes for later iteration
Then solve for item B: vehicle I serves nodes 4 & 5, vehicle II is idle
Solve for remaining node 3: vehicle I serves node 3
EDIT
I adjusted my approach to fit #mizux answer. Below the code:
EDIT2 Fixed a bug where the demand callback function from the first loops would still reference the product_index variable and thus return the wrong demand. Fix by using functools.partial.
import functools
from ortools.constraint_solver import pywrapcp, routing_enums_pb2
class CVRP():
def __init__(self, data):
# assert all(data['demands'] < max(data['vehicle_capacities'])) # if any demand exceeds cap no solution possible
self.data = data
self.vehicle_names_internal = [f'{i}:{j}' for j in data['products'] for i in data['vehicle_names']]
self.manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']), len(self.vehicle_names_internal), data['depot'])
self.routing = pywrapcp.RoutingModel(self.manager)
transit_callback_id = self.routing.RegisterTransitCallback(self._dist_callback)
self.routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_id)
# set up dimension for each product type for vehicle capacity constraint
for product_index, product in enumerate(data['products']):
dem_product_callback = functools.partial(self._dem_callback_generic, product_index=product_index)
dem_callback_id = self.routing.RegisterUnaryTransitCallback(dem_product_callback)
vehicle_product_capacity = [0 for i in range(len(self.vehicle_names_internal))]
vehicle_product_capacity[product_index*data['num_vehicles']:product_index*data['num_vehicles']+data['num_vehicles']] = data['vehicle_capacities']
print(product_index, product)
print(self.vehicle_names_internal)
print(vehicle_product_capacity)
self.routing.AddDimensionWithVehicleCapacity(
dem_callback_id,
0,
vehicle_product_capacity,
True,
f'capacity_{product}',
)
# disjunction (allow node drops)
penalty = int(self.data['distance_matrix'].sum()+1) # penalty needs to be higher than total travel distance in order to only drop locations if not other feasible solution
for field_pos_idx_arr in self.data['disjunctions']:
self.routing.AddDisjunction([self.manager.NodeToIndex(i) for i in field_pos_idx_arr], penalty)
def _dist_callback(self, i, j):
return self.data['distance_matrix'][self.manager.IndexToNode(i)][self.manager.IndexToNode(j)]
def _dem_callback_generic(self, i, product_index):
node = self.manager.IndexToNode(i)
if node == self.data['depot']:
return 0
else:
return self.data['demands'][node, product_index]
def solve(self, verbose=False):
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.AUTOMATIC)
search_parameters.local_search_metaheuristic = (
routing_enums_pb2.LocalSearchMetaheuristic.AUTOMATIC)
search_parameters.time_limit.FromSeconds(30)
self.solution = self.routing.SolveWithParameters(search_parameters)
You should create two capacity dimensions, one for each type,
At each location you increase the relevant dimension.
You can duplicate your vehicle for each item type i.e.:
v0, Vehicle 1 Type A with: capacity A: 20, capacity B: 0
v1, Vehicle 1 Type B with: capacity A: 0, capacity B: 20
v2, Vehicle 2 Type A with: capacity A: 10, capacity B: 0
v3, Vehicle 2 Type B with: capacity A: 0, capacity B: 10
note: you can replicate it to allow multi-trips
You can create a "gate" node to allow only one vehicle configuration.
e.g. To only allow v0 or v1 to do some visit
v0_start = routing.Start(0)
v0_end = routing.End(0)
v1_start = routing.Start(1)
v1_end = routing.End(1)
gate_index = manager.NodeToIndex(gate_index)
routing.NextVar(v0_start).setValues[gate_index, v0_end]
routing.NextVar(v1_start).setValues[gate_index, v1_end]
Since node can only be visited once, one vehicle among v0 and v1 can pass by the gate node while the other has no choice but to go to end node i.e. empty route you can remove when post processing the assignment.
You can also add a vehicle FixedCost to incentive solver to use vehicle II if it is cheaper than vehicle I etc...
Add each location to a disjunction so the solver can drop them if needed
location_index = manager.NodeToIndex(location_id)
routing.AddDisjunction(
[location_index], # locations
penalty,
max_cardinality=1 # you can omit it since it is already 1 by default
)
I am using a genetic algorithm implemented with the DEAP library for Python. In order to avoid premature convergence, and to force exploration of the feature space, I would like the mutation probability to be high during the first generations. But to prevent drifting away from extremum once they are identified, I would like the mutation probability to be lower in the last generations. How do I get the mutation probability to decrease over the generations? Is there any built-in function in DEAP to get this done?
When I register a mutation function, for instance
toolbox.register('mutate', tools.mutPolynomialBounded, eta=.6, low=[0,0], up=[1,1], indpb=0.1)
the indpb parameter is a float. How can I make it a function of something else?
Sounds like a job for Callbackproxy which evaluates function arguments each time they are called. I added a simple example where I modified the official DEAP n-queen example
such that the mutation rate is set to 2/N_GENS (arbitrary choice just to make the point).
Notice that Callbackproxy receives a lambda, so you have to pass the mutation rate argument as a function (either using a fully blown function or just a lambda). The result anyway is that each time the indpb parameter is evaluated this lambda will be called, and if the lambda contains reference to a global variable generation counter, you got what you want.
# This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see <http://www.gnu.org/licenses/>.
import random
from objproxies import CallbackProxy
import numpy
from deap import algorithms
from deap import base
from deap import creator
from deap import tools
# Problem parameter
NB_QUEENS = 20
N_EVALS = 0
N_GENS = 1
def evalNQueens(individual):
global N_EVALS, N_GENS
"""Evaluation function for the n-queens problem.
The problem is to determine a configuration of n queens
on a nxn chessboard such that no queen can be taken by
one another. In this version, each queens is assigned
to one column, and only one queen can be on each line.
The evaluation function therefore only counts the number
of conflicts along the diagonals.
"""
size = len(individual)
# Count the number of conflicts with other queens.
# The conflicts can only be diagonal, count on each diagonal line
left_diagonal = [0] * (2 * size - 1)
right_diagonal = [0] * (2 * size - 1)
# Sum the number of queens on each diagonal:
for i in range(size):
left_diagonal[i + individual[i]] += 1
right_diagonal[size - 1 - i + individual[i]] += 1
# Count the number of conflicts on each diagonal
sum_ = 0
for i in range(2 * size - 1):
if left_diagonal[i] > 1:
sum_ += left_diagonal[i] - 1
if right_diagonal[i] > 1:
sum_ += right_diagonal[i] - 1
N_EVALS += 1
if N_EVALS % 300 == 0:
N_GENS += 1
return sum_,
creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
creator.create("Individual", list, fitness=creator.FitnessMin)
# Since there is only one queen per line,
# individual are represented by a permutation
toolbox = base.Toolbox()
toolbox.register("permutation", random.sample, range(NB_QUEENS), NB_QUEENS)
# Structure initializers
# An individual is a list that represents the position of each queen.
# Only the line is stored, the column is the index of the number in the list.
toolbox.register("individual", tools.initIterate, creator.Individual, toolbox.permutation)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
toolbox.register("evaluate", evalNQueens)
toolbox.register("mate", tools.cxPartialyMatched)
toolbox.register("mutate", tools.mutShuffleIndexes, indpb=CallbackProxy(lambda: 2.0 / N_GENS))
toolbox.register("select", tools.selTournament, tournsize=3)
def main(seed=0):
random.seed(seed)
pop = toolbox.population(n=300)
hof = tools.HallOfFame(1)
stats = tools.Statistics(lambda ind: ind.fitness.values)
stats.register("Avg", numpy.mean)
stats.register("Std", numpy.std)
stats.register("Min", numpy.min)
stats.register("Max", numpy.max)
algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=1, ngen=100, stats=stats,
halloffame=hof, verbose=True)
return pop, stats, hof
if __name__ == "__main__":
main()
I have a list which consist of 2 attributes which are cost and rating. I need to find possible best flights which have lower cost and a higher rating. This is a multi objecting optimization problem with minimizing and maximizing objectives. How can I implement this in DEAP?
I'm struggling to implement Individual since im very new to DEAP.
# Attribute generator
toolbox.register("cost", random.randrange, NBR_ITEMS)
toolbox.register("rating", random.randrange, NBR_ITEMS)
# Structure initializers
toolbox.register("individual", tools.initRepeat, creator.Individual, toolbox.cost, toolbox.rating) #
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
Maybe you could try to encode the cost and the rating using binary numbers.
For example, let's assume the most expensive plane ticket you can get is $16384, you could store that in 14 bits (2^14 = 16384) and the rating is a number from 0 to 10 so you can store that in 4 bits so in total you can store your individuals using 18 bits.
Now you need a function to decode it:
def decode_individual(individual):
decoded_individual = ['', '']
# Decode cost (14 bits)
for i in range(0, 14):
decoded_individual[0] += str(individual[i])
# Decode rating (4 bits)
for i in range(1, 3):
decoded_individual[0] += str(individual[13 + i])
return tuple(map(lambda x: int(x, 2), decoded_individual))
You need to set up your fitness functions for a multi-objective problem, i.e. you need to provide some weights for each function that is positive if you are trying to maximize the function and negative if you are trying to minimize it. In your case, I guess you are trying to maximize de rating and minimize the cost so you could set it up as follows:
creator.create('Fitness', base.Fitness, weights=(1.0, -0.5,))
creator.create('Individual', list, fitness=creator.Fitness)
Your fitness methods should return the result of the functions you are trying to maximize and minimize in the same order as specified in the weights:
def function_cost(individual):
decoded_individual = decode_individual(individual)
return decoded_individual[0]
def function_rating(individual):
decoded_individual = decode_individual(individual)
return decoded_individual[1]
def fitness(individual):
return (function_cost(individual), function_rating(individual)),
Then, instead of registering 2 fitness functions like in your example, register just one:
toolbox.register('evaluate', fitness)
Configure DEAP to use binary data:
toolbox.register('attrBinary', random.randint, 0, 1)
toolbox.register('individual', tools.initRepeat, creator.Individual, toolbox.attrBinary, n=18) # Here you need to specify the number of bits you are using
toolbox.register('population', tools.initRepeat, list, toolbox.individual)
# Register the evaluation function (was named fitness in this example)
toolbox.register('evaluate', fitness)
# Configure your mate and mutation methods, e.g.
toolbox.register('mate', tools.cxTwoPoint)
toolbox.register('mutate', tools.mutFlipBit, indpb=0.15)
Your selection method must support multi-objective problems, NSGA2 as pointed out in your question can be used:
toolbox.register('select', tools.selNSGA2)
Then run the algorithm, you can try different values for the number of individuals (population) the number of generations and the ratings of mating and mutation:
num_pop = 50
num_gen = 100
cx_prob = 0.7
mut_prob = 0.2
best = []
for gen in range(num_gen):
offspring = algorithms.varAnd(population, toolbox, cxpb=cx_prob, mutpb=mut_prob)
fits = toolbox.map(toolbox.evaluate, offspring)
for fit, ind in zip(fits, offspring):
ind.fitness.values = fit
population = toolbox.select(offspring, k=len(population))
top = tools.selBest(population, k=1)
fitness = fitness(top[0])
print(gen, fitness, decode_individual(top[0]), top[0])
best.append(fitness[0])
You may also want to display the best individuals of each generation in a graph:
x = list(range(num_gen))
plt.plot(x, best)
plt.title("Best ticket - Cost / Rating")
plt.show()
Haven't tested this myself and I got largely inspired by some exercise I did at University so hopefully it will work for you.
The following is the objective function:
The idea is that a mean-variance optimization has already been done on a universe of securities. This gives us the weights for a target portfolio. Now suppose the investor already is holding a portfolio and does not want to change their entire portfolio to the target one.
Let w_0 = [w_0(1),w_0(2),...,w_0(N)] be the initial portfolio, where w_0(i) is the fraction of the portfolio invested in
stock i = 1,...,N. Let w_t = [w_t(1), w_t(2),...,w_t(N)] be the target portfolio, i.e., the portfolio
that it is desirable to own after rebalancing. This target portfolio may be constructed using quadratic optimization techniques such as variance minimization.
The objective is to decide the final portfolio w_f = [w_f (1), w_f (2),..., w_f(N)] that satisfies the
following characteristics:
(1) The final portfolio is close to our target portfolio
(2) The number of transactions from our initial portfolio is sufficiently small
(3) The return of the final portfolio is high
(4) The final portfolio does not hold many more securities that our initial portfolio
An objective function which is to be minimized is created by summing together the characteristic terms 1 through 4.
The first term is captured by summing the absolute difference in weights from the final and the target portfolio.
The second term is captured by the sum of an indicator function multiplied by a user specified penalty. The indicator function is y_{transactions}(i) where it is 1 if the weight of security i is different in the initial portfolio and the final portfolio, and 0 otherwise.
The third term is captured by the total final portfolio return multiplied by a negative user specified penalty since the objective is minimization.
The final term is the count of assets in the final portfolio (ie. sum of an indicator function counting the number of positive weights in the final portfolio), multiplied by a user specified penalty.
Assuming that we already have the target weights as target_w how do I setup this optimization problem in docplex python library? Or if anyone is familiar with mixed integer programming in NAG it would be helpful to know how to setup such a problem there as well.
`
final_w = [0.]*n
final_w = np.array(final_w)
obj1 = np.sum(np.absolute(final_w - target_w))
pen_trans = 1.2
def ind_trans(final,inital):
list_trans = []
for i in range(len(final)):
if abs(final[i]-inital[i]) == 0:
list_trans.append(0)
else:
list_trans.append(1)
return list_trans
obj2 = pen_trans*sum(ind_trans(final_w,initial_w))
pen_returns = 0.6
returns_np = np.array(df_secs['Return'])
obj3 = (-1)*np.dot(returns_np,final_w)
pen_count = 1.
def ind_count(final):
list_count = []
for i in range(len(final)):
if final[i] == 0:
list_count.append(0)
else:
list_count.append(1)
return list_count
obj4 = sum(ind_count(final_w))
objective = obj1 + obj2 + obj3 + obj4
The main issue in your code is that final_w is not a an array of variables but an array of data. So there will be nothing to optimize. To create an array of variables in docplex you have to do something like this:
from docplex.mp.model import Model
with Model() as m:
final = m.continuous_var_list(n, 0.0, 1.0)
That creates n variables that can take values between 0 and 1. With that in hand you can start things. For example:
obj1 = m.sum(m.abs(initial[i] - final[i]) for i in range(n))
For the next objective things become harder since you need indicator constraints. To simplify definition of these constraints first define a helper variable delta that gives the absolute difference between stocks:
delta = m.continuous_var_list(n, 0.0, 1.0)
m.add_constraints(delta[i] == m.abs(initial[i] - final[i]) for i in range(n))
Next you need an indicator variable that is 1 if a transaction is required to adjust stock i:
needtrans = m.binary_var_list(n)
for i in range(n):
# If needtrans[i] is 0 then delta[i] must be 0.
# Since needtrans[i] is penalized in the objective, the solver will
# try hard to set it to 0. It will only set it to 1 if delta[i] != 0.
# That is exactly what we want
m.add_indicator(needtrans[i], delta[i] == 0, 0)
With that you can define the second objective:
obj2 = pen_trans * m.sum(needtrans)
once all objectives have been defined, you can add their sum to the model:
m.minimize(obj1 + obj2 + obj3 + obj4)
and then solve the model and display its solution:
m.solve()
print(m.solution.get_values(final))
If any of the above is not (yet) clear to you then I suggest you take a look at the many examples that ship with docplex and also at the (reference) documentation.
I am working on a code to solve for the optimum combination of diameter size of number of pipelines. The objective function is to find the least sum of pressure drops in six pipelines.
As I have 15 choices of discrete diameter sizes which are [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80] that can be used for any of the six pipelines that I have in the system, the list of possible solutions becomes 15^6 which is equal to 11,390,625
To solve the problem, I am using Mixed-Integer Linear Programming using Pulp package. I am able to find the solution for the combination of same diameters (e.g. [2,2,2,2,2,2] or [4,4,4,4,4,4]) but what I need is to go through all combinations (e.g. [2,4,2,2,4,2] or [4,2,4,2,4,2] to find the minimum. I attempted to do this but the process is taking a very long time to go through all combinations. Is there a faster way to do this ?
Note that I cannot calculate the pressure drop for each pipeline as the choice of diameter will affect the total pressure drop in the system. Therefore, at anytime, I need to calculate the pressure drop of each combination in the system.
I also need to constraint the problem such that the rate/cross section of pipeline area > 2.
Your help is much appreciated.
The first attempt for my code is the following:
from pulp import *
import random
import itertools
import numpy
rate = 5000
numberOfPipelines = 15
def pressure(diameter):
diameterList = numpy.tile(diameter,numberOfPipelines)
pressure = 0.0
for pipeline in range(numberOfPipelines):
pressure += rate/diameterList[pipeline]
return pressure
diameterList = [2,4,6,8,12,16,20,24,30,36,40,42,50,60,80]
pipelineIds = range(0,numberOfPipelines)
pipelinePressures = {}
for diameter in diameterList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(diameter))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[diameter] = pressureList
print 'pipepressure', pipelinePressures
prob = LpProblem("Warehouse Allocation",LpMinimize)
use_diameter = LpVariable.dicts("UseDiameter", diameterList, cat=LpBinary)
use_pipeline = LpVariable.dicts("UsePipeline", [(i,j) for i in pipelineIds for j in diameterList], cat = LpBinary)
## Objective Function:
prob += lpSum(pipelinePressures[j][i] * use_pipeline[(i,j)] for i in pipelineIds for j in diameterList)
## At least each pipeline must be connected to a diameter:
for i in pipelineIds:
prob += lpSum(use_pipeline[(i,j)] for j in diameterList) ==1
## The diameter is activiated if at least one pipelines is assigned to it:
for j in diameterList:
for i in pipelineIds:
prob += use_diameter[j] >= lpSum(use_pipeline[(i,j)])
## run the solution
prob.solve()
print("Status:", LpStatus[prob.status])
for i in diameterList:
if use_diameter[i].varValue> pressureTest:
print("Diameter Size",i)
for v in prob.variables():
print(v.name,"=",v.varValue)
This what I did for the combination part which took really long time.
xList = np.array(list(itertools.product(diameterList,repeat = numberOfPipelines)))
print len(xList)
for combination in xList:
pressures = []
for pipeline in range(numberOfPipelines):
pressures.append(pressure(combination))
pressureList = dict(zip(pipelineIds,pressures))
pipelinePressures[combination] = pressureList
print 'pipelinePressures',pipelinePressures
I would iterate through all combinations, I think you would run into memory problems otherwise trying to model ALL combinations in a MIP.
If you iterate through the problems perhaps using the multiprocessing library to use all cores, it shouldn't take long just remember only to hold information on the best combination so far, and not to try and generate all combinations at once and then evaluate them.
If the problem gets bigger you should consider Dynamic Programming Algorithms or use pulp with column generation.