I am attempting to understand the behavior of the constraints in scipy.optimize.minimize:
First, I create 4 assets and 100 scenarios of returns. The average returning funds are in order best to worse D > B > A > C
#seed first
np.random.seed(1)
df_returns = pd.DataFrame(np.random.rand(100,4) - 0.25, columns =list('ABCD'))
df_returns.head()
A B C D
0 0.167022 0.470324 -0.249886 0.052333
1 -0.103244 -0.157661 -0.063740 0.095561
2 0.146767 0.288817 0.169195 0.435220
3 -0.045548 0.628117 -0.222612 0.420468
4 0.167305 0.308690 -0.109613 -0.051899
and a set of weights
weights = pd.Series([0.25, 0.25, 0.25, 0.25], index=list('ABCD'))
0
A 0.25
B 0.25
C 0.25
D 0.25
we create an objective function:
def returns_objective_function(weights, df_returns):
result = -1. * (df_returns * weights).mean().sum()
return result
and constraints and bounds
cons = ({'type': 'eq', 'fun': lambda weights: np.sum(weights) -1 })
bnds = ((0.01, .8), (0.01, .8), (0.01, .8), (0.01, .75))
Let's optimize
optimize.minimize(returns_objective_function, weights, (df_returns),
bounds=bnds, constraints=cons, method= 'SLSQP')
And we get success.
status: 0
success: True
njev: 8
nfev: 48
fun: -0.2885398923185326
x: array([ 0.01, 0.23, 0.01, 0.75])
message: 'Optimization terminated successfully.'
jac: array([-0.24384782, -0.2789166 , -0.21977262, -0.29300382, 0. ])
nit: 8
Now I wish to add constraints starting with a basic inequality:
scipy.optimize.minimize documentation states
Equality constraint means that the constraint function result is to be zero whereas inequality means that it is to be non-negative.
cons = (
{'type': 'eq', 'fun': lambda weights: np.sum(weights) -1 }
,{'type': 'ineq', 'fun': lambda weights: np.sum(weights) + x}
)
Depending on x, I get unexpected behavior.
x = -100
Based on the bounds, weights can be a maximum of 3.15 and, of course, must sum to 1 by the first equality constraint np.sum(weights) - 1, but, as a result, np.sum(weights) + x would always be negative. I believe no solution should be found, yet scipy.optimize.minimize returns success.
With a simpler model I get the same behavior:
x = [1,2]
optimize.minimize(
lambda x: x[0]**2+x[1]**2,
x,
constraints = (
{'type':'eq','fun': lambda x: x[0]+x[1]-1},
{'type':'ineq','fun': lambda x: x[0]-2}
),
bounds = ((0,None),(0,None)),
method='SLSQP')
with results:
nfev: 8
fun: 2.77777777777712
nit: 6
jac: array([ 3.33333334e+00, 2.98023224e-08, 0.00000000e+00])
x: array([ 1.66666667e+00, 1.39888101e-14])
success: True
message: 'Optimization terminated successfully.'
status: 0
njev: 2
There should be some flag that this is an infeasible solution.
SLSQP is also available from R:
> slsqp(c(1,2),
+ function(x) {x[1]^2+x[2]^2},
+ heq=function(x){x[1]+x[2]-1},
+ hin=function(x){x[1]-2},
+ lower=c(0,0))
$par
[1] 1.666667e+00 4.773719e-11
$value
[1] 2.777778
$iter
[1] 105
$convergence
[1] -4
$message
[1] "NLOPT_ROUNDOFF_LIMITED: Roundoff errors led to a breakdown of the optimization algorithm. In this case, the returned minimum may still be useful. (e.g. this error occurs in NEWUOA if one tries to achieve a tolerance too close to machine precision.)"
At least we see some warning signals here.
Related
hi im trying to minimize a simple 3 variable function with some range costraints in the x variables .. but im getting 'Inequality constraints incompatible - any idea ? thanks !!
from scipy.optimize import minimize
def f(x):
return (int(558*x[0]*x[1]*x[2])-(x[2]*(558-int(558*x[0])))-(x[2]*558))
x0 = [0.4, 1.0, 2.0]
#real data Ranges
#x[0] 0..1
#x[1] 1..3
#x[2] 5..50
cons=(
{'type': 'ineq','fun': lambda x: x[0]},
{'type': 'ineq','fun': lambda x: 1-x[0]},
{'type': 'ineq','fun': lambda x: x[1]-1},
{'type': 'ineq','fun': lambda x: 3-x[1]},
{'type': 'ineq','fun': lambda x: x[2]-5},
{'type': 'ineq','fun': lambda x: 50-x[2]}
)
res = minimize(f, x0, constraints=cons)
print(res)
full result is
fun: -33490.99993615066
jac: array([ 6.7108864e+07, 6.7108864e+07, -8.9300000e+02])
message: 'Inequality constraints incompatible'
nfev: 8
nit: 2
njev: 2
status: 4
success: False
x: array([ 0.4 , 1. , 49.99999993])
Hello I suspect that the problem comes from the numerical method used.
By default with constraints, minimize use Sequential Least Squares Programming (SLSQP) which is a gradient method. Therefore the function to be minimized has to be regular, but given your use of int it is not.
Using the alternative method: Constrained Optimization BY Linear Approximation (COBYLA) which is of a different nature I get the following
from scipy.optimize import minimize
def f(x):
return (int(558*x[0]*x[1]*x[2])-(x[2]*(558-int(558*x[0])))-(x[2]*558))
x0 = [0.4, 1.0, 2.0]
#real data Ranges
#x[0] 0..1
#x[1] 1..3
#x[2] 5..50
cons=(
{'type': 'ineq','fun': lambda x: x[0]},
{'type': 'ineq','fun': lambda x: 1-x[0]},
{'type': 'ineq','fun': lambda x: x[1]-1},
{'type': 'ineq','fun': lambda x: 3-x[1]},
{'type': 'ineq','fun': lambda x: x[2]-5},
{'type': 'ineq','fun': lambda x: 50-x[2]}
)
res = minimize(f, x0, constraints=cons, method="cobyla")
print(res)
with the display
fun: -55800.0
maxcv: 7.395570986446986e-32
message: 'Optimization terminated successfully.'
nfev: 82
status: 1
success: True
x: array([-7.39557099e-32, 1.93750000e+00, 5.00000000e+01])
I'm trying to replicate this simple optimisation problem in order to get started with Scipy.optimize.
The problem is a classic product mix-problem, where the objective is to maximize profit given some production and ingredients constraints, in this case it is a coffee shop with three coffee types.
x0 -> Regular
x1 -> Latte
x2 -> Mocha
Constraints:
x0+x1+x2 <= 500
x1+x2 <= 350
x2 <= 125
Objective function
Maximize x0*1.5 + x1*2.00 + x2*2.25
Here is my code (please note that I run it in a notebook):
from scipy.optimize import minimize
#profit / cup
Reg = 1.25
Lat = 2.00
Moc = 2.25
#objective function to maximize
def objective(x):
return (x[0]*Reg + x[1]*Lat + x[2]*Moc)
#constraints
def cons_total_production(x):
return (sum(x))-500
def cons_choc(x):
return (x[2])-125
def cons_milk(x):
return (sum(x[1:]))-350
cons1 = {'type':'ineq', 'fun':cons_total_production}
cons2 = {'type':'ineq', 'fun':cons_choc}
cons3 = {'type':'ineq', 'fun':cons_milk}
cons = [cons1,cons2,cons3]
#boundries
bnds = ((0,None),(0,None),(0,None))
#initial guess
x0 = [250,150,100]
#let scipy do its magic
sol = minimize(objective, x0, constraints=cons, bounds=bnds)
This produces the correct output. How is that even possible when i use scipy minimize?
fun: 918.7499999999358
jac: array([ 1.25, 2. , 2.25])
message: 'Optimization terminated successfully.'
nfev: 50
nit: 10
njev: 10
status: 0
success: True
x: array([ 150., 225., 125.])
But when I try to add one more constrain, the output is wrong. If I for instance would make a constrain that states that x0 must be equal to x1, I would change the following and run the model again:
def cons_eq(x):
return x[0]-x[1]
cons4 = {'type':'eq', 'fun':cons_eq}
cons = [cons1, cons2, cons3, cons4]
But now my constrain x2<=125 is ignored:
fun: 937.4999999999934
jac: array([ 1.25, 2. , 2.25])
message: 'Optimization terminated successfully.'
nfev: 35
nit: 7
njev: 7
status: 0
success: True
x: array([ 150., 150., 200.])
Any suggestions? thx...
The problem lies in both your objective function and the constraints. Since you are using scipy's minimize function, you must minimize the negative of the objective function to find the maximum of the function (It is slightly tricky).
#objective function to maximize
def objective(x):
return -1.0*(x[0]*Reg + x[1]*Lat + x[2]*Moc)
You have also incorrectly typed the inequality functions. If you look at the scipy documentation, all inequalities are in the form g(x)>=0. So, for example, if you want x2 <= 125, you must multiply by negative 1 (to switch the inequality), then add 125 which gets: g(x) = 125-x2 >= 0. So for the rest of your constraints:
#constraints
def cons_total_production(x):
return (-1.0*sum(x))+500
def cons_choc(x):
return (-1.0*x[2])+125
def cons_milk(x):
return (-1.0*sum(x[1:]))+350
Which will give you the outputs:
fun: -918.7499999999989
jac: array([-1.25, -2. , -2.25])
message: 'Optimization terminated successfully.'
nfev: 35
nit: 7
njev: 7
status: 0
success: True
x: array([ 150., 225., 125.])
fun: -890.62499998809
jac: array([-1.25, -2. , -2.25])
message: 'Optimization terminated successfully.'
nfev: 10
nit: 2
njev: 2
status: 0
success: True
x: array([ 187.5, 187.5, 125. ])
Linear programming is so cool!
Hope it helps :)
I would like to optimize the following formula with scipy adding the constraint of x[0] - x[1] > 0. When printing this expression in the objective function it gives negative values as well, while the optimization terminates successfully. The final goal would be something like minimizing the sqrt(0.1*x[0]*x[1]) which due to math error fails.
import numpy as np
from scipy.optimize import minimize
def f(x):
print x[0] - x[1]
#return sqrt(0.1*x[0]*x[1])
return 0.1*x[0]*x[1]
def ineq_constraint(x):
return x[0] - x[1]
con = {'type': 'ineq', 'fun': ineq_constraint}
x0 = [1, 1]
res = minimize(f, x0, method='SLSQP', constraints=con)
print res
And the output:
0.0
0.0
1.49011611938e-08
-1.49011611938e-08
0.0
0.0
1.49011611938e-08
-1.49011611938e-08
0.0
0.0
1.49011611938e-08
-1.49011611938e-08
4.65661176285e-10
4.65661176285e-10
1.53668223701e-08
-1.44355000176e-08
fun: 1.7509862319755833e-18
jac: array([ 3.95812066e-10, 4.42378184e-10, 0.00000000e+00])
message: 'Optimization terminated successfully.'
nfev: 16
nit: 4
njev: 4
status: 0
success: True
x: array([ 4.42378184e-09, 3.95812066e-09])
In the general case, we don't know your whole task, constraints are not enforced at all steps (as observed)! Without changing the optimizer there is not much to do. And even finding an appropriate optimizer is maybe not easy.
For your case, it would work if your variables are nonnegative! If that's something you can use in your other task, we don't know.
Now there are two approaches for nonnegativity:
inequalities
bounds
Using bounds, explicit handling is used (as far is i know) and those will not be violated during optimization.
Example:
import numpy as np
from scipy.optimize import minimize
from math import sqrt
def f(x):
print(x)
return sqrt(0.1*x[0]*x[1])
def ineq_constraint(x):
return x[0] - x[1]
con = {'type': 'ineq', 'fun': ineq_constraint}
x0 = [1, 1]
res = minimize(f, x0, method='SLSQP', constraints=con, bounds=[(0, None) for i in range(len(x0))])
print(res)
Output:
[1. 1.]
[1. 1.]
[1.00000001 1. ]
[1. 1.00000001]
[0.84188612 0.84188612]
[0.84188612 0.84188612]
[0.84188613 0.84188612]
[0.84188612 0.84188613]
[0.05131671 0.05131669]
[0.05131671 0.05131669]
[0.05131672 0.05131669]
[0.05131671 0.0513167 ]
[0. 0.]
[0. 0.]
[1.49011612e-08 0.00000000e+00]
[0.00000000e+00 1.49011612e-08]
fun: 0.0
jac: array([0., 0.])
message: 'Optimization terminated successfully.'
nfev: 16
nit: 4
njev: 4
status: 0
success: True
x: array([0., 0.])
I am solving the following optimization problem:
with this Python code:
from scipy.optimize import minimize
import math
def f(x):
return math.log(x[0]**2 + 1) + x[1]**4 + x[0]*x[2]
x0 = [0, 0, 0]
cons=({'type': 'ineq',
'fun': lambda x: x[0]**3 - x[1]**2 - 1},
{'type': 'ineq',
'fun': lambda x: x[0]},
{'type': 'ineq',
'fun': lambda x: x[2]})
res = minimize(f, x0, constraints=cons)
print res
I am getting an error
message: 'Inequality constraints incompatible'
What can cause this error?
The issue seems to be with your initial guess. If I change your starting values to
x0 = [1.0, 1.0, 1.0]
Then your code will execute fine (at least on my machine)
Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32
message: 'Optimization terminated successfully.'
njev: 10
jac: array([ 1., 0., 1., 0.])
fun: 0.6931471805582502
nit: 10
status: 0
x: array([ 1.00000000e+00, -1.39724765e-06, 1.07686548e-14])
success: True
nfev: 51
Scipy's optimize module has lots of options. See the documentation or this tutorial. Since you didn't specify the method here, it will use Sequential Least SQuares Programming (SLSQP). Alternatively, you could use the Trust-Region Constrained Algorithm (trust-const).
For this problem, I found that trust-const seemed much more robust to starting values than SLSQP, handling starting values from [-2,-2,-2] to [10,10,10], although negative initial values resulted in increased iterations, as you'd expect. Negative values below -2 exceeded the max iterations, although I suspect might still converge if you increased max iterations, although specifying negative values at all for x1 and x3 is kind of silly, of course, I just did it to get a sense of how robust it was to a range of starting values.
The specifications for SLSQP and trust-const are conceptually the same, but the syntax is a little different (in particular, note the use of NonlinearConstraint).
from scipy.optimize import minimize, NonlinearConstraint, SR1
def f(x):
return math.log(x[0]**2 + 1) + x[1]**4 + x[0]*x[2]
constr_func = lambda x: np.array( [ x[0]**3 - x[1]**2 - 1,
x[0],
x[2] ] )
x0=[0.,0.,0.]
nonlin_con = NonlinearConstraint( constr_func, 0., np.inf )
res = minimize( f, x0, method='trust-constr',
jac='2-point', hess=SR1(),
constraints = nonlin_con )
Here are the results, edited for conciseness:
fun: 0.6931502233468916
message: '`gtol` termination condition is satisfied.'
x: array([1.00000063e+00, 8.21427026e-09, 2.40956900e-06])
Note that the function value and x values are the same as in #CoryKramer's answer. The x array may look superficially different at first glance, but both answers round to [1, 0, 0].
I am using scipy.optimize.minimize to find the optimum value from a function. Here is the simplest example, using the built-in Rosenbrock function:
>>> from scipy.optimize import minimize, rosen
>>> x0 = [1.3, 0.7, 0.8, 1.9, 1.2]
>>> # Minimize returns a scipy.optimize.OptimizeResult object...
>>> res = minimize(rosen, x0, method='Nelder-Mead')
>>> print res
status: 0
nfev: 243
success: True
fun: 6.6174817088845322e-05
x: array([ 0.99910115, 0.99820923, 0.99646346, 0.99297555, 0.98600385])
message: 'Optimization terminated successfully.'
nit: 141
x is just the final, optimum input vector. ​Can I get a list for all iterations (i.e. an objective function with corresponding input vector) from the returned scipy.optimize.OptimizeResult object?
Yes, you could add the optional argument 'return_all'.
Example:
from scipy.optimize import minimize
def f(params):
x1, x2 = params
f = 4 * ((x1**2+(10-x2)**2)**0.5 - 10)**2 \
+ (1/2)*((x1**2+(10+x2)**2)**0.5-10)**2 \
-5*(x1+x2)
return f
x0 = [-4, 4]
res = minimize(f, x0, method='CG', options={'return_all':True})
# This example returns all iteration.