I have a constrained optimization problem that I am trying to solve using scipy.optimize.minimize.
Basically, I am fitting one parameter, f, to a system of ODEs with the constraint:
f >0 and
f*1000< 500
I wrote a MWE below. In this simple case, it is obvious that 0<f<0.5, but in my real problem, it is not obvious the a-priori upper bound, hence the inequality contraint.
from __future__ import division
import numpy as np
from scipy.integrate import odeint
from scipy.optimize import minimize
def myeqs(y,t,beta, gamma,N):
dy0 = -(beta/N)*y[0]*y[1]
dy1 = (beta/N)*y[0]*y[1] - gamma*y[1]
dy2 = gamma*y[1]
return [dy0,dy1,dy2]
def runModel(f, extraParams):
[beta, gamma, N, initCond, time]= extraParams
S0 = (1-f)*initCond[0]
I0 = initCond[1]
R0 = f*initCond[0]
tspan = np.linspace(time[0], time[1], int(time[1] - time[0]))
sol = odeint(myeqs, [S0,I0,R0], tspan, args=(beta, gamma, N))
return sol
y0 = [1000, 1, 0]
extraParams = [0.5, 0.25, 1000, y0, [0,150]]
def computeSimple(f, extraParams):
sol = runModel(f, extraParams)
return np.abs((sol[-1,2] - sol[0,2]))
def c1(f):
return f*1000.0 - 500.0
cons = ({'type': 'ineq', 'fun': c1})
#cons = ({'type': 'ineq',
# 'fun': lambda x: x[0]*1000 - 500}, )
res = minimize(computeSimple, 0.2, args=(extraParams, ), method='SLSQP', bounds=((0,1.5), ), constraints=cons)
print res.x
print c1(res.x)
If you run this, you will see that res.x is always the upper bound of bounds, regardless of the contraints...
Where is my mistake?
thanks in advance
You got your constraint backward. This constraint:
def c1(f):
return f*1000.0 - 500.0
is constraining f to be at least 0.5, instead of at most 0.5
Related
I'm trying to solve a constraint maximization problem with bounds in scipy minimize SLSQP. But why am
I getting this message: 'Singular matrix C in LSQ subproblem'? How to resolve this? When I'm removing the constraint and trying to minimize the objective function it's working fine but when I'm trying to maximize it, it shows 'Positive directional derivative for linesearch'. Code follows below:
#Import libs
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize, Bounds
#Initial activity level
x0 = np.array([60, 40])
#Revenue function
def revenue(X):
dollarperTRx = 300
coeff_x1 = 0.234
coeff_x2 = 0.127
predwo= 1.245
nhcp = 400
r = dollarperTRx * nhcp * predwo * (pow(1 + (1+ X[0]),coeff_x1)) * (pow((1 + X[1]),coeff_x2))
return r
#Objective function
def objective(X, sign = -1.0):
return sign * revenue(X)
#Spend
cost_per_promo = np.array([400, 600])
def spend(X):
return np.dot(cost_per_promo, x0.transpose())
#Initial Spend
s0 = spend(x0)
#Spend constraint
def spend_constraint(X):
return spend(X) - s0
#Getting the constraints into a dictionary
cons = ({'type':'eq', 'fun': spend_constraint})
#Bounds
bounds1 = (30, 90)
bounds2 = (20, 60)
#Optimize
minimize(objective, x0, method='SLSQP', constraints = cons, bounds = (bounds1, bounds2))
Your spend function does not depend on your design vector, therefore your constraint also does not depend on it. This makes the problem singular. I changed X0 to the current design vector in your example, that way it converges. You have to verify, if that is what you meant to do with the constraint. But with X0 it always gives 0.
#Import libs
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize, Bounds
#Initial activity level
x0 = np.array([60, 40])
#Revenue function
def revenue(X):
dollarperTRx = 300
coeff_x1 = 0.234
coeff_x2 = 0.127
predwo= 1.245
nhcp = 400
r = dollarperTRx * nhcp * predwo * (pow(1 + (1+ X[0]),coeff_x1)) * (pow((1 + X[1]),coeff_x2))
return r
revenue0 = revenue(x0)
#Objective function
def objective(X):
return -revenue(X) / revenue0
#Spend
cost_per_promo = np.array([400., 600.])
def spend(X):
return np.dot(cost_per_promo, X.transpose())
#Initial Spend
s0 = spend(x0)
#Spend constraint
def spend_constraint(X):
return spend(X) - s0
#Getting the constraints into a dictionary
cons = ({'type':'eq', 'fun': spend_constraint})
#Bounds
bounds1 = (30., 90.)
bounds2 = (20., 60.)
#Optimize
res = minimize(objective, x0, method='SLSQP',
constraints = cons,
bounds = (bounds1, bounds2))
print(res)
Results in:
fun: -1.0157910949030706
jac: array([-0.00297113, -0.00444862])
message: 'Optimization terminated successfully.'
nfev: 36
nit: 9
njev: 9
status: 0
success: True
x: array([78.0015639, 27.9989574])
I am trying to minimize a function that basically looks like this:
In reality it has two independent variables, but since x1 + x2 = 1, they're not REALLY independent.
now here's the objective function
def calculatePVar(w,covM):
w = np.matrix(w)
return (w*covM*w.T) [0,0]
wnere w is a list of the weights of each asset and covM is the covariance matrix that is returned by .cov() of pandas
Here's where the optimization function is called:
w0 = []
for sec in portList:
w0.append(1/len(portList))
bnds = tuple((0,1) for x in w0)
cons = ({'type': 'eq', 'fun': lambda x: np.sum(x)-1.0})
res= minimize(calculatePVar, w0, args=nCov, method='SLSQP',constraints=cons, bounds=bnds)
weights = res.x
now there is a clear minimum to the function but minimize will just spit out the initial values as the result and it does say "Optimization terminated sucessfully". Any suggestions?
optimization results:
P.S. images as links because I don't meet the reqs!
Your code had just some confusing variables so I just cleared that out and simplified some lines, now the minimization works correctly. However, the question now is: if the results are correct? and do they make sense? and that is for you to judge:
import numpy as np
from scipy.optimize import minimize
def f(w, cov_matrix):
return (np.matrix(w) * cov_matrix * np.matrix(w).T)[0,0]
cov_matrix = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
p = [1, 2, 3]
w0 = [(1/len(p)) for e in p]
bnds = tuple((0,1) for e in w0)
cons = ({'type': 'eq', 'fun': lambda w: np.sum(w)-1.0})
res = minimize(f, w0,
args = cov_matrix,
method = 'SLSQP',
constraints = cons,
bounds = bnds)
weights = res.x
print(res)
print(weights)
Update:
Based on your comments, it seems to me that -maybe- your function has has multiple minima and that's why scipy.optimize.minimize gets trapped in there. I suggest scipy.optimize.basinhopping as an alternative, this would use a random step to go over most of the minima of your function and it will still be fast. Here is the code:
import numpy as np
from scipy.optimize import basinhopping
class MyBounds(object):
def __init__(self, xmax=[1,1], xmin=[0,0] ):
self.xmax = np.array(xmax)
self.xmin = np.array(xmin)
def __call__(self, **kwargs):
x = kwargs["x_new"]
tmax = bool(np.all(x <= self.xmax))
tmin = bool(np.all(x >= self.xmin))
return tmax and tmin
def f(w):
global cov_matrix
return (np.matrix(w) * cov_matrix * np.matrix(w).T)[0,0]
cov_matrix = np.array([[0.000244181, 0.000198035],
[0.000198035, 0.000545958]])
p = ['ABEV3', 'BBDC4']
w0 = [(1/len(p)) for e in p]
bnds = tuple((0,1) for e in w0)
cons = ({'type': 'eq', 'fun': lambda w: np.sum(w)-1.0})
bnds = MyBounds()
minimizer_kwargs = {"method":"SLSQP", "constraints": cons}
res = basinhopping(f, w0,
accept_test = bnds)
weights = res.x
print(res)
print("weights: ", weights)
Output:
fun: 2.3907094432990195e-09
lowest_optimization_result: fun: 2.3907094432990195e-09
hess_inv: array([[ 2699.43934183, -1184.79396719],
[-1184.79396719, 1210.50404805]])
jac: array([1.34548553e-06, 2.00122166e-06])
message: 'Optimization terminated successfully.'
nfev: 60
nit: 6
njev: 15
status: 0
success: True
x: array([0.00179748, 0.00118076])
message: ['requested number of basinhopping iterations completed successfully']
minimization_failures: 0
nfev: 6104
nit: 100
njev: 1526
x: array([0.00179748, 0.00118076])
weights: [0.00179748 0.00118076]
I had a similar problem and the issue turned out to be that the function and the constraint were outputting numpy arrays with a single element. Changing the output of those two functions to be floats solved the problem.
A very simple solution to a perplexing problem.
I am trying to perform a constrained nonlinear optimization problem, specifically minimization problem, using scipy.optimize.minimize. Specifically, I am trying to perform the constrained linear regression, which can be written as:
Min y-(b1x1+b2x2+...+bnxn+bn+1xn+1+....bpxp)
subject to
b1+b2+...+bn = 1,
bn+1+bn+2+...+bp = 1,
0 <= bi <= 1 for i=1,2,...,p
This is my code so far:
import pandas as pd
import numpy as np
from scipy.optimize import minimize
import warnings
def loss(x):
loss_function = np.sum(np.square((np.dot(x, m) - y)))
return loss_function
m = np.vstack([var_1, var_2, var_3,..., var_n, var_n+1,...,var_p])
y = np.array(ret['var_y'])
cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1.0})
x0 = np.zeros(m.shape[0])
res = minimize(loss, x0, method='SLSQP', constraints=cons,
bounds=[(0, 1) for i in range(m.shape[0])], options={'disp': True})
My question is, how do I implement those constraints into this model?
I have a set of data points (x_i,y_i) and would like to fit a curve. I have a parametrization of the function, so f(x;a) is defined by the parameters a.
I'm interested in the parameters minimizing the uniform norm, i.e., the objective is min_a max_i |f(x_i;a)-y_i|.
If not possible, I'm also interested in the L1 norm, i.e. with the objective \min_a \sum_i |f(x_i;a)-y_i|.
I'm aware of curve_fit from scipy.optimize, but the library only works for the objective of least squares, i.e., L2 norm.
The problems I want to solve are of small size, approx 100-200 data points and 4-5 parameters, so if the algorithm is super slow is not a big deal.
Any library recommendations would be much appreciated.
Thanks!
One way to approach this (i only tackle the L1-norm here):
Convert:
non-differentiable (because of L1-norm) unconstrained optimization problem
to: differentiable constrained optimization problem!
Basic idea:
instead of Min(Sum(L1(x))):
introduce variables y with shape of x (aux-vars)
introduce constraints: -y <= x <= y
solve: Min(Sum(y))
A basic example using scipy:
import numpy as np
import matplotlib.pyplot as plt
from scipy import optimize
np.random.seed(0)
""" Example function and curve_fit """
def f(t, omega, phi):
return np.cos(omega * t + phi)
x_ = np.linspace(0, 3, 50)
y = f(x_, 1.5, 1) + .5*np.random.normal(size=50)
y[5] += 5 # synthetic outlier
y[10] += 5 # """
params, params_cov = optimize.curve_fit(f, x_, y)
t = np.linspace(0, 3, 1000)
""" Prototype l1 solver """
N = 2 # n-vars
M = x_.shape[0] # number of samples
def g(x):
return sum(x[N+M:])
cons = ({'type': 'eq',
'fun': lambda x: x[N:N+M] - (f(x_, *x[:N]) - y),
}
,
{'type': 'ineq',
'fun': lambda x: x[N+M:] + x[N:M+2], # -u_i <= x_i
},
{'type': 'ineq',
'fun': lambda x: -x[N:M+2] + x[N+M:]} # x_i <= u_i
)
res = optimize.minimize(g, np.random.uniform(high=0.05, size=N+M+M), constraints=cons, options={'disp': True})
params_ = res.x[:N]
print(res.x[N:N+M]) # losses
print(res.x[N+M:]) # l1(losses)
""" Plot results """
plt.plot(x_, y, color='red', label='orig')
plt.plot(t, f(t, *params), color='blue', label='curve_fit')
plt.plot(t, f(t, *params_), color='black', label='l1')
plt.legend()
plt.show()
Output
Remarks
As general solvers for the non-convex case are used, the problem is highly dependent on initial values / not robust
I am using scipy.minimize with the COBYLA method, but I seem to be unable to write the constraints properly because when I check the values of the objective function, they do not respect those constraints.
Basically, the objective function accepts an array as argument, that must follow two constraints:
Each value in the array must be greater than 0
The sum of the values must be inferior to 1
So far I wrote it this way:
constraints = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x[0]},
{'type': 'ineq', 'fun': lambda x: x[1]}]
However, sometimes I get values greater than 1...
Here is an example:
from __future__ import division
from math import pow, exp
import numpy as np
from scipy.optimize import minimize
nbStudy = 3
nbCYP = 2
raucObserved = [3.98, 2.0, 0.12]
IXmat = np.matrix([[-0.98, 0], [-0.3, -0.98], [7.7, 4.2]])
NBITER = 50
estimatedCR = []
raucPred = []
varR = [0.0085, 0.0048, 0.0110]
sdR = [0.0922, 0.0692, 0.1051]
cnstrts = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x}]
def fun(CR):
dum = []
for i in range(nbStudy):
crix = 0
for j in range(nbCYP):
crix += CR[j] * IXmat[i, j]
raucPredicted = 1 / (1 + crix)
dum.append(pow((np.log(raucPredicted) - np.log(raucObservedBiased[i])), 2) / varR[i])
output = np.sum(dum)
return output
for iter in range(NBITER):
raucObservedBiased = []
for k in range(nbStudy):
raucObservedBiased.append(raucObserved[k] * exp(sdR[k] * np.random.normal()))
initialCR = np.matrix([[(1 / nbCYP) * np.random.uniform()], [(1 / nbCYP) * np.random.uniform()]])
output = minimize(fun, initialCR, method='COBYLA', constraints=cnstrts)
estimatedCR.append(output.x)
Apparently a version problem, the issue has been fixed since. I was using Python 2.7 and Scipy 0.13...
You are not checking that the solver converged (output.success == True) --- and in your case it does not converge. If there is no convergence, nothing is guaranteed about the constraints.