I am trying to perform a constrained nonlinear optimization problem, specifically minimization problem, using scipy.optimize.minimize. Specifically, I am trying to perform the constrained linear regression, which can be written as:
Min y-(b1x1+b2x2+...+bnxn+bn+1xn+1+....bpxp)
subject to
b1+b2+...+bn = 1,
bn+1+bn+2+...+bp = 1,
0 <= bi <= 1 for i=1,2,...,p
This is my code so far:
import pandas as pd
import numpy as np
from scipy.optimize import minimize
import warnings
def loss(x):
loss_function = np.sum(np.square((np.dot(x, m) - y)))
return loss_function
m = np.vstack([var_1, var_2, var_3,..., var_n, var_n+1,...,var_p])
y = np.array(ret['var_y'])
cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1.0})
x0 = np.zeros(m.shape[0])
res = minimize(loss, x0, method='SLSQP', constraints=cons,
bounds=[(0, 1) for i in range(m.shape[0])], options={'disp': True})
My question is, how do I implement those constraints into this model?
Related
For a project that I am currently working on, I need to optimize a convex problem. As can be read only there are different optimization tools in python in order to obtain the solution of a convex problem (e.g. scipy, CVXPY). However, by implementing these minimization algorithms I keep obtaining that success = 'False'.
Convex problem
To clearify: \beta = (\beta_0,\beta_1) is the optimization variable and \beta_k = (\beta_{k0},\beta{k1}) are just constants.
My code would look as follows:
import pandas as pd
import numpy as np
import math
from scipy import optimize
import matplotlib.pyplot as plt
from scipy.optimize import fmin
from scipy.optimize import Bounds
t = np.array([1,2,4,6,9,13])
y = np.array([3,4,6,10,17,25])
beta_k = [4.0,0.0]
def h_0():
test_h_0 = 0
for i in range(len(y)):
partial = 2*y[i]*beta_k[0]*np.exp(beta_k[1]*t[i]) + 20*t[i]*np.exp(4*t[i])*(beta_k[0]**2)
test_h_0 += partial
return test_h_0
def partial_h_0_1(beta):
test_h_0_1 = 0
for i in range(len(y)):
partial = (2*y[i]*np.exp(beta_k[1]*t[i]) + 40*t[i]*beta_k[0]*np.exp(4*t[i]))*(beta[0]-beta_k[0])
test_h_0_1 += partial
return test_h_0_1
def partial_h_0_2(beta):
test_h_0_2 = 0
for i in range(len(y)):
partial = (2*t[i]*y[i]*beta_k[0]*np.exp(beta_k[1]*t[i]))*(beta[1]-beta_k[1])
test_h_0_2 += partial
return test_h_0_2
def h_0_hat(beta):
return h_0() + partial_h_0_1(beta) + partial_h_0_2(beta)
def g_0(beta):
test_g_0 = 0
for i in range(len(y)):
partial = (beta[0]**2)*np.exp(2*beta[1]*t[i]) + 20*t[i]*np.exp(4*t[i])*(beta[0]**2) + (y[i]**2)
test_g_0 += partial
return test_g_0
def f_0(beta):
return g_0(beta) - h_0_hat(beta)
bnds = Bounds([1,-np.inf],[10,2])
cons = ({'type': 'ineq',
'fun': lambda beta: -beta[0] + 10.0},
{'type': 'ineq',
'fun': lambda beta: beta[0] - 1.0},
{'type': 'ineq',
'fun': lambda beta: -beta[1]+2.0})
test = optimize.minimize(f_0, x0=[0.0,0.0], method='trust-constr', constraints= cons)
beta = test.x
I'm setting up a new linear optimization code with Python. Unfortunately, I don't have the same results with Pulp, Scipy and Gekko packages.
I tried to implement code with different packages for Linear Optimization in Python.
OPTIMIZATION WITH GEKKO
from gekko import GEKKO
import numpy as np
import matplotlib.pyplot as plt
m = GEKKO() # create GEKKO model
x = m.Var(value=0, lb=0, ub=400000) # define new variable, initial value=0
y = m.Var(value=0, lb=0, ub=200) # define new variable, initial value=1
z = m.Var(value=0, lb=0)
m.Equation(x+y+z==100)
m.Obj(1.2*x + y + z) # equations
m.solve(disp=False) # solve
print("Solution with The GEKKO package")
print(x.value, y.value , z.value)# # print solution
OPTIMIZATION WITH Scipy
import numpy as np
from scipy.optimize import minimize
def objective(m):
x = m[0]
y = m[1]
z = m[2]
return 1.2*x + y + z
def constraint1(m):
return m[0] + m[1] + m[2] - 100
def constraint2(x):
return x[2]
x0 = [0,0,0]
b1 = (0,400000)
b2 = (0,200)
b3= (0,None)
bnds = (b1,b2,b3)
con1 = {'type' : 'eq', 'fun' : constraint1}
con2 = {'type' : 'ineq', 'fun' : constraint2}
cons = [con1,con2]
sol = minimize(objective,x0,method='SLSQP', bounds=bnds , constraints=cons)
print("Solution with The SCIPY package")
print(sol)
OPTIMIZATION WITH PULP
from pulp import *
prob = LpProblem("Problem",LpMinimize)
x = LpVariable("X",0,400000,LpContinuous)
y = LpVariable("Y",0,200,LpContinuous)
z = LpVariable("Z",0,None,LpContinuous)
prob += 1.2*x + y + z
prob += (x + y + z == 100)
prob.solve()
print("Solution with The PULP package")
print("Status:", LpStatus[prob.status])
for v in prob.variables():
print(v.name, "=", v.varValue)
I expect to have the same results, but the actual outputs are Different unfortunately :
The solution with The GEKKO package
[0.0] [36.210291349] [63.789708661]
The solution with The SCIPY package
fun: 100.0000000000001
jac: array([1.19999981, 1. , 1. ])
message: 'Optimization terminated successfully.'
nfev: 35
nit: 7
njev: 7
status: 0
success: True
x: array([4.88498131e-13, 5.00000000e+01, 5.00000000e+01])
The Solution with The PULP package
X = 0.0
Y = 100.0
Z = 0.0
All results are correct / Every solver is correct!
Each solution is reaching the minimum in it's objective: 100.
Each solution is preserving the variable-bounds
Each solution is preserving the "simplex-like" contraint: sum(x) = 100
Ignoring floating-point limitations, there are infinitely many different optimal solutions for your problem.
Different solvers including different solving approaches can lead to different solutions (picking one of many solutions). Here for example:
LP-algorithms like Simplex (Pulp)
NLP-algorithms like sequential least squares (scipy)
(keep in mind: there are also LP-solvers within scipy and more specialized solvers are usually better given some a-priori defined optimization problem -> LP vs. NLP)
I have a set of data points (x_i,y_i) and would like to fit a curve. I have a parametrization of the function, so f(x;a) is defined by the parameters a.
I'm interested in the parameters minimizing the uniform norm, i.e., the objective is min_a max_i |f(x_i;a)-y_i|.
If not possible, I'm also interested in the L1 norm, i.e. with the objective \min_a \sum_i |f(x_i;a)-y_i|.
I'm aware of curve_fit from scipy.optimize, but the library only works for the objective of least squares, i.e., L2 norm.
The problems I want to solve are of small size, approx 100-200 data points and 4-5 parameters, so if the algorithm is super slow is not a big deal.
Any library recommendations would be much appreciated.
Thanks!
One way to approach this (i only tackle the L1-norm here):
Convert:
non-differentiable (because of L1-norm) unconstrained optimization problem
to: differentiable constrained optimization problem!
Basic idea:
instead of Min(Sum(L1(x))):
introduce variables y with shape of x (aux-vars)
introduce constraints: -y <= x <= y
solve: Min(Sum(y))
A basic example using scipy:
import numpy as np
import matplotlib.pyplot as plt
from scipy import optimize
np.random.seed(0)
""" Example function and curve_fit """
def f(t, omega, phi):
return np.cos(omega * t + phi)
x_ = np.linspace(0, 3, 50)
y = f(x_, 1.5, 1) + .5*np.random.normal(size=50)
y[5] += 5 # synthetic outlier
y[10] += 5 # """
params, params_cov = optimize.curve_fit(f, x_, y)
t = np.linspace(0, 3, 1000)
""" Prototype l1 solver """
N = 2 # n-vars
M = x_.shape[0] # number of samples
def g(x):
return sum(x[N+M:])
cons = ({'type': 'eq',
'fun': lambda x: x[N:N+M] - (f(x_, *x[:N]) - y),
}
,
{'type': 'ineq',
'fun': lambda x: x[N+M:] + x[N:M+2], # -u_i <= x_i
},
{'type': 'ineq',
'fun': lambda x: -x[N:M+2] + x[N+M:]} # x_i <= u_i
)
res = optimize.minimize(g, np.random.uniform(high=0.05, size=N+M+M), constraints=cons, options={'disp': True})
params_ = res.x[:N]
print(res.x[N:N+M]) # losses
print(res.x[N+M:]) # l1(losses)
""" Plot results """
plt.plot(x_, y, color='red', label='orig')
plt.plot(t, f(t, *params), color='blue', label='curve_fit')
plt.plot(t, f(t, *params_), color='black', label='l1')
plt.legend()
plt.show()
Output
Remarks
As general solvers for the non-convex case are used, the problem is highly dependent on initial values / not robust
I have a constrained optimization problem that I am trying to solve using scipy.optimize.minimize.
Basically, I am fitting one parameter, f, to a system of ODEs with the constraint:
f >0 and
f*1000< 500
I wrote a MWE below. In this simple case, it is obvious that 0<f<0.5, but in my real problem, it is not obvious the a-priori upper bound, hence the inequality contraint.
from __future__ import division
import numpy as np
from scipy.integrate import odeint
from scipy.optimize import minimize
def myeqs(y,t,beta, gamma,N):
dy0 = -(beta/N)*y[0]*y[1]
dy1 = (beta/N)*y[0]*y[1] - gamma*y[1]
dy2 = gamma*y[1]
return [dy0,dy1,dy2]
def runModel(f, extraParams):
[beta, gamma, N, initCond, time]= extraParams
S0 = (1-f)*initCond[0]
I0 = initCond[1]
R0 = f*initCond[0]
tspan = np.linspace(time[0], time[1], int(time[1] - time[0]))
sol = odeint(myeqs, [S0,I0,R0], tspan, args=(beta, gamma, N))
return sol
y0 = [1000, 1, 0]
extraParams = [0.5, 0.25, 1000, y0, [0,150]]
def computeSimple(f, extraParams):
sol = runModel(f, extraParams)
return np.abs((sol[-1,2] - sol[0,2]))
def c1(f):
return f*1000.0 - 500.0
cons = ({'type': 'ineq', 'fun': c1})
#cons = ({'type': 'ineq',
# 'fun': lambda x: x[0]*1000 - 500}, )
res = minimize(computeSimple, 0.2, args=(extraParams, ), method='SLSQP', bounds=((0,1.5), ), constraints=cons)
print res.x
print c1(res.x)
If you run this, you will see that res.x is always the upper bound of bounds, regardless of the contraints...
Where is my mistake?
thanks in advance
You got your constraint backward. This constraint:
def c1(f):
return f*1000.0 - 500.0
is constraining f to be at least 0.5, instead of at most 0.5
I am using scipy.minimize with the COBYLA method, but I seem to be unable to write the constraints properly because when I check the values of the objective function, they do not respect those constraints.
Basically, the objective function accepts an array as argument, that must follow two constraints:
Each value in the array must be greater than 0
The sum of the values must be inferior to 1
So far I wrote it this way:
constraints = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x[0]},
{'type': 'ineq', 'fun': lambda x: x[1]}]
However, sometimes I get values greater than 1...
Here is an example:
from __future__ import division
from math import pow, exp
import numpy as np
from scipy.optimize import minimize
nbStudy = 3
nbCYP = 2
raucObserved = [3.98, 2.0, 0.12]
IXmat = np.matrix([[-0.98, 0], [-0.3, -0.98], [7.7, 4.2]])
NBITER = 50
estimatedCR = []
raucPred = []
varR = [0.0085, 0.0048, 0.0110]
sdR = [0.0922, 0.0692, 0.1051]
cnstrts = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x}]
def fun(CR):
dum = []
for i in range(nbStudy):
crix = 0
for j in range(nbCYP):
crix += CR[j] * IXmat[i, j]
raucPredicted = 1 / (1 + crix)
dum.append(pow((np.log(raucPredicted) - np.log(raucObservedBiased[i])), 2) / varR[i])
output = np.sum(dum)
return output
for iter in range(NBITER):
raucObservedBiased = []
for k in range(nbStudy):
raucObservedBiased.append(raucObserved[k] * exp(sdR[k] * np.random.normal()))
initialCR = np.matrix([[(1 / nbCYP) * np.random.uniform()], [(1 / nbCYP) * np.random.uniform()]])
output = minimize(fun, initialCR, method='COBYLA', constraints=cnstrts)
estimatedCR.append(output.x)
Apparently a version problem, the issue has been fixed since. I was using Python 2.7 and Scipy 0.13...
You are not checking that the solver converged (output.success == True) --- and in your case it does not converge. If there is no convergence, nothing is guaranteed about the constraints.