I am trying to perform a constrained nonlinear optimization problem, specifically minimization problem, using scipy.optimize.minimize. Specifically, I am trying to perform the constrained linear regression, which can be written as:
Min y-(b1x1+b2x2+...+bnxn+bn+1xn+1+....bpxp)
subject to
b1+b2+...+bn = 1,
bn+1+bn+2+...+bp = 1,
0 <= bi <= 1 for i=1,2,...,p
This is my code so far:
import pandas as pd
import numpy as np
from scipy.optimize import minimize
import warnings
def loss(x):
loss_function = np.sum(np.square((np.dot(x, m) - y)))
return loss_function
m = np.vstack([var_1, var_2, var_3,..., var_n, var_n+1,...,var_p])
y = np.array(ret['var_y'])
cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1.0})
x0 = np.zeros(m.shape[0])
res = minimize(loss, x0, method='SLSQP', constraints=cons,
bounds=[(0, 1) for i in range(m.shape[0])], options={'disp': True})
My question is, how do I implement those constraints into this model?
I was trying to perform a scipy.opimization using minimize function. I am looking to find all the variables like Iz,Iy,J,kz,ky,Yc,Yg such that the error between vector K_P_X and f is minimum. That is objective function K_P_X-fshould be minimum. I think my mistake is related to the calculation involving numpy.linalg.norm(sol-f)where the sol is assigned with a symbolic vector (K_P_X). Due to the data type conflict i am getting this error. If that's the case, Q1. Can anyone please suggest a better way to represent the equality constraint equation (ie. constr1()) such that this error can be avoided. The full code is given below,
import scipy.optimize as optimize
from sympy import symbols,zeros,Matrix,Transpose
import numpy
#Symobolic K matrix
Zc,Yc,Zg,Yg=symbols("Zc Yc Zg Yg",real=True)
A,Iz,Iy,J,kz,ky,E,G,L=symbols("A Iz Iy J kz ky E G L",real=True,positive=True)
E=10400000;G=3909800;L=5
def phi_z():
phi_z=(12*E*Iy)/(kz*A*G*L**2)
return phi_z
def phi_y():
phi_y=(12*E*Iz)/(ky*A*G*L**2)
return phi_y
K_P=zeros(12,12)
K1=Matrix(([E*A/L,0,0],[0,(12*E*Iz)/((1+phi_y())*L**3),0],[0,0,(12*E*Iy)/((1+phi_z())*L**3)]))
K2=Matrix(([G*J/L,0,0],[0,E*Iy/L,0],[0,0,E*Iz/L]))
Q1=Matrix(([0,Zg,-Yg],[-Zc,0,L/2],[Yc,-L/2,0]))
Q1_T=Transpose(Q1)
Q2=Matrix(([0,Zg,-Yg],[-Zc,0,-L/2],[Yc,L/2,0]))
Q2_T=Transpose(Q2)
K11=K1; K12=K1*Q1; K13=-K1; K14=-K1*Q2; K22=Q1_T*K1*Q1+K2; K23=-Q1_T*K1; K24=-Q1_T*K1*Q2-K2; K33=K1; K34=K1*Q2; K44=Q2_T*K1*Q2+K2
K_P[0:3,0:3]=K11; K_P[0:3,3:6]=K12; K_P[0:3,6:9]=K13; K_P[0:3,9:12]=K14; K_P[3:6,3:6]=K22; K_P[3:6,6:9]=K23; K_P[3:6,9:12]=K24 ;K_P[6:9,6:9]=K33; K_P[6:9,9:12]=K34; K_P[9:12,9:12]=K44
##Converting Upper triangular stiffness matrix to Symmetric stiffness matrix##
for i in range(0,12):
for j in range(0,12):
K_P[j,i]=K_P[i,j]
K_P = K_P.subs({A: 7.55})
K_P = K_P.subs({Zc: 0})
K_P = K_P.subs({Zg: 0})
X= numpy.matrix([[1],[1],[1],[1],[1],[1],[1],[1],[1],[1],[1],[1]])
K_P_X=K_P*X
f= numpy.matrix([[-9346.76033789],[1595512.77906],[-1596283.83112],[274222.872543],[4234010.18889],[4255484.3549],[9346.76033789],[-1595512.77906],[1596283.83112],[-275173.513088],[3747408.91068],[3722085.0499]])
function=K_P_X-f
def Obj_func(variables):
Iz,Iy,J,kz,ky,Yc,Yg=variables
function=K_P_X-f #K_P_X matrix contains the variables like Iz,Iy,J,kz,ky,Yc,Yg.
return function
def constr1(variables):
sol = K_P_X #Here the variables are in the symbolic vector K_P_X
if numpy.allclose(sol, f):
return 0.00 #If Error is equal to zero hence required accuracy is reached. So stop optimization
else:
return numpy.linalg.norm(sol-f)
initial_guess=[10,10,10,0.1,0.1,0.001,0.001]
cons = ({'type':'eq', 'fun': constr1},{'type': 'ineq', 'fun': lambda variables: -variables[3]+1},{'type': 'ineq', 'fun': lambda variables: variables[3]-0.001},{'type': 'ineq', 'fun': lambda variables: -variables[4]+1},{'type': 'ineq', 'fun': lambda variables: variables[4]-0.001},{'type': 'ineq', 'fun': lambda variables: -variables[5]+0.5},{'type': 'ineq', 'fun': lambda variables: variables[5]-0},{'type': 'ineq', 'fun': lambda variables: -variables[6]+0.5},{'type': 'ineq', 'fun': lambda variables: variables[6]-0})
bnds = ((1, 60), (1, 60),(1, 60),(0.1, 1),(0.1, 1),(0.001, 0.5),(0.001, 0.5))
res=optimize.minimize(Obj_func,initial_guess, bounds=bnds,constraints=cons)
I'll list some of the things that are wrong here.
As hpaulj said, you can't directly pass SymPy objects to SciPy or NumPy. But you can lambdify and then use that in the minimization routine
Your minimization setup does not make sense. Minimizing a function with the constraint that that same function must be zero... this is not what constrained minimization means. Constraints are something different from the objective.
It's better to use least_squares here which is dedicated to minimizing the norm of the difference (some vector function - target vector).
With that in mind, here is your script reworked:
import scipy.optimize as optimize
from sympy import symbols, Matrix, lambdify
import numpy
Iz,Iy,J,kz,ky,Yc,Yg = symbols("Iz Iy J kz ky Yc Yg",real=True,positive=True)
K_P_X = Matrix([[37.7776503296448*Yg + 8.23411191827681],[-340.454138522391*Iz/(21.1513673253807*Iz/ky + 125)],[-9.4135635827062*Iy*Yc/(21.1513673253807*Iy/kz + 125) - 368.454956983948*Iy/(21.1513673253807*Iy/kz + 125)],[-9.4135635827062*Iy*Yc**2/(21.1513673253807*Iy/kz + 125) - 368.454956983948*Iy*Yc/(21.1513673253807*Iy/kz + 125) - 0.0589826136148473*J],[23.5339089567655*Iy*Yc/(21.1513673253807*Iy/kz + 125) + 2.62756822555969*Iy + 921.137392459871*Iy/(21.1513673253807*Iy/kz + 125)],[-5.00660515891599*Iz - 851.135346305977*Iz/(21.1513673253807*Iz/ky + 125) - 37.7776503296448*Yg**2 - 8.23411191827681*Yg],[-37.7776503296448*Yg - 8.23411191827681],[340.454138522391*Iz/(21.1513673253807*Iz/ky + 125)],[9.4135635827062*Iy*Yc/(21.1513673253807*Iy/kz + 125) + 368.454956983948*Iy/(21.1513673253807*Iy/kz + 125)],[9.4135635827062*Iy*Yc**2/(21.1513673253807*Iy/kz + 125) + 368.454956983948*Iy*Yc/(21.1513673253807*Iy/kz + 125) + 0.0589826136148473*J],[23.5339089567655*Iy*Yc/(21.1513673253807*Iy/kz + 125) - 2.62756822555969*Iy + 921.137392459871*Iy/(21.1513673253807*Iy/kz + 125)],[5.00660515891599*Iz - 851.135346305977*Iz/(21.1513673253807*Iz/ky + 125) + 37.7776503296448*Yg**2 + 8.23411191827681*Yg]])
f = Matrix([[-1],[-1],[-1],[-1.00059553353],[3.99999996539],[-5.99940443072],[1],[1],[1],[1],[1],[1]])
obj = lambdify([Iz,Iy,J,kz,ky,Yc,Yg], tuple(K_P_X - f))
initial_guess=[10,10,10,0.1,0.1,0.001,0.001]
bnds = ((1, 60), (1, 60),(1, 60),(0.1, 1),(0.1, 1),(0.001, 0.5),(0.001, 0.5))
lower = [a for (a, b) in bnds]
upper = [b for (a, b) in bnds]
res = optimize.least_squares(lambda x: obj(x[0], x[1], x[2], x[3], x[4], x[5], x[6]), initial_guess, bounds=(lower, upper))
print(res)
Changes:
Prior to lambdify, we should have a SymPy expression. So both K_P_X and f are SymPy matrices now.
Lambdified function takes 7 scalar arguments and returns a tuple of components of K_P_X - f
The bounds are separated into lower and upper, as the syntax of least_squares requires
We can't directly pass obj to least_squares, because it will receive one array parameter instead of 7 scalars. Hence the additional lambda step for unpacking the vector.
Believe it or not, minimization works. It returns res.x, the minimum point, as
[ 1.00000000e+00, 1.00000000e+00, 1.69406332e+01,
1.00000000e-01, 1.00000000e-01, 1.00000000e-03,
1.00000000e-03]
which looks suspiciously round at first, but this is only because the point hits against the bounds you placed (10, 1, 0.1 and so on). Only the third variable ended up with an inactive constaint.
I am trying to optimize the first function given the latter two constraining function, using Scipy.
def entropy(x):
entropy = 0
for i in range(6):
entropy = entropy + x[i]*np.log(x[i])
return entropy
def constraint_1(x):
validpmf = 0
for i in range(6):
validpmf = validpmf + x[i]
return validpmf - 1
def constraint_2(x):
mean = 0
for i in range(7):
mean = mean + (i*x[i-1])
return mean - 4.5
Here is the Scipy code.
ans = sp.optimize.minimize(entropy, [.04,.08,.1,.15,.25,.35], \
constraints = cons, jac = False, method = 'SLSQP')
I am receiving the actual correct answer back, but I am getting a runtime warning:
[ 0.05447023 0.07863089 0.1140969 0.16556351 0.23970755 0.34753092]
RuntimeWarning: invalid value encountered in log
entropy = entropy + x[i]*np.log(x[i])
I had this issue before with a simpler optimization problem where it was returning an incorrect answer which I fixed by changing my initial guesses. I don't understand why that had worked. In this case however, the initial guesses are quite good approximations so I want to keep them, and also changing them around hasn't managed to mitigate the runtime warning.
To summarize, solution is correct but I don't understand the runtime warning.
The log is not defined at zero, or if you are a computer around zero.
You can suppress the warning by restricting to positive values:
import numpy as np
import scipy as sp
import scipy.optimize
def entropy(x):
entropy = 0
for i in range(6):
entropy = entropy + x[i]*np.log(x[i])
return entropy
def constraint_1(x):
validpmf = 0
for i in range(6):
validpmf = validpmf + x[i]
return validpmf - 1
def constraint_2(x):
mean = 0
for i in range(6):
mean = mean + (i*x[i])
return mean - 4.5
abstol = 1e-6
cons = ({'type': 'eq', 'fun': constraint_1},
{'type': 'eq', 'fun': constraint_2},
{'type': 'ineq', 'fun': lambda x: x - abstol})
ans = sp.optimize.minimize(entropy, [.04,.08,.1,.15,.25,.35], \
constraints = cons, jac = False, method = 'SLSQP')
I have a constrained optimization problem that I am trying to solve using scipy.optimize.minimize.
Basically, I am fitting one parameter, f, to a system of ODEs with the constraint:
f >0 and
f*1000< 500
I wrote a MWE below. In this simple case, it is obvious that 0<f<0.5, but in my real problem, it is not obvious the a-priori upper bound, hence the inequality contraint.
from __future__ import division
import numpy as np
from scipy.integrate import odeint
from scipy.optimize import minimize
def myeqs(y,t,beta, gamma,N):
dy0 = -(beta/N)*y[0]*y[1]
dy1 = (beta/N)*y[0]*y[1] - gamma*y[1]
dy2 = gamma*y[1]
return [dy0,dy1,dy2]
def runModel(f, extraParams):
[beta, gamma, N, initCond, time]= extraParams
S0 = (1-f)*initCond[0]
I0 = initCond[1]
R0 = f*initCond[0]
tspan = np.linspace(time[0], time[1], int(time[1] - time[0]))
sol = odeint(myeqs, [S0,I0,R0], tspan, args=(beta, gamma, N))
return sol
y0 = [1000, 1, 0]
extraParams = [0.5, 0.25, 1000, y0, [0,150]]
def computeSimple(f, extraParams):
sol = runModel(f, extraParams)
return np.abs((sol[-1,2] - sol[0,2]))
def c1(f):
return f*1000.0 - 500.0
cons = ({'type': 'ineq', 'fun': c1})
#cons = ({'type': 'ineq',
# 'fun': lambda x: x[0]*1000 - 500}, )
res = minimize(computeSimple, 0.2, args=(extraParams, ), method='SLSQP', bounds=((0,1.5), ), constraints=cons)
print res.x
print c1(res.x)
If you run this, you will see that res.x is always the upper bound of bounds, regardless of the contraints...
Where is my mistake?
thanks in advance
You got your constraint backward. This constraint:
def c1(f):
return f*1000.0 - 500.0
is constraining f to be at least 0.5, instead of at most 0.5
I am using scipy.minimize with the COBYLA method, but I seem to be unable to write the constraints properly because when I check the values of the objective function, they do not respect those constraints.
Basically, the objective function accepts an array as argument, that must follow two constraints:
Each value in the array must be greater than 0
The sum of the values must be inferior to 1
So far I wrote it this way:
constraints = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x[0]},
{'type': 'ineq', 'fun': lambda x: x[1]}]
However, sometimes I get values greater than 1...
Here is an example:
from __future__ import division
from math import pow, exp
import numpy as np
from scipy.optimize import minimize
nbStudy = 3
nbCYP = 2
raucObserved = [3.98, 2.0, 0.12]
IXmat = np.matrix([[-0.98, 0], [-0.3, -0.98], [7.7, 4.2]])
NBITER = 50
estimatedCR = []
raucPred = []
varR = [0.0085, 0.0048, 0.0110]
sdR = [0.0922, 0.0692, 0.1051]
cnstrts = [{'type': 'ineq', 'fun': lambda x: 1 - sum(x)},
{'type': 'ineq', 'fun': lambda x: x}]
def fun(CR):
dum = []
for i in range(nbStudy):
crix = 0
for j in range(nbCYP):
crix += CR[j] * IXmat[i, j]
raucPredicted = 1 / (1 + crix)
dum.append(pow((np.log(raucPredicted) - np.log(raucObservedBiased[i])), 2) / varR[i])
output = np.sum(dum)
return output
for iter in range(NBITER):
raucObservedBiased = []
for k in range(nbStudy):
raucObservedBiased.append(raucObserved[k] * exp(sdR[k] * np.random.normal()))
initialCR = np.matrix([[(1 / nbCYP) * np.random.uniform()], [(1 / nbCYP) * np.random.uniform()]])
output = minimize(fun, initialCR, method='COBYLA', constraints=cnstrts)
estimatedCR.append(output.x)
Apparently a version problem, the issue has been fixed since. I was using Python 2.7 and Scipy 0.13...
You are not checking that the solver converged (output.success == True) --- and in your case it does not converge. If there is no convergence, nothing is guaranteed about the constraints.