Develop Reference

Problems minimizing a function with 2 constraints - Python

I'm writing a program to minimize a function of several parameters subjected to constraints and bounds. Just in case you want to run the program, the function is given by:
def Fnret(mins):
Bj, Lj, a, b = mins.reshape(4,N)
S1 = 0; S2 = 0
Binf = np.zeros(N); Linf = np.zeros(N);
for i in range(N):
sbi=(Bi/2); sli=(Li/2)
for j in range(i+1):
sbi -= Bj[j]
sli -= Lj[j]
Binf[i]=sbi
Linf[i]=sli
for i in range(N):
S1 += (C*(1-np.sin(a[i]))+T*np.sin(a[i])) * ((2*Bj[i]*Binf[i]+Bj[i]**2)/(np.tan(b[i])*np.cos(a[i]))) +\
(C*(1-np.sin(b[i]))+T*np.sin(b[i])) * ((2*Bj[i]*Linf[i]+Lj[i]*Bj[i])/(np.sin(b[i])))
S2 += (gamma*Bj[0]/(6*np.tan(b[0])))*((Bi/2)*(Li/2) + 4*(Binf[0]+Bj[0])*(Linf[0]+Lj[0]) + Binf[0]*Linf[0])
j=1
while j<(N):
S2 += (gamma*Bj[j]/(6*np.tan(b[j])))*(Binf[j-1]*Linf[j-1] + 4*(Binf[j]+Bj[j])*(Linf[j]+Lj[j]) + Binf[j]*Linf[j])
j += 1
F = 2*(S1+S2)
return F
where Bj,Lj,a, and b are the minimization results given by N-sized arrays with N being an input of the program, I double-checked the function and it is working correctly. My constraints are given by:
def Brhs(mins): # Constraint over Bj
return np.sum(mins.reshape(4,N)[0]) - Bi
def Lrhs(mins): # Constraint over Lj
return np.sum(mins.reshape(4,N)[1]) - Li
cons = [{'type': 'eq', 'fun': lambda Bj: 1.0*Brhs(Bj)},
{'type': 'eq', 'fun': lambda Lj: 1.0*Lrhs(Lj)}]
In such a way that the sum of all components of Bj must be equal to Bi (same thing with Lj). The bounds of the variables are given by:
bounds = [(0,None)]*2*N + [(0,np.pi/2)]*2*N
For the problem to be reproducible, it's important to use the following inputs:
# Inputs:
gamma = 17.
C = 85.
T = C
Li = 1.
Bi = 0.5
N = 3
For the minimization, I'm using the cyipopt library (that is just similar to the scipy.optimize). Then, the minimization is given by:
from cyipopt import minimize_ipopt
x0 = np.ones(4*N) # Initial guess
res = minimize_ipopt(Fnret, x0, constraints=cons, bounds=bounds)
The problem is that the result is not obeying the conditions I imposed on the constraints (i.e. the sum of Bj or Lj values is different than Bi or Li on the results). But, for instance, if I only write one of the two constraints (over Lj or Bj) it works fine for that variable. Maybe I'm missing something when using 2 constraints and I can't find the error, it seems that it doesn't work with both constraints together. Any help would be truly appreciated. Thank you in advance!
P.S.: In addition, I would like that the function result F to be positive as well. How can I impose this condition? Thanks!

Not a complete answer, just some hints in arbitrary order:
Your initial point x0 is not feasible because it contradicts both of your constraints. This can easily be observed by evaluating your constraints at x0. Under the hood, Ipopt typically detects this and tries to find a feasible initial point. However, it's highly recommended to provide a feasible initial point whenever possible.
Your variable bounds are not well-defined. Evaluating your objective at your bounds yields multiple divisions by zero. For example, the denominator np.tan(b[i]) is zero if and only if b[i] = 0, so 0 isn't a valid value for all of your b[i]s. Proceeding similarly for the other terms, you should obtain 0 < b[i] < pi/2 and 0 ≤ a[i] < pi/2. Here, you can model the strict inequalities by 0 + eps ≤ b[i] ≤ pi/2 - eps and 0 ≤ a[i] ≤ pi/2 - eps, where eps is a sufficiently small positive number.
If you really want to impose that the objective function is always positive, you can simply add the inequality constraint Fnret(x) >= 0, i.e. {'type': 'ineq', 'fun': Fnret}.
In Code:
# bounds
eps = 1e-8
bounds = [(0, None)]*2*N + [(0, np.pi/2 - eps)]*N + [(0+eps, np.pi/2 - eps)]*N
# (feasible) initial guess
x0 = eps*np.ones(4*N)
x0[[0, N]] = [Bi-(N-1)*eps, Li-(N-1)*eps]
# constraints
cons = [{'type': 'eq', 'fun': Brhs},
{'type': 'eq', 'fun': Lrhs},
{'type': 'ineq', 'fun': Fnret}]
res = minimize_ipopt(Fnret, x0, constraints=cons, bounds=bounds, options={'disp': 5})
Last but not least, this still doesn't converge to a stationary point, so chances are that there's indeed no local minimum. From here, you can try experimenting with other (feasible!) initial points and double-check the math of your problem. It's also worth providing the exact gradient and constraint Jacobians.

So, based on #joni suggestions, I could find a stationary point respecting the constraints by adopting the trust-constr method of scipy.optimize.minimize library. My code is running as follows:
import numpy as np
from scipy.optimize import minimize
# Inputs:
gamma = 17
C = 85.
T = C
Li = 2.
Bi = 1.
N = 3 # for instance
# Constraints:
def Brhs(mins):
return np.sum(mins.reshape(4,N)[0]) - Bi/2
def Lrhs(mins):
return np.sum(mins.reshape(4,N)[1]) - Li/2
# Function to minimize:
def Fnret(mins):
Bj, Lj, a, b = mins.reshape(4,N)
S1 = 0; S2 = 0
Binf = np.zeros(N); Linf = np.zeros(N);
for i in range(N):
sbi=(Bi/2); sli=(Li/2)
for j in range(i+1):
sbi -= Bj[j]
sli -= Lj[j]
Binf[i]=sbi
Linf[i]=sli
for i in range(N):
S1 += (C*(1-np.sin(a[i]))+T*np.sin(a[i])) * ((2*Bj[i]*Binf[i]+Bj[i]**2)/(np.tan(b[i])*np.cos(a[i]))) +\
(C*(1-np.sin(b[i]))+T*np.sin(b[i])) * ((2*Bj[i]*Linf[i]+Lj[i]*Bj[i])/(np.sin(b[i])))
S2 += (gamma*Bj[0]/(6*np.tan(b[0])))*((Bi/2)*(Li/2) + 4*(Binf[0]+Bj[0])*(Linf[0]+Lj[0]) + Binf[0]*Linf[0])
j=1
while j<(N):
S2 += (gamma*Bj[j]/(6*np.tan(b[j])))*(Binf[j-1]*Linf[j-1] + 4*(Binf[j]+Bj[j])*(Linf[j]+Lj[j]) + Binf[j]*Linf[j])
j += 1
F = 2*(S1+S2)
return F
eps = 1e-3
bounds = [(0,None)]*2*N + [(0+eps,np.pi/2-eps)]*2*N # Bounds
cons = ({'type': 'ineq', 'fun': Fnret},
{'type': 'eq', 'fun': Lrhs},
{'type': 'eq', 'fun': Brhs})
x0 = np.ones(4*N) # Initial guess
res = minimize(Fnret, x0, method='trust-constr', bounds = bounds, constraints=cons, tol=1e-6)
F = res.fun
Bj = (res.x).reshape(4,N)[0]
Lj = (res.x).reshape(4,N)[1]
ai = (res.x).reshape(4,N)[2]
bi = (res.x).reshape(4,N)[3]
Which is essentially the same just changing the minimization technique. From np.sum(Bj) and np.sum(Lj) is easy to see that the results are in agreement with the constraints imposed, which were not working previously.

Marginal Likelihood optimization with python

I'm trying to optimize the marginal likelihood to estimate parameters for a Gaussian process regression.
So i defined the marginal log likelihood this way:
def marglike(par,X,Y):
l,sigma_n = par
n = len(X)
dist_X = (X.T - X)**2
k = np.exp(-(1/(2*(l**2)))*dist_X)
inverse = np.linalg.inv(k + (sigma_n**2)*np.eye(len(k)))
ml = (1/2)*np.dot(np.dot(Y.T,inverse),Y) + (1/2)*np.log(np.linalg.det(k + (sigma_n**2)*np.eye(len(k)))) + (n/2)*np.log(2*np.pi)
return ml
Where the parameter to be optimized are " l " and " sigma_n ".
With some initial values and data, the function give some value back:
X = np.linspace(1,10,20)
F = np.sin(X)
start = np.array([1,0.05]) #initial parameters values
marglike(start,X,F)
marglike(start,X,F)
Out[75]: array([[1872.6511786]])
But when i try to optimize the parameters with " minimize ", i get this:
re = minimize(marglike,start,args=(X,F),method="BFGS",options = {'disp':True})
re = minimize(marglike,start,args=(X,F),method="BFGS",options = {'disp':True})
Optimization terminated successfully.
Current function value: 22.863446
Iterations: 8
Function evaluations: 60
Gradient evaluations: 15
re.x
Out[89]: array([1. , 0.70845989])
I dont know why but the parameter "l" don't seem to be optimized, but it matches the starting value that i fixed.
Any suggest ?

You need to reshape X to 2d first for X.T-X to work. Also, you need to add one more parameter called variance (var in the code below) in optimization. Let me know if the below code solves your problem.
from scipy.optimize import minimize
def marglike(par,X,Y):
# print(par)
l,var,sigma_n = par
n = len(X)
dist_X = (X - X.T)**2
# print(dist_X)
k = var*np.exp(-(1/(2*(l**2)))*dist_X)
inverse = np.linalg.inv(k + (sigma_n**2)*np.eye(len(k)))
ml = (1/2)*np.dot(np.dot(Y.T,inverse),Y) + (1/2)*np.log(np.linalg.det(k + (sigma_n**2)*np.eye(len(k)))) + (n/2)*np.log(2*np.pi)
return ml
X = np.linspace(1,10,20).reshape(-1,1) # Reshaping
F = np.sin(X)
start = np.array([1.1,1.6,0.05]) #initial parameters values
print(marglike(start,X,F))
re = minimize(marglike,start,args=(X,F),method="L-BFGS-B",options = {'disp':True})
re.x

solving a minimization problem using scipy.optimize in python

I have a question regarding solving a minimization problem using scipy.optimize in python. I have an 1-D array (x) containing about 2000 elements as the variables of this problem, and a list of {constraint,dict}elements as the constraints of the optimization problem.The issue is that I want to calculate the sum of a large number of variables within a for loop to define objective function and the constraints. I have attached a simple example of my code below, but it should be considered that according to the complication of my code, it is not possible to do the calculation without for loops. However, by doing this, I face with this error: 'function' object is not iterable.
def objective(x):
sum = 0
for i in range(1, 1000):
sum += x[i]
return sum
def constraints(x):
constraint = []
for i in range(1, 1000):
sum = 0
for j in range(1, 100):
sum += j*x[i]
constraint.append({'type': 'ineq', 'fun': 10 - sum})
return constraint
sol = minimize(objective, x0, method='slsqp', constraints=constraints)

looked at the video you referenced, think you may have misunderstood a couple of things. I'll start with just a basic example to get you started.
If we take the below code:
from scipy.optimize import minimize
def objective(x):
x1=x[0]
x2=x[1]
x3=x[2]
x4=x[3]
return x1+x2+x3+x4
x0=(1,1,1,1)
solution=minimize(objective,x0)
Simple setup. You see the python function objective is returning a math function. That math function is what you are minimizing. Note how all your variables are defined by calling the value of a list of values (x0 in this case). Thus x1 is x0[0], x2 is x0[1], etc.
In your above example, you don't define x0, and you don't have an equation. Finally, sum is a python function (it takes the sum of a list). Instead, to do what you wish to do, lets rewrite the above using a for loop now:
from scipy.optimize import minimize
import numpy as np
def objective(x):
equation = 0
for i in range(4):
equation += x[i]
return equation
x0=np.ones(4)
solution=minimize(objective,x0)
This will give you the exact same output as above; however now you can see I've changed sum to equation (now you won't get issues with built in python functions), and you've now defined your variables). I'm not going to go through all your code, but I hope this example clears enough up that it gets you going.
If you want 1000s values of x, just give x0 an array of 1000 values.
Edit:
To resolve the issue in the 2nd part of your code:
Presuming you fixed using sum. Again, constraints is a dict where you input a function (mathematical). I don't get the error that you are getting, so I don't know where that is coming from. But you can create a constraint function using your above example.
def constraints(x):
constraint = []
for i in range(4):
value = 0
for j in range(4):
value += 10-j*x[i]
constraint.append(value)
return constraint
x0=np.ones(4)
con={'type': 'ineq', 'fun': constraints}
sol = minimize(objective, x0, method='slsqp', constraints=con)
Edit 2:
You can use the exact same ideology above to create a 2nd constraint:
from scipy.optimize import minimize
import numpy as np
def objective(x):
equation = 0
for i in range(4):
equation += x[i]
return equation
def constraints(x):
constraint = []
for i in range(4):
value = 0
for j in range(4):
value += 10-j*x[i]
constraint.append(value)
return constraint
def constraints2(x):
constraint2 = []
for a in range(4):
value = 0
for b in range(4):
value += 5-a*x[b]
constraint2.append(value)
return constraint2
x0=np.ones(4)
con={'type': 'ineq', 'fun': constraints}
con2={'type': 'eq', 'fun': constraints2}
combined=[con,con2]
sol = minimize(objective, x0, method='slsqp', constraints=combined)
This works without any problems or errors, so again, I don't see the problem you are having.

scipy.optimize.minimize results differ between Python 2.x-3.x

Basically, I have a nonlinear constrained problem using the SLSQP solver in scipy.optimize.minimize. Unfortunately, the problem (same file, same code) is returning different results on different computers (one Windows, one Linux). The scipy version is the same (1.2.1). Here is my code:
import numpy as np
from scipy.optimize import minimize
class OptimalAcc():
def __init__(self, v0, tg, tr, D0, sgr, l, t0, a0, b0,
rho_t=0.5, rho_u=0.5, vM=15, vm=2.78, aM=2.5, am=-2.9):
# Problem constants
self.v0 = v0
self.D0 = D0
self.sgr = sgr
self.l = l
self.T = tg + tr
self.D = tg / self.T
self.t0 = t0
self.a0 = a0
self.b0 = b0
self.rho_t = rho_t
self.rho_u = rho_u
self.vM = vM
self.vm = vm
self.aM = aM
self.am = am
def cost_fn(self, x):
# Acceleration profile variables
t = x[:1]
a = x[1:2]
b = x[2:3]
# Objective function
f = self.rho_t*x[:1]+self.rho_u*(a**2*t**3/3 +
a*b*t**2 +
b**2*t)
return f
def solve(self):
# Inequality constraints
ineq = ({'type':'ineq',
'fun':lambda x: np.array([self.aM - x[2],
x[2]-self.am,
x[0],
self.vM - (self.v0 + x[2]*x[0] + 0.5*x[1]*x[0]**2),
self.v0 + x[2]*x[0] + 0.5*x[1]*x[0]**2 - self.vm,
np.sin(np.pi*self.D - np.pi/2)-
np.sin(2*np.pi*(x[0] -((self.D0*self.T)/abs(self.sgr-2)))/self.T + 3*np.pi/2 - np.pi*self.D)])})
# Equality constraints
eq = ({'type':'eq',
'fun':lambda x: np.array([x[1]*x[0] + x[2],
self.v0*x[0] + 0.5*x[2]*x[0]**2 + x[1]*x[0]**3/6 - self.l])})
# Starting points
x0 = np.array([self.t0, self.a0, self.b0])
# Solve optimization problem
res = minimize(self.cost_fn,
x0=x0,
constraints=[ineq,eq],
options={'disp': True})
return res
if __name__== "__main__":
v0 = 1
tg = 20
tr = 20
D0 = 1
sgr = 1
l = 70
t0 = 10
a0 = -0.1
b0 = 1.5
# Create instance of optimization problem class
obj = OptimalAcc(v0, tg, tr, D0, sgr, l, t0, a0, b0)
# Solve problem and return optimal profile
u_t = obj.solve().x
print('x_1:',u_t[0])
print('x_2:',u_t[1])
print('x_3:',u_t[2])
The Windows machine yields:
Optimization terminated successfully. (Exit mode 0)
Current function value: 8.696191258640086
Iterations: 7
Function evaluations: 35
Gradient evaluations: 7
x_1: 13.508645429307041
x_2: -0.06874922875473621
x_3: 0.9287089606820067
I believe these results are locally optimal and I can verify the same output with fmincon in MATLAB.
However, the Linux machine yields:
Positive directional derivative for linesearch (Exit mode 8)
Current function value: 14.4116342889
Iterations: 17
Function evaluations: 147
Gradient evaluations: 13
x_1: 7.65875894797259
x_2: -0.241800477348664
x_3: 2.5000000000000053
Clearly, the optimizer is getting stuck in the Linux computer. What could be causing this? My only guess is that there's some precision within numpy that's throwing off the numbers.

As discussed in the comments, the issue is most likely not related to Windows vs. Linux, but more a Python 2 vs. Python 3 one. For example, the term
a**2*t**3/3
can look different between Python 2 and 3 as there might be only integers involved (there are more examples like this in your code).
An easy fix might be to include
from __future__ import division
at the top of your script which would take care of the differences on how divisions are performed in Python 2 and 3.

Penalty function method

I am trying to implement penalty function method for minimizing function. I need to find the minimum of Rosenbrok's function.
I am using this penalty function:
First of all, I have found the minimum using scipy.optimize.minimize:
from scipy.optimize import minimize, rosen
rz = lambda x: (1-x[0])**2 + 100*(x[1] - x[0]**2)**2;
h_1 = lambda x: (x[0] - 2 * x[1] + 2);
h_2 = lambda x: (-x[0] - 2 * x[1] + 6);
h_3 = lambda x: (-x[0] + 2 * x[1] + 2);
x0 = [2.3, 5];
cons = ({'type': 'ineq', 'fun': h_1},
{'type': 'ineq', 'fun': h_2},
{'type': 'ineq', 'fun': h_3})
minimize(rz, x0, constraints=cons)
The answer is x: array([ 0.99971613, 0.99942073])
Then I am trying to find the minimum using my implementation of penalty method:
x_c = [2.3, 3];
i = 1;
while i < 1000:
curr_func = lambda x: rz(x) + i*(h_1(x)**2 + h_2(x)**2 + h_3(x)**2)
x_c = minimize(curr_func, x_c).x;
i *= 1.2;
print(answer.x);
Which gives me [ 2.27402022 1.4157964 ] (if I increase the number of iterations, final values are even greater).
Where is the mistake in my implementation?
Thanks.
P.S. Function curr_func is specific for my constraints, of course, when they are all 'inequals' type.

The problem you have is that the h_i in your formula are for equality constraints, whereas the problem you are solving is for inequality constraints, which correspond to the g_i in your formula. Hence, your penalty function should be using terms like min(0, h_1(x))**2 instead of h_1(x)**2. To see why this is the case, just think about what happens if i = 1000 and x is the desired solution (1, 1). Then, the penalty will include a term i * h_1(x)**2 = 1000, which is huge.
Note that I used min instead of max because it seems like the inequality you want to enforce is h_1(x) >= 0. That means as long as h_1(x) >= 0, the penalty should be zero, but as soon as h_1(x) goes negative, you start penalizing. If it's actually h_1(x) <= 0 you want, then you use max (then you'll have to switch h_1 with -h_1 when you use scipy.optimize.minimize).
BTW, since i is usually an index variable, it's probably better to name the penalty weight something else, like a.