Develop Reference

Problem while looping variables for Scipy Optimization (SLSQP, COBYLA)

I have a constrained optimization problem where I am trying to minimize an objective function of 100+ variables which is of the form
Min F(x) = f(x1) + f(x2) + ... + f(xn)
Subject to functional constraint
(g(x1) + g(x2) + ... + g(xn))/(f(x1) + f(x2) + ... + f(xn)) - constant >= 0
I also have individual bounds for each variable x1, x2, x3...xn
a <= x1 <= b
c <= x2 <= d
...
For this, I wrote a python script, using the scipy.optimize.minimize implementation with constraints and bounds, but I am unable to fulfill my bounds and constraints in the solutions. These are all cases where optimization could converge to a solution (message: success)
Here is a sample of my code:
df is my pandas dataset
B(x) is LogNorm transform based on x and other constants
Values U, c, lb, ub are pre-calculated constant dictionaries for each index in df
import scipy
df = pd.DataFrame(..)
k = set(df.index.values) ## list of indexes to iterate on
val = 0.25 ## Arbitrary
def obj(x):
fn = 0
for n,i in enumerate(k):
x0 = x[n]
fn1 = (U[i]) * B(x0) * (x0)
fn += fn1
return fn
def cons(x):
cn = 1
c1 = 0
c2 = 0
for n,i in enumerate(k):
x0 = x[n]
c1 += (U[i]) * (B(x0) * (x0 - c[i])
c2 += (U[i]) * (B(x0) * (x0)
cn = c1/(c2)
return cn - val
const = [{'type':'ineq', 'fun':cons}]
bnds = tuple((lb[i], ub[i]) for i in k) ## Lower, Upper for each element ((lb1, ub1), (lb2, ub2)...)
x_init = [lb[i] for i in k] ## for eg. starting from lower bound
## Solution
sol = scipy.optimize.minimize(obj, x_init, method = 'COBYLA', bounds = bnds, constraints = const)
I have more pointed questions if that helps:
Is there a way to construct the same equation concisely/ without the use of loops (given the number of variables could depend on input data and I have no control over it)?
Is there any noticeable issue in my application of bounds? I can't seem to get the final values of all variables follow individual bounds.
Similarly, is there a visible flaw in the construction on constraint equation? My results often DO NOT follow the constraints is repeated runs with different inputs.
Any help with either of the questions can help me progress further at work.
I have also looked into a Lagrangian solution of the same but so far I am unable to solve it for undefined number of (n) variables.
Thanks!

Simple Inequality Constraint Breaks QP Problem (Control Allocation)

Optimization problem: Just for context, not strictly needed. Using SciPy's "minimize" the code is allocating optimal control forces to thrusters on a ship given environmental forces acting on the ship (wind, waves & current) in the 3 DOFs surge, sway and yaw to hold the ship's position (Important for Dynamic Positioning).
This ship has 4 thrusters, 1 fixed-angle thruster (tunnel) and 3 rotating thrusters (azimuth). The fixed thruster has a force in one direction (x1) and the rotating thrusters has forces in two directions, x and y, implemented as variable pairs (x2, x3), (x4, x5), (x6, x7). E.g. x2 and x3 is the force in x and y, respectively, for rotating thruster 1. There are also 3 slack variables for the 3 DOFs to ensure a solution.
Problem on standard form
Programming problem: I want to limit one of the thrusters' force by implementing an inequality constraint on it, but by implementing the constraint "ineq1" all outputs are minimized to zero. The constraint should limit x1 and is very simply like this: (x1 < x1_max) Follow-up: "minimize" seems to ignore the first constraint in the constraint dictionary and it doesn't matter if it's an equality or inequality constraint. The second constraint is however upheld.
Output with only equality constraint:
Correct output
Output with breaking inequality constraint:
Incorrect output with inequality
What I've tried is
... trying different algorithms
... test every combination of positives/negatives in the constraint
... use an array/vector A as in the standard form A*x < b instead of just x[0]
... using a number instead of fmax[0]
... tried other design variables and combinations of them
Code:
This can be readily run. Change between "cons1" and "cons2" in line 42 to see the different results.
import numpy as np
from scipy.optimize import minimize
# Parameters
nf = 1 # No. of fixed thrusters
nr = 3 # No. of rotating thrusters
nt = nf + 2*nr # No. of actuators
ns = 3 # No. of slack variables (slacks)
n = nt + ns # Design variables
Lx = np.array([30.5, 22.25, -35.5, -35.5]) # Thruster position
W = np.diag([1, 1, 1, 1, 1, 1, 1]) # Weighting for thrusters
Q = np.diag([1, 1, 1])*5000 # Weighting for slacks
Te = np.array([[0,1,0,1,0,1,0], # Thrust config. matrix
[1,0,1,0,1,0,1],
[Lx[0],0,Lx[1],0,Lx[2],0,Lx[3]]])
fmax = np.array([ 50000, 60000, 75000, 75000])
tau = np.array([50000,35000,0]) # Environmental forces in surge, sway & yaw
x0 = np.zeros(n)
# Problem definition
def obj(x):
fe = x[:nt]
s = x[nt:]
return fe.transpose()#W#fe + s.transpose()#Q#s
def eqcon(x):
fe = x[:nt]
s = x[nt:]
return tau + s - Te#fe # = 0
def ineq1(x):
return fmax[0] - x[0] # > 0
cons1 = {'type': 'eq', 'fun': eqcon}
cons2 = {'type': 'eq', 'fun': eqcon,
'type': 'ineq', 'fun': ineq1}
# Computing
opt = {'disp': True}
res = minimize(obj, x0, method='SLSQP', constraints=cons2, options=opt)
# Debugging
print("Environmental forces (surge, sway & yaw): ", tau)
print("Forces: ", np.round(res.x[:nt],1))
print("Slacks: ", np.round(res.x[nt:],1))
print(f"Thruster 1 has angle 90.00 with force {res.x[0]/1000:0.2f} kN")
angles = np.zeros(nr)
ux = res.x[nf:nt][::2] # Every other force element in x for azimuth: ux2, ux3, ...
uy = res.x[nf:nt][1:][::2] # uy2, uy3, ...
for i in range(nr):
angles[i] = np.arctan2(uy[i], ux[i]) * 180 / np.pi
print(f"Thruster {nf+i+1} has angle {angles[i]:0.2f} with force {np.sqrt(ux[i]**2+uy[i]**2)/1000:0.2f} kN")
Any ideas?

The constraints are expected to be a list of dictionaries, so using
cons2 = [{'type': 'eq', 'fun': eqcon},
{'type': 'ineq', 'fun': ineq1}]
should fix your problem. That being said, the keys inside a dictionary should be unique. Otherwise, you aren't able to access a specific value in the dictionary by ambiguous keys.

Minimization of a function with iterative bounds in Python

I'm trying to minimize a function of N parameters (e.g. x[1],x[2],x[3]...,x[N]) where the boundaries for the minimization depend on the minimized parameters themselves. For instance, assume that all values of x could vary between 0 and 1 in such a way that the summing then all I got 1, then I have the following inequalities for the boundaries:
0 <= x[1] <= 1
x[1] <= x[2] <= 1 - x[1]
x[2] <= x[3] <= 1-x[1]-x[2]
...
x[N-1] <= x[N] <= 1-x[1]-x[2]-x[3]-...-x[N]
Does anyone have an idea on how can I construct some algorithm like that on python? Or maybe if I can adopt an existent method from Scipy for example?

As a rule of thumb: As soon as your boundaries depend on the optimization variables, they are inequality constraints instead of boundaries. Using 0-based indices, your inequalities can be written as
# left-hand sides
-x[0] <= 0
x[i] - x[i+1] <= 0 for all i = 0, ..., n-1
# right-hand sides
sum(x[i], i = 0, .., j) - 1 <= 0 for all j = 0, .., n
Both can be expressed by a simple matrix-vector product:
import numpy as np
D_lhs = np.diag(np.ones(N-1), k=-1) - np.diag(np.ones(N))
D_rhs = np.tril(np.ones(N))
def lhs(x):
return D_lhs # x
def rhs(x):
return D_rhs # x - np.ones(x.size)
As a result, you can use scipy.optimize.minimize to minimize your objective function subject to lhs(x) <= 0 and rhs(x) <= 0 like this:
from scipy.optimize import minimize
# minmize expects eqach inequality constraint in the form con(x) >= 0,
# so lhs(x) <= 0 is the same as -1.0*lhs(x) >= 0
con1 = {'type': 'ineq', 'fun': lambda x: -1.0*lhs(x)}
con2 = {'type': 'ineq', 'fun': lambda x: -1.0*rhs(x)}
result = minimize(your_obj_fun, x0=inital_guess, constraints=(con1, con2))

Penalty function method

I am trying to implement penalty function method for minimizing function. I need to find the minimum of Rosenbrok's function.
I am using this penalty function:
First of all, I have found the minimum using scipy.optimize.minimize:
from scipy.optimize import minimize, rosen
rz = lambda x: (1-x[0])**2 + 100*(x[1] - x[0]**2)**2;
h_1 = lambda x: (x[0] - 2 * x[1] + 2);
h_2 = lambda x: (-x[0] - 2 * x[1] + 6);
h_3 = lambda x: (-x[0] + 2 * x[1] + 2);
x0 = [2.3, 5];
cons = ({'type': 'ineq', 'fun': h_1},
{'type': 'ineq', 'fun': h_2},
{'type': 'ineq', 'fun': h_3})
minimize(rz, x0, constraints=cons)
The answer is x: array([ 0.99971613, 0.99942073])
Then I am trying to find the minimum using my implementation of penalty method:
x_c = [2.3, 3];
i = 1;
while i < 1000:
curr_func = lambda x: rz(x) + i*(h_1(x)**2 + h_2(x)**2 + h_3(x)**2)
x_c = minimize(curr_func, x_c).x;
i *= 1.2;
print(answer.x);
Which gives me [ 2.27402022 1.4157964 ] (if I increase the number of iterations, final values are even greater).
Where is the mistake in my implementation?
Thanks.
P.S. Function curr_func is specific for my constraints, of course, when they are all 'inequals' type.

The problem you have is that the h_i in your formula are for equality constraints, whereas the problem you are solving is for inequality constraints, which correspond to the g_i in your formula. Hence, your penalty function should be using terms like min(0, h_1(x))**2 instead of h_1(x)**2. To see why this is the case, just think about what happens if i = 1000 and x is the desired solution (1, 1). Then, the penalty will include a term i * h_1(x)**2 = 1000, which is huge.
Note that I used min instead of max because it seems like the inequality you want to enforce is h_1(x) >= 0. That means as long as h_1(x) >= 0, the penalty should be zero, but as soon as h_1(x) goes negative, you start penalizing. If it's actually h_1(x) <= 0 you want, then you use max (then you'll have to switch h_1 with -h_1 when you use scipy.optimize.minimize).
BTW, since i is usually an index variable, it's probably better to name the penalty weight something else, like a.

How to specify a list of constraints for scipy.optimize?

I'm struggling to figure out how to do this from the optimize manual
(using the minimization routine SLSQP). My problem has an array of variables x0 and from that I'm trying to construct a list of constraints for my function.
L = 2
N = 2
x0 = np.random.uniform(-1.0, 1.0, size=(L*L*N*2,))
x0 = x0.reshape(L,L,N,2)
ff = x0[:, :, :, 0] + 1j*x0[:, :, :, 1]
The constraints need to be of the form: ff[x,y,1]**2 + ff[x,y,2]**2 + ... + ff[x,y,N]**2 = 1
I think this should be doable but I'm learning Python as I go with this problem. I hope it's not a trivial place to have gotten stuck. My attempt so far,
for i_x in range(L):
for i_y in range(L):
cons = {'type':'eq', 'fun': np.sum(np.abs(ff[i_x, i_y, :])**2) - 1}
x0 = x0.reshape(L*L*N*2, )
scipy.optimize.minimize(func, x0, constraints=cons)
And now,
def constraints(x0, L, N, cons):
x0 = x0.reshape(L, L, N, 2)
for i_x in range(L):
for i_y in range(L):
def con(x0, i_x = i_x, i_y = i_y):
return np.sum(np.abs((x0[i_x, i_y, :, 0] + 1j*x0[i_x, i_y, :, 1])**2)) - 1
cons.append({'type':'eq', 'fun': con})
return cons
But I receive the error, "too many indices for the array"

I think a problem with your code is that it overwrites the constraint cons. As a result, the only constraint that the scipy.optimize.minimize function sees is np.sum(np.abs(ff[-1, -1, :])**2) = 1 where -1 means the last element L-1. I suggest initializing an empty list, then appending to it:
cons = []
for i_x in range(L):
for i_y in range(L):
cons.append({'type':'eq', 'fun': ...)
The second problem is that the constraint must be parametrized by the same vector as the function you are trying to minimize, i.e. it must be a function of a vector with shape (L * L * N * 2, ). Yours instead is a function of ff, which has shape (L,L,N,) and is a constant ("frozen" at creation time with the initial value x0) that does not depend on the current iteration of the minimization algorithm.