I'm trying to minimize a function of N parameters (e.g. x[1],x[2],x[3]...,x[N]) where the boundaries for the minimization depend on the minimized parameters themselves. For instance, assume that all values of x could vary between 0 and 1 in such a way that the summing then all I got 1, then I have the following inequalities for the boundaries:
0 <= x[1] <= 1
x[1] <= x[2] <= 1 - x[1]
x[2] <= x[3] <= 1-x[1]-x[2]
...
x[N-1] <= x[N] <= 1-x[1]-x[2]-x[3]-...-x[N]
Does anyone have an idea on how can I construct some algorithm like that on python? Or maybe if I can adopt an existent method from Scipy for example?
As a rule of thumb: As soon as your boundaries depend on the optimization variables, they are inequality constraints instead of boundaries. Using 0-based indices, your inequalities can be written as
# left-hand sides
-x[0] <= 0
x[i] - x[i+1] <= 0 for all i = 0, ..., n-1
# right-hand sides
sum(x[i], i = 0, .., j) - 1 <= 0 for all j = 0, .., n
Both can be expressed by a simple matrix-vector product:
import numpy as np
D_lhs = np.diag(np.ones(N-1), k=-1) - np.diag(np.ones(N))
D_rhs = np.tril(np.ones(N))
def lhs(x):
return D_lhs # x
def rhs(x):
return D_rhs # x - np.ones(x.size)
As a result, you can use scipy.optimize.minimize to minimize your objective function subject to lhs(x) <= 0 and rhs(x) <= 0 like this:
from scipy.optimize import minimize
# minmize expects eqach inequality constraint in the form con(x) >= 0,
# so lhs(x) <= 0 is the same as -1.0*lhs(x) >= 0
con1 = {'type': 'ineq', 'fun': lambda x: -1.0*lhs(x)}
con2 = {'type': 'ineq', 'fun': lambda x: -1.0*rhs(x)}
result = minimize(your_obj_fun, x0=inital_guess, constraints=(con1, con2))
Related
I have a constrained optimization problem where I am trying to minimize an objective function of 100+ variables which is of the form
Min F(x) = f(x1) + f(x2) + ... + f(xn)
Subject to functional constraint
(g(x1) + g(x2) + ... + g(xn))/(f(x1) + f(x2) + ... + f(xn)) - constant >= 0
I also have individual bounds for each variable x1, x2, x3...xn
a <= x1 <= b
c <= x2 <= d
...
For this, I wrote a python script, using the scipy.optimize.minimize implementation with constraints and bounds, but I am unable to fulfill my bounds and constraints in the solutions. These are all cases where optimization could converge to a solution (message: success)
Here is a sample of my code:
df is my pandas dataset
B(x) is LogNorm transform based on x and other constants
Values U, c, lb, ub are pre-calculated constant dictionaries for each index in df
import scipy
df = pd.DataFrame(..)
k = set(df.index.values) ## list of indexes to iterate on
val = 0.25 ## Arbitrary
def obj(x):
fn = 0
for n,i in enumerate(k):
x0 = x[n]
fn1 = (U[i]) * B(x0) * (x0)
fn += fn1
return fn
def cons(x):
cn = 1
c1 = 0
c2 = 0
for n,i in enumerate(k):
x0 = x[n]
c1 += (U[i]) * (B(x0) * (x0 - c[i])
c2 += (U[i]) * (B(x0) * (x0)
cn = c1/(c2)
return cn - val
const = [{'type':'ineq', 'fun':cons}]
bnds = tuple((lb[i], ub[i]) for i in k) ## Lower, Upper for each element ((lb1, ub1), (lb2, ub2)...)
x_init = [lb[i] for i in k] ## for eg. starting from lower bound
## Solution
sol = scipy.optimize.minimize(obj, x_init, method = 'COBYLA', bounds = bnds, constraints = const)
I have more pointed questions if that helps:
Is there a way to construct the same equation concisely/ without the use of loops (given the number of variables could depend on input data and I have no control over it)?
Is there any noticeable issue in my application of bounds? I can't seem to get the final values of all variables follow individual bounds.
Similarly, is there a visible flaw in the construction on constraint equation? My results often DO NOT follow the constraints is repeated runs with different inputs.
Any help with either of the questions can help me progress further at work.
I have also looked into a Lagrangian solution of the same but so far I am unable to solve it for undefined number of (n) variables.
Thanks!
I'm writing a program to minimize a function of several parameters subjected to constraints and bounds. Just in case you want to run the program, the function is given by:
def Fnret(mins):
Bj, Lj, a, b = mins.reshape(4,N)
S1 = 0; S2 = 0
Binf = np.zeros(N); Linf = np.zeros(N);
for i in range(N):
sbi=(Bi/2); sli=(Li/2)
for j in range(i+1):
sbi -= Bj[j]
sli -= Lj[j]
Binf[i]=sbi
Linf[i]=sli
for i in range(N):
S1 += (C*(1-np.sin(a[i]))+T*np.sin(a[i])) * ((2*Bj[i]*Binf[i]+Bj[i]**2)/(np.tan(b[i])*np.cos(a[i]))) +\
(C*(1-np.sin(b[i]))+T*np.sin(b[i])) * ((2*Bj[i]*Linf[i]+Lj[i]*Bj[i])/(np.sin(b[i])))
S2 += (gamma*Bj[0]/(6*np.tan(b[0])))*((Bi/2)*(Li/2) + 4*(Binf[0]+Bj[0])*(Linf[0]+Lj[0]) + Binf[0]*Linf[0])
j=1
while j<(N):
S2 += (gamma*Bj[j]/(6*np.tan(b[j])))*(Binf[j-1]*Linf[j-1] + 4*(Binf[j]+Bj[j])*(Linf[j]+Lj[j]) + Binf[j]*Linf[j])
j += 1
F = 2*(S1+S2)
return F
where Bj,Lj,a, and b are the minimization results given by N-sized arrays with N being an input of the program, I double-checked the function and it is working correctly. My constraints are given by:
def Brhs(mins): # Constraint over Bj
return np.sum(mins.reshape(4,N)[0]) - Bi
def Lrhs(mins): # Constraint over Lj
return np.sum(mins.reshape(4,N)[1]) - Li
cons = [{'type': 'eq', 'fun': lambda Bj: 1.0*Brhs(Bj)},
{'type': 'eq', 'fun': lambda Lj: 1.0*Lrhs(Lj)}]
In such a way that the sum of all components of Bj must be equal to Bi (same thing with Lj). The bounds of the variables are given by:
bounds = [(0,None)]*2*N + [(0,np.pi/2)]*2*N
For the problem to be reproducible, it's important to use the following inputs:
# Inputs:
gamma = 17.
C = 85.
T = C
Li = 1.
Bi = 0.5
N = 3
For the minimization, I'm using the cyipopt library (that is just similar to the scipy.optimize). Then, the minimization is given by:
from cyipopt import minimize_ipopt
x0 = np.ones(4*N) # Initial guess
res = minimize_ipopt(Fnret, x0, constraints=cons, bounds=bounds)
The problem is that the result is not obeying the conditions I imposed on the constraints (i.e. the sum of Bj or Lj values is different than Bi or Li on the results). But, for instance, if I only write one of the two constraints (over Lj or Bj) it works fine for that variable. Maybe I'm missing something when using 2 constraints and I can't find the error, it seems that it doesn't work with both constraints together. Any help would be truly appreciated. Thank you in advance!
P.S.: In addition, I would like that the function result F to be positive as well. How can I impose this condition? Thanks!
Not a complete answer, just some hints in arbitrary order:
Your initial point x0 is not feasible because it contradicts both of your constraints. This can easily be observed by evaluating your constraints at x0. Under the hood, Ipopt typically detects this and tries to find a feasible initial point. However, it's highly recommended to provide a feasible initial point whenever possible.
Your variable bounds are not well-defined. Evaluating your objective at your bounds yields multiple divisions by zero. For example, the denominator np.tan(b[i]) is zero if and only if b[i] = 0, so 0 isn't a valid value for all of your b[i]s. Proceeding similarly for the other terms, you should obtain 0 < b[i] < pi/2 and 0 ≤ a[i] < pi/2. Here, you can model the strict inequalities by 0 + eps ≤ b[i] ≤ pi/2 - eps and 0 ≤ a[i] ≤ pi/2 - eps, where eps is a sufficiently small positive number.
If you really want to impose that the objective function is always positive, you can simply add the inequality constraint Fnret(x) >= 0, i.e. {'type': 'ineq', 'fun': Fnret}.
In Code:
# bounds
eps = 1e-8
bounds = [(0, None)]*2*N + [(0, np.pi/2 - eps)]*N + [(0+eps, np.pi/2 - eps)]*N
# (feasible) initial guess
x0 = eps*np.ones(4*N)
x0[[0, N]] = [Bi-(N-1)*eps, Li-(N-1)*eps]
# constraints
cons = [{'type': 'eq', 'fun': Brhs},
{'type': 'eq', 'fun': Lrhs},
{'type': 'ineq', 'fun': Fnret}]
res = minimize_ipopt(Fnret, x0, constraints=cons, bounds=bounds, options={'disp': 5})
Last but not least, this still doesn't converge to a stationary point, so chances are that there's indeed no local minimum. From here, you can try experimenting with other (feasible!) initial points and double-check the math of your problem. It's also worth providing the exact gradient and constraint Jacobians.
So, based on #joni suggestions, I could find a stationary point respecting the constraints by adopting the trust-constr method of scipy.optimize.minimize library. My code is running as follows:
import numpy as np
from scipy.optimize import minimize
# Inputs:
gamma = 17
C = 85.
T = C
Li = 2.
Bi = 1.
N = 3 # for instance
# Constraints:
def Brhs(mins):
return np.sum(mins.reshape(4,N)[0]) - Bi/2
def Lrhs(mins):
return np.sum(mins.reshape(4,N)[1]) - Li/2
# Function to minimize:
def Fnret(mins):
Bj, Lj, a, b = mins.reshape(4,N)
S1 = 0; S2 = 0
Binf = np.zeros(N); Linf = np.zeros(N);
for i in range(N):
sbi=(Bi/2); sli=(Li/2)
for j in range(i+1):
sbi -= Bj[j]
sli -= Lj[j]
Binf[i]=sbi
Linf[i]=sli
for i in range(N):
S1 += (C*(1-np.sin(a[i]))+T*np.sin(a[i])) * ((2*Bj[i]*Binf[i]+Bj[i]**2)/(np.tan(b[i])*np.cos(a[i]))) +\
(C*(1-np.sin(b[i]))+T*np.sin(b[i])) * ((2*Bj[i]*Linf[i]+Lj[i]*Bj[i])/(np.sin(b[i])))
S2 += (gamma*Bj[0]/(6*np.tan(b[0])))*((Bi/2)*(Li/2) + 4*(Binf[0]+Bj[0])*(Linf[0]+Lj[0]) + Binf[0]*Linf[0])
j=1
while j<(N):
S2 += (gamma*Bj[j]/(6*np.tan(b[j])))*(Binf[j-1]*Linf[j-1] + 4*(Binf[j]+Bj[j])*(Linf[j]+Lj[j]) + Binf[j]*Linf[j])
j += 1
F = 2*(S1+S2)
return F
eps = 1e-3
bounds = [(0,None)]*2*N + [(0+eps,np.pi/2-eps)]*2*N # Bounds
cons = ({'type': 'ineq', 'fun': Fnret},
{'type': 'eq', 'fun': Lrhs},
{'type': 'eq', 'fun': Brhs})
x0 = np.ones(4*N) # Initial guess
res = minimize(Fnret, x0, method='trust-constr', bounds = bounds, constraints=cons, tol=1e-6)
F = res.fun
Bj = (res.x).reshape(4,N)[0]
Lj = (res.x).reshape(4,N)[1]
ai = (res.x).reshape(4,N)[2]
bi = (res.x).reshape(4,N)[3]
Which is essentially the same just changing the minimization technique. From np.sum(Bj) and np.sum(Lj) is easy to see that the results are in agreement with the constraints imposed, which were not working previously.
I am starting to use sympy. I computed a convolution, but the result was not correct:
This result is wrong: The correct result is
So what did I do wrong? I had used sympy to integrate piecewise fucntions before, with no problems...
The code:
from sympy import *
init_session()
f = lambda x: Piecewise( (1 , (x >= -1) & (x <= 1)) , (0 , True) )
Conv = lambda x: integrate( f(x-y)*f(y) , (y, -oo, +oo) )
There is nothing that you did wrong. It's Sympy having an issue with the product of two piecewise expressions. By calling piecewise_fold(f(x-y)*f(y)) you can see that it does not manage to sort this product out, leaving it as a nested Piecewise construction.
Piecewise((Piecewise((1, And(x - y <= 1, x - y >= -1)), (0, True)), And(y <= 1, y >= -1)), (0, True))
The symbolic integration routine trips up on this nested thing, which may be worth filing an issue on GitHub.
Workaround
If you flatten this nested Piecewise by hand, the integration works correctly.
g = Piecewise((1, And(x-y <= 1, x-y >= -1, y <= 1, y >= -1)), (0, True))
integrate(g, (y, -oo, +oo))
outputs Min(1, x + 1) - Min(1, x + 1, Max(-1, x - 1)) which is correct although perhaps not in the form one would expect.
I'm computing thousands of gradients and would like to vectorize the computations in Python. The context is SVM and the loss function is Hinge Loss. Y is Mx1, X is MxN and w is Nx1.
L(w) = lam/2 * ||w||^2 + 1/m Sum i=1:m ( max(0, 1-y[i]X[i]w) )
The gradient of this is
grad = lam*w + 1/m Sum i=1:m {-y[i]X[i].T if y[i]*X[i]*w < 1, else 0}
Instead of looping through each element of the sum and evaluating the max function, is it possible to vectorize this? I want to use something like np.where like the following
grad = np.where(y*X.dot(w) < 1, -X.T.dot(y), 0)
This does not work because where the condition is true, -X.T*y is the wrong dimension.
edit: list comprehension version, would like to know if there's a cleaner or more optimal way
def grad(X,y,w,lam):
# cache y[i]*X[i].dot(w), each row of Xw is multiplied by a single element of y
yXw = y*X.dot(w)
# cache y[i]*X[i], note each row of X is multiplied by a single element of y
yX = X*y[:,np.newaxis]
# return the average of this max function
return lam*w + np.mean( [-yX[i] if yXw[i] < 1 else 0 for i in range(len(y))] )
you have two vectors A and B, and you want to return array C, such that C[i] = A[i] if B[i] < 1 and 0 else, consequently all you need to do is
C := A * sign(max(0, 1-B)) # suprisingly similar to the original hinge loss, right?:)
since
if B < 1 then 1-B > 0, thus max(0, 1-B) > 0 and sign(max(0, 1-B)) == 1
if B >= 1 then 1-B <= 0, thus max(0, 1-B) = 0 and sign(max(0, 1-B)) == 0
so in your code it will be something like
A = (y*X.dot(w)).ravel()
B = (X*y[:,np.newaxis]).ravel()
C = A * np.sign(np.maximum(0, 1-B))
I am trying to implement penalty function method for minimizing function. I need to find the minimum of Rosenbrok's function.
I am using this penalty function:
First of all, I have found the minimum using scipy.optimize.minimize:
from scipy.optimize import minimize, rosen
rz = lambda x: (1-x[0])**2 + 100*(x[1] - x[0]**2)**2;
h_1 = lambda x: (x[0] - 2 * x[1] + 2);
h_2 = lambda x: (-x[0] - 2 * x[1] + 6);
h_3 = lambda x: (-x[0] + 2 * x[1] + 2);
x0 = [2.3, 5];
cons = ({'type': 'ineq', 'fun': h_1},
{'type': 'ineq', 'fun': h_2},
{'type': 'ineq', 'fun': h_3})
minimize(rz, x0, constraints=cons)
The answer is x: array([ 0.99971613, 0.99942073])
Then I am trying to find the minimum using my implementation of penalty method:
x_c = [2.3, 3];
i = 1;
while i < 1000:
curr_func = lambda x: rz(x) + i*(h_1(x)**2 + h_2(x)**2 + h_3(x)**2)
x_c = minimize(curr_func, x_c).x;
i *= 1.2;
print(answer.x);
Which gives me [ 2.27402022 1.4157964 ] (if I increase the number of iterations, final values are even greater).
Where is the mistake in my implementation?
Thanks.
P.S. Function curr_func is specific for my constraints, of course, when they are all 'inequals' type.
The problem you have is that the h_i in your formula are for equality constraints, whereas the problem you are solving is for inequality constraints, which correspond to the g_i in your formula. Hence, your penalty function should be using terms like min(0, h_1(x))**2 instead of h_1(x)**2. To see why this is the case, just think about what happens if i = 1000 and x is the desired solution (1, 1). Then, the penalty will include a term i * h_1(x)**2 = 1000, which is huge.
Note that I used min instead of max because it seems like the inequality you want to enforce is h_1(x) >= 0. That means as long as h_1(x) >= 0, the penalty should be zero, but as soon as h_1(x) goes negative, you start penalizing. If it's actually h_1(x) <= 0 you want, then you use max (then you'll have to switch h_1 with -h_1 when you use scipy.optimize.minimize).
BTW, since i is usually an index variable, it's probably better to name the penalty weight something else, like a.