Develop Reference

Is there a faster optimization algorithm than SLSQP for my problem?

I have a medium sized optimization problem that I have used scipy optimize with the SLSQP method to solve. I am wondering if there is a faster algorithm?
Here is my code:
from scipy.optimize import minimize, Bounds
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.rand(500,5),columns=['pred','var1','var2','var3','weights'])
def obj(x,df=df):
return -(x*df['pred']).sum()
def c1(x,df=df):
return 5-abs((x*df['var1']).sum())
def c2(x,df=df):
return 5-abs((x*df['var2']).sum())
def c3(x,df=df):
return 5-abs((x*df['var3']).sum())
sol = minimize(
fun=obj,
x0=df['weights'],
method='SLSQP',
bounds=Bounds(-0.03, 0.03),
constraints=[{'type': 'ineq', 'fun': c1},{'type': 'ineq', 'fun': c2},{'type': 'ineq', 'fun': c3}],
options={'maxiter': 1000})
As you can see there are three constraints (sometimes 4 or 5) and the objective is to optimize about 500 weights. There are also bounds. The dataframe df is dense, I don't think there is a single zero.
Is the SLSQP method the fastest way at tackling this problem? I am using google colab.

After setting a random seed by np.random.seed(1) at the top of your code snippet in order to reproduce the results, we can time your code snippet:
In [15]: def foo1():
...: sol = minimize(
...: fun=obj,
...: x0=df['weights'],
...: method='SLSQP',
...: bounds=Bounds(-0.03, 0.03),
...: constraints=[{'type': 'ineq', 'fun': c1},{'type': 'ineq', 'fun': c2},{'type': 'ineq', 'fun': c3}],
...: options={'maxiter': 1000})
...: return sol
...:
In [16]: %timeit foo1()
10.7 s ± 299 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
As already mentioned in the comments, your constraints can be written as linear functions which turns your optimization problem into a linear optimization problem (LP) which can be solved by means of scipy.optimize.linprog. As a rule of thumb: If your problem can be written as an LP instead of an NLP, pursue the LP approach as it's much faster to solve in most cases.
Your constraints basically read as | v.T # x | <= 5 which is simply the absolute value of the dot product (scalar product) of two vectors v and x. Here, v.T denotes the transpose of the vector v and # denotes python's matrix multiplication operator. It's easy to see that
| v1.T # x | <= 5 <=> -5 <= v1.T # x <= 5
| v2.T # x | <= 5 <=> -5 <= v2.T # x <= 5
| v3.T # x | <= 5 <=> -5 <= v3.T # x <= 5
And hence, your LP reads:
min c^T # x
s.t.
v1.T # x <= 5
-v1.T # x <= 5
v2.T # x <= 5
-v2.T # x <= 5
v3.T # x <= 5
-v3.T # x <= 5
-0.03 <= x <= 0.03
This can be solved as follows:
from scipy.optimize import linprog
c = -1*df['pred'].values
v1 = df['var1'].values
v2 = df['var2'].values
v3 = df['var3'].values
A_ub = np.block([v1, -v1, v2, -v2, v3, -v3]).reshape(6, -1)
b_ub = np.array([5, 5, 5, 5, 5, 5])
bounds = [(-0.03, 0.03)]*c.size
res = linprog(c, A_ub=A_ub, b_ub=b_ub, A_eq=None, b_eq=None, bounds=bounds)
Timing this approach yields
In [17]: %timeit res = linprog(c, A_ub=A_ub, b_ub=b_ub, A_eq=None, b_eq=None, bounds=bounds)
2.32 ms ± 163 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
which is roughly 4300x faster.

Where am I going wrong in the following LP code?

I am trying to solve an LP problem with two variables with two constraints where one is inequality and the other one is equality constraint in Scipy.
To convert the inequality in the constraint I have added another variable in it called A.
Min(z) = 80x + 60y
Constraints:
0.2x + 0.32y <= 0.25
x + y = 1
x, y <= 0
I have changed the inequality constraints by the following equations by adding an extra variable A
0.2x + 0.32y + A = 0.25
Min(z) = 80x + 60y + 0A
X+ Y + 0A = 1
from scipy.optimize import linprog
import numpy as np
z = np.array([80, 60, 0])
C = np.array([
[0.2, 0.32, 1],
[1, 1, 0]
])
b = np.array([0.25, 1])
x1 = (0, None)
x2 = (0, None)
sol = linprog(-z, A_eq = C, b_eq = b, bounds = (x1, x2), method='simplex')
However, I am getting an error message
Invalid input for linprog with method = 'simplex'. Length of bounds
is inconsistent with the length of c
How can I fix this?

The problem is that you do not provide bounds for A. If you e.g. run
linprog(-z, A_eq = C, b_eq = b, bounds = (x1, x2, (0, None)), method='simplex')
you will obtain:
con: array([0., 0.])
fun: -80.0
message: 'Optimization terminated successfully.'
nit: 3
slack: array([], dtype=float64)
status: 0
success: True
x: array([1. , 0. , 0.05])
As you can see, the constraints are met:
0.2 * 1 + 0.32 * 0.0 + 0.05 = 0.25 # (0.2x + 0.32y + A = 0.25)
and also
1 + 0 + 0 = 1 # (X + Y + 0A = 1)

Maximizing the value of an equation given a constraint in python

There are four variables
(S1, S2, S3, S4)
with the constraint
(S1+S2+S3+S4=100).
There are four given constants (C1, C2, C3, C4). I want to maximize the value of (S1/C1 + S2/C2 + S3/C3 + S4/C4). Here is my code in python:
#!/usr/bin/env python3
import numpy as np
from scipy.optimize import minimize
S0 = [25, 25, 25, 25]
C = [89415,8991,10944,15164]
def objective(S, C):
total = 0
for index in range(4):
total = total + S[index]/C[index]
return -total
def constraint(S):
return (100 - S[0] - S[1] - S[2] - S[3])
b = (0.0, 100.0)
boundaries = (b,b,b,b)
con = ({'type':'eq', 'fun':constraint})
solution = minimize(objective,S0,args=(C),method='SLSQP',bounds=boundaries,constraints=con)
print (solution)
My code is simply returning the initial guess for S as the final result
fun: -0.0069931517268763755
jac: array([-1.11838453e-05, -1.11222384e-04, -9.13742697e-05, -6.59456709e-05])
message: 'Optimization terminated successfully.'
nfev: 6
nit: 1
njev: 1
status: 0
success: True
x: array([25., 25., 25., 25.])
Where am I going wrong?

It looks like the differences in the output values of your functions are within the default tolerance for the optimizer to stop optimizing your function between iterations. Setting your tolerance to a smaller value like 1e-12 helps with this:
solution = minimize(objective,S0,args=(C),method='SLSQP',bounds=boundaries,constraints=con, tol=1e-12)
Result:
fun: -0.01112223334445557
jac: array([ -1.11837871e-05, -1.11222267e-04, -9.13742697e-05,
-6.59456709e-05])
message: 'Optimization terminated successfully.'
nfev: 192
nit: 32
njev: 32
status: 0
success: True
x: array([ 0.00000000e+00, 1.00000000e+02, 3.01980663e-14,
0.00000000e+00])
which is approximately equal to the absolute maximum solution [0,100,0,0].

Scipy minimize ignores constraint

I have the following code:
def constraint(params):
if abs(params[0] - 15) < 2 and abs(params[1] + 10) < 2:
return -1
else:
return 0
def f(params):
x, z = params
if abs(x - 15) < 2 and abs(z + 10) < 2:
return -9999999
return (x - 15) ** 2 + (z + 10) ** 2 * numpy.sqrt(numpy.abs(numpy.sin(x)))
# Last: 15.00024144, -9.99939634
result = optimize.minimize(f, (-15, -15),
bounds=((-15.01, 15.01,), (-15.01, 15.01,),),
method="SLSQP",
options={'maxiter': 1024 * 1024},
jac=False,
constraints={
'type': 'ineq',
'fun': constraint,
})
print(result)
print(f(result.x))
And it gives the following result:
fun: -9999999.0
jac: array([0., 0.])
message: 'Optimization terminated successfully.'
nfev: 12
nit: 7
njev: 3
status: 0
success: True
x: array([ 15.01 , -11.60831378])
-9999999
The given values [ 15.01, -11.60831378] should be dropped by the constraint (and they were: if I add more verbose logging, I see that constraint function returns -1, but scipy ignores it. Why?
I'm pretty far from data science and maths, so I'm sorry for stupid mistakes if they are there.

To help the algorithm find the right direction, you need to separate your constraints:
def f(params):
print(params)
x, z = params
if abs(x - 15) < 2 and abs(z + 10) < 2:
return -9999999
return (x - 15) ** 2 + (z + 10) ** 2 * numpy.sqrt(numpy.abs(numpy.sin(x)))
# Last: 15.00024144, -9.99939634
result = optimize.minimize(f, (-15, -15),
bounds=((-15.01, 15.01,), (-15.01, 15.01,),),
method="SLSQP",
options={'disp':True, 'maxiter': 1024 * 1024},
jac=False,
constraints=({
'type': 'ineq',
'fun': lambda params : abs(params[0] - 15) -2,
},
{
'type': 'ineq',
'fun': lambda params : abs(params[1] + 10) -2,
},)
)
print(result)
print(f(result.x))
Gives:
Optimization terminated successfully. (Exit mode 0)
Current function value: 6.5928117149596535
Iterations: 6
Function evaluations: 24
Gradient evaluations: 6
fun: 6.5928117149596535
jac: array([-1.2001152, 2.5928117])
message: 'Optimization terminated successfully.'
nfev: 24
nit: 6
njev: 6
status: 0
success: True
x: array([13., -8.])
[13. -8.]
6.5928117149596535
Bingo!

scipy.optimize with SLSQP. 'Singular matrix C in LSQ subproblem'

I'm trying to minimize a dot product of 2 vectors but it doesn't work and I have no idea why. Can someone please help me?
I have a matrix c of this form:
c = [[c11, c12, c13, c14, c15],
[c21, c22, c23, c24, c25]]
I want to get a matrix p of this form:
p = [[p11, p12, p13, p14, p15],
[p21, p22, p23, p24, p25]]
I want to maximize this value :
c11*p11 + c12*p12 +c13*p13 + c14*p14 + c15*p15 + c21*p21 + c22*p22 +c23*p23 + c24*p24 + c25*p25
To get that I convert the c and p to 1-D vector and do the dot product so that my function to maximize is:
f(p) = c.dot(p)
The constraints are:
c11 + c12 + c13 + c14 + c15 = 1
c21 + c22 + c23 + c24 + c25 = 1
every element in p must be between 0.01 and 0.99.
I have tried scipy.optimize.linprog and it works:
from scipy.optimize import linprog
c = np.array([0. , 0. , 0. , 0. , 0. , 0. , 20094.21019108, 4624.08079143, 6625.51724138, 3834.81081081])
A_eq = np.array([[1,1,1,1,1,0,0,0,0,0],
[0,0,0,0,0,1,1,1,1,1]])
b_eq = np.array([1, 1])
res = linprog(-c, A_eq=A_eq, b_eq=b_eq, bounds=(0.01, 0.99))
res
Out[561]:
fun: -19441.285871873002
message: 'Optimization terminated successfully.'
nit: 13
slack: array([0.03, 0.98, 0.98, 0.98, 0.98, 0.98, 0.03, 0.98, 0.98, 0.98, 0. ,
0. , 0.95, 0. , 0. , 0. , 0. , 0. , 0. , 0. ])
status: 0
success: True
x: array([0.96, 0.01, 0.01, 0.01, 0.01, 0.01, 0.96, 0.01, 0.01, 0.0
But I'm trying to use scipy.optimize.minimize with SLSQP instead and that's where I get this 'Singular matrix C in LSQ subproblem' . Here is what I've done:
from scipy.optimize import minimize
def build_objective(ck, sign = -1.00):
"""
Builds the objective fuction for matrix ck
"""
# Here I turn my c matrix to a 1-D matrix
ck = np.concatenate(ck)
def objective(P):
return sign*(ck.dot(P))
return objective
def build_constraint_rows(ck):
"""
Builds the constraint functions that specify that the sum of the proportions for
each bin equals 1
"""
ncol = ck.shape[1]
nrow = ck.shape[0]
constrain_dict = []
for i in range(nrow):
vector = np.zeros((nrow,ncol))
vector[i, :] = 1
vector = np.concatenate(vector)
def row_constrain(P):
return 1 - vector.dot(P)
constrain_dict.append({'type': 'eq', 'fun': row_constrain})
return constrain_dict
# Matrix: Notice that this is not in vector form yet
c = np.array([[0. , 0. , 0. , 0., 0.],
[0. , 20094.21019108, 4624.08079143, 6625.51724138, 3834.81081081]])
# I need some initial p matrix for the function 'minimize'. I look for the value of the row that is the highest and assign it a proportion p of 0.96 and the rest 0.01 so the sum in 1 per row
P_initial = np.ones(c.shape)*0.01
nrow = test.shape[0]
for i in range(nrow):
index= np.where(c[i,] == np.max(c[i,]))[0]
if index.shape[0] > 1:
index = int(np.random.choice(index, size = 1))
else:
index = int(index)
P_initial[i,index] = 0.96
# I turn the P_initial to vector form
P_initial = np.concatenate(P_initial)
# These are the bounds of each p value
b = (0.01,0.99)
bnds = (b,)*c.size
# I then use my previous functions
objective_fun = build_objective(c)
cons = build_constraint_rows(c)
res = minimize(objective_fun,P_initial,method='SLSQP',\
bounds=bnds,constraints=cons)
This is my final result:
res
Out[546]:
fun: -19434.501741138763
jac: array([0. , 0.,0. , 0. ,0. , 0., -20094.21020508, -4624.08056641, -6625.51708984, -3834.81079102])
message: 'Singular matrix C in LSQ subproblem'
nfev: 24
nit: 2
njev: 2
status: 6
success: False
x: array([0.96 , 0.01 , 0.01 , 0.01 , 0.01 ,
0.01020202, 0.95962502, 0.01006926, 0.01001178, 0.01009192])
Please help me understand what I'm doing wrong.
Thank you in advanced,
Karol