Develop Reference

Scipy.optimize violating/not respecting constraint

The problem definition is as follows:
Objective function: maximize Z = 45x1 + 20x2
Constraint 1 (material): 20x1 + 5x2 ≤ 9500
Constraint 2 (time): 0.04x1 + 0.12x2 ≤ 40
Constraint 3 (storage): x1 + x2 ≤ 550
Positivity: x1, x2 ≥ 0
This is done as follow in Python:
# import libraries
import numpy as np
import scipy.optimize as so
# Initialization
bnds = ((0, 550), (0, 550))
initial = np.asarray([400, 150])
# objective function
def maxZ(x_in, sign=1.0):
x1, x2 = x_in[0:2]
return sign * (45 * x1 + 20 * x2)
# constraints
def Cmaterial(x_in):
x1, x2 = x_in[0:2]
return 20 * x1 + 5 * x2 - 9500
def Ctime(x_in):
x1, x2 = x_in[0:2]
return 0.04 * x1 + 0.12 * x2 - 40
def Cstorage(x_in):
x1, x2 = x_in[0:2]
return x1 + x2 - 550
# constraint terms
con1 = {'type':'ineq', 'fun':Cmaterial}
con2 = {'type':'ineq', 'fun':Ctime}
con3 = {'type':'ineq', 'fun':Cstorage}
cons = [con1, con2, con3]
# Optimization
out = so.minimize(maxZ, initial, method='SLSQP', bounds=bnds, constraints=cons, args=(1.0,))
print(f"Optimum solution occurs at {out.x}, where Z = {out.fun}")
print(f"Material: {Cmaterial(out.x) + 9500}")
print(f"Production time: {Ctime(out.x) + 40}")
print(f"Storage: {Cstorage(out.x) + 550}")
The outcome is:
Optimum solution occurs at [427.27272727 190.90909091], where Z = 23045.45454545614
Material: 9500.00000000076
Production time: 40.00000000000009
Storage: 618.1818181818464
I have verified through graphical method and Excel verification that the expected result should be x1 = 450, x2 = 100.
The result from Scipy.optimize is x1 = 427.27, x2 = 190.91.
My question: the storage constraint of x1 + x2 ≤ 550 is clearly violated since the result is 618.18. What could be the reason for this?

First, you need to transform your maximization problem into a minimization problem, i.e. the sign argument inside maxZ should be -1.0, not 1.0. Note also that scipy.optimize.minimize expects inequality constraints with g(x) >= 0, not g(x) <= 0. Hence, you have to transform your constraints accordingly:
con1 = {'type':'ineq', 'fun': lambda x: -1.0*Cmaterial(x)}
con2 = {'type':'ineq', 'fun': lambda x: -1.0*Ctime(x)}
con3 = {'type':'ineq', 'fun': lambda x: -1.0*Cstorage(x)}
cons = [con1, con2, con3]
out = so.minimize(maxZ, initial, method='SLSQP', bounds=bnds, constraints=cons, args=(-1.0,))
yields your expected solution. Last but not least, this is a linear optimization problem (LP) and thus, should be solved with scipy.optimize.linprog. However, this requires that you formulate the problem in the standard LP form .

Problem solving a minimization problem in cvxpy

I have a linear optimization problem, which can be expressed in a cost function code like this:
value_to_minimize = 0.0;
for i in range(0, len(v_1)):
value_to_minimize += np.abs(v_1[i] - (v_2[i] * c1 + v_3[i] * c2 + v_4[i] * c3));
The task of the solver should be to find values for the variables c1, c2, c3 which minimize the value. As boundary conditions, c1, c2, c3 together should result in 1.0 and not be negative.
v_1, v_2, v_3 and v_4 are vectors with 10000 float values.
Here is the outline to solve this minimization problem in cvxpy, but without the parameter pass in cp.Minimize(...):
V1 = np.array(v_1).reshape(10000, 1)
V2 = np.array(v_2 + v_3 + v_4).reshape(10000, 3)
c = cp.Variable((3,1),nonneg=True)
prob = cp.Problem(cp.Minimize(..., # ???
[sum(c) == 1]))
prob.solve(verbose=True)
How would the minimize function for cvxpy look in that case?

If you don't mind using another library, I would recommend scipy for this one:
from scipy.optimize import minimize
import numpy as np
def OF(x0, v_1, v_2, v_3, v_4):
value_to_minimize = 0.0
for i in range(0, len(v_1)):
value_to_minimize += np.abs(v_1[i] - (v_2[i] * x0[0] + v_3[i] * x0[1] + v_4[i] * x0[2]))
return value_to_minimize
if __name__ == '__main__':
x0 = np.array([0, 0, 0])
v_1 = np.random.randint(10, size = 10000)
v_2 = np.random.randint(10, size = 10000)
v_3 = np.random.randint(10, size = 10000)
v_4 = np.random.randint(10, size = 10000)
minx0 = np.repeat(0, [len(x0)] , axis = 0)
maxx0 = np.repeat(np.inf, [len(x0)] , axis = 0)
bounds = tuple(zip(minx0, maxx0))
cons = {'type':'eq',
'fun':lambda x0: 1 - sum(x0)}
res_cons = minimize(OF, x0, (v_1, v_2, v_3, v_4), bounds = bounds, constraints=cons, method='SLSQP')
print(res_cons)
print('Current value of objective function: ' + str(res_cons['fun']))
print('Current value of controls:')
print(res_cons['x'])
Output is:
fun: 27919.666908810435
jac: array([5092. , 5672. , 5108.39868164])
message: 'Optimization terminated successfully.'
nfev: 126
nit: 21
njev: 21
status: 0
success: True
x: array([0.33333287, 0.33333368, 0.33333345])
Current value of objective function: 27919.666908810435
Current value of controls:
[0.33333287 0.33333368 0.33333345]
But obviously the actual values here do not mean much since I just used random integers for the v_ values... just a demo that this model would meet your constraint of c values adding to 1 and boundary of not less than zero (negative).
edit update: did not look at the OF/constraints closely enough to realize this was a linear problem... should be using a linear solver algorithm (though, you can use a nonlinear, it's overkill though).
scipy's linear solvers are not great for complex optimization problems like this one, reverting back to cvxpy :
import numpy as np
import cvxpy as cp
# Create two scalar optimization variables.
x = cp.Variable()
y = cp.Variable()
z = cp.Variable()
v_1 = np.random.randint(10, size = 10000)
v_2 = np.random.randint(10, size = 10000)
v_3 = np.random.randint(10, size = 10000)
v_4 = np.random.randint(10, size = 10000)
constraints = [x + y + z == 1, x >= 0, y >= 0, z >= 0]
objective = cp.Minimize(cp.sum(cp.abs(v_1 - (v_2 * x + v_3 * y + v_4 * z))))
prob = cp.Problem(objective, constraints)
print("Value of OF:", prob.solve())
print('Current value of controls:')
print(x.value, y.value, z.value)
output:
Value of OF: 27621.999978414093
Current value of controls:
0.3333333333016109 0.33333333406414983 0.3333333326298208

I strongly recommend removing one of the parameters and a constraint. If you know that c1 + c2 + c3 = 1., then use c3 = 1. - c1 - c2! This makes the task of minimizer much easier. Also if v_1 etc. are numpy arrays, then use them as arrays, e.g.,
c3 = 1. - c1 - c2
value_to_minimize = np.sum(np.abs(v_1 - (v_2 * c1 + v_3 * c2 + v_4 * c3)))

Python scipy fsolve to solve a large number of nonlinear equations

I want to solve a system of 6 nonlinear equations using Python. I found that I can use scipy's fsolve pretty easily to solve a system of 3 nonlinear equations. However, when I expand this to a larger system, I find that the solution does not solve the system of equations. Is there something I can correct that will allow for the solution of 6 nonlinear equations?
import numpy as np
from scipy.optimize import fsolve
def system(z):
#arbitrary system of 3 nonlinear equations
x1 = z[0]
x2 = z[1]
x3 = z[2]
F = np.empty((3))
F[0] = 20* x1 + x2**2
F[1] = x2 - x1
F[2] = x3 + 5 - x1*x2
return F
def system2(z):
#arbitrary system of 6 nonlinear equations
x1 = z[0]
x2 = z[1]
x3 = z[2]
x4 = z[3]
x5 = z[4]
x6 = z[5]
F = np.empty((6))
F[0] = 20* x1 + x2**2
F[1] = x2 - x1
F[2] = x3 + 5 - x1*x2
F[3] = x3 + x2
F[4] = x5 + x4**2
F[5] = x6**2 + x1 - 20
return F
uInitial = np.array([1,1,1])
u = fsolve(system,uInitial)
print('Solution: ',u)
print('Solution check: ',system(u),'\n') #yields zeros as expected
vInitial = np.array([1,1,1,1,1,1])
v = fsolve(system2,vInitial)
print('Solution: ',v)
print('Solution check: ',system2(v)) #unexpectedly does not yield zeros. Equations not solved correctly.
When applying the given solution back into the system of equations, I should expect to receive zeros (or nearly zero). This would confirm that the computed solution solves the given set of equations. I tried checking with this method for both the system of 3 equations and the system of 6 equations, but only the system of 3 equations is solved correctly with this check. What can I do to solve the system of 6 nonlinear equations?

Your system is inconsistent and your initial guess is off. Try adding fourth equation to the first system of three equations:
F[0] = 20 * x1 + x2**2 # "first" equation
F[1] = x2 - x1 # "second" (=> x1 == x2)
F[2] = x3 + 5 - x1*x2 # "third"
F[3] = x3 + x2 # "fourth" equation (=> x3 == -x2)
First, let's solve first three equations. From the second equation it follows that x1 is equal to x2. Therefore the first equation can be re-written as:
F[0] = 20 * x1 + x1**2
which leads to x1 = -20 (and x2 = -20). Using this in the third equation leads to x3 = 395. Try to modify initial conditions for the first system to uInitial = np.array([-30, -30, 1]) - you should get the correct answer.
Now, let's solve all four equations. The third equation, using the fact that x2 == x1, can be re-written as:
F[2] = x3 + 5 - x1**2
From the fourth equation it follows that x3 == -x2 (and so x3 == -x1 as well). Therefore, this equation can be rewritten as x3 + 5 - x3**2 == 0 => x3 = 0.5 +(-) sqrt(21)/2 which is different from 395 that we got above using first three equations.
This shows that you have an inconsistent system of equations which has no solution.

How to calculate weight to minimize variance?

given several vectors:
x1 = [3 4 6]
x2 = [2 8 1]
x3 = [5 5 4]
x4 = [6 2 1]
I wanna find weight w1, w2, w3 to each item, and get the weighted sum of each vector: yi = w1*i1 + w2*i2 + w3*i3. for example, y1 = 3*w1 + 4*w2 + 6*w3
to make the variance of these values(y1, y2, y3, y4) to be minimized.
notice: w1, w2, w3 should > 0, and w1 + w2 + w3 = 1
I don't know what kind of problems it should be... and how to solve it in python or matlab?

You can start with building a loss function stating the variance and the constraints on w's. The mean is m = (1/4)*(y1 + y2 + y3 + y4). The variance is then (1/4)*((y1-m)^2 + (y2-m)^2 + (y3-m)^2 + (y4-m)^2) and the constraint is a*(w1+w2+w3 - 1) where a is the Lagrange multiplier. The problem looks like to me a convex optimisation with convex constraints since the loss function is quadratic with respect to target variables (w1,w2,w3) and the constraints are linear. You can look for projected gradient descent algorithms which respect to the constraints provided. Take a look to here http://www.ifp.illinois.edu/~angelia/L5_exist_optimality.pdf There are no straightforward analytic solutions to such kind of problems in general.

w = [5, 6, 7]
x1 = [3, 4, 6]
x2 = [2, 8, 1]
x3 = [5, 5, 4]
y1, y2, y3 = 0, 0, 0
for index, i in enumerate(w):
y1 = y1 + i * x1[index]
y2 = y2 + i * x2[index]
y3 = y3 + i * x3[index]
print(min(y1, y2, y3))
I think I maybe get the purpose of your problem.But if you want to find the smallest value, I hope this can help you.
I just make the values fixed, you can make it to be the def when you see this is one way to solve your question.

I don't know much about optimization problem, but I get the idea of gradient descent so I tried to reduce the weight between the max score and min score, my script is below:
# coding: utf-8
import numpy as np
#7.72
#7.6
#8.26
def get_max(alist):
max_score = max(alist)
idx = alist.index(max_score)
return max_score, idx
def get_min(alist):
max_score = min(alist)
idx = alist.index(max_score)
return max_score, idx
def get_weighted(alist,aweight):
res = []
for i in range(0, len(alist)):
res.append(alist[i]*aweight[i])
return res
def get_sub(list1, list2):
res = []
for i in range(0, len(list1)):
res.append(list1[i] - list2[i])
return res
def grad_dec(w,dist, st = 0.001):
max_item, max_item_idx = get_max(dist)
min_item, min_item_idx = get_min(dist)
w[max_item_idx] = w[max_item_idx] - st
w[min_item_idx] = w[min_item_idx] + st
def cal_score(w, x):
score = []
print 'weight', w ,x
for i in range(0, len(x)):
score_i = 0
for j in range(0,5):
score_i = w[j]*x[i][j] + score_i
score.append(score_i)
# check variance is small enough
print 'score', score
return score
# cal_score(w,x)
if __name__ == "__main__":
init_w = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2]
x = [[7.3, 10, 8.3, 8.8, 4.2], [6.8, 8.9, 8.4, 9.7, 4.2], [6.9, 9.9, 9.7, 8.1, 6.7]]
score = cal_score(init_w,x)
variance = np.var(score)
round = 0
for round in range(0, 100):
if variance < 0.012:
print 'ok'
break
max_score, idx = get_max(score)
min_score, idx2 = get_min(score)
weighted_1 = get_weighted(x[idx], init_w)
weighted_2 = get_weighted(x[idx2], init_w)
dist = get_sub(weighted_1, weighted_2)
# print max_score, idx, min_score, idx2, dist
grad_dec(init_w, dist)
score = cal_score(init_w, x)
variance = np.var(score)
print 'variance', variance
print score
In my practice it really can reduce the variance. I am very glad but I don't know whether my solution is solid in math.

My full solution can be viewed in PDF.
The trick is to put the vectors x_i as columns of a matrix X.
Then writing the problem becomes a Convex Problem with constrain of the solution to be on the Unit Simplex.
I solved it using Projected Sub Gradient Method.
I calculated the Gradient of the objective function and created a projection to the Unit Simplex.
Now all needed is to iterate them.
I validated my solution using CVX.
% StackOverflow 44984132
% How to calculate weight to minimize variance?
% Remarks:
% 1. sa
% TODO:
% 1. ds
% Release Notes
% - 1.0.000 08/07/2017
% * First release.
%% General Parameters
run('InitScript.m');
figureIdx = 0; %<! Continue from Question 1
figureCounterSpec = '%04d';
generateFigures = OFF;
%% Simulation Parameters
dimOrder = 3;
numSamples = 4;
mX = randi([1, 10], [dimOrder, numSamples]);
vE = ones([dimOrder, 1]);
%% Solve Using CVX
cvx_begin('quiet')
cvx_precision('best');
variable vW(numSamples)
minimize( (0.5 * sum_square_abs( mX * vW - (1 / numSamples) * (vE.' * mX * vW) * vE )) )
subject to
sum(vW) == 1;
vW >= 0;
cvx_end
disp([' ']);
disp(['CVX Solution - [ ', num2str(vW.'), ' ]']);
%% Solve Using Projected Sub Gradient
numIterations = 20000;
stepSize = 0.001;
simplexRadius = 1; %<! Unit Simplex Radius
stopThr = 1e-6;
hKernelFun = #(vW) ((mX * vW) - ((1 / numSamples) * ((vE.' * mX * vW) * vE)));
hObjFun = #(vW) 0.5 * sum(hKernelFun(vW) .^ 2);
hGradFun = #(vW) (mX.' * hKernelFun(vW)) - ((1 / numSamples) * vE.' * (hKernelFun(vW)) * mX.' * vE);
vW = rand([numSamples, 1]);
vW = vW(:) / sum(vW);
for ii = 1:numIterations
vGradW = hGradFun(vW);
vW = vW - (stepSize * vGradW);
% Projecting onto the Unit Simplex
% sum(vW) == 1, vW >= 0.
vW = ProjectSimplex(vW, simplexRadius, stopThr);
end
disp([' ']);
disp(['Projected Sub Gradient Solution - [ ', num2str(vW.'), ' ]']);
%% Restore Defaults
% set(0, 'DefaultFigureWindowStyle', 'normal');
% set(0, 'DefaultAxesLooseInset', defaultLoosInset);
You can see the full code in StackOverflow Q44984132 (PDF is available as well).

Triple Integration: ValueError: negative number cannot be raised to a fractional power

I'm trying to triple integrate a function.
from scipy.integrate import tplquad
S = 40
P1 = 0.37
P2 = 0.43
P3V = .05
UND = 1 - (P1+P2+P3V)
b1 = S*P1
b2 = S*P2
b3 = S*P3
b4 = S*UND
x1 = 48
x2 = 47
x3 = 4
x4 = 1
tp1 = tplquad(lambda x, y, z: ((x**(b1 + x1 - 1))*(y**(b2 + x2 - 1))*(z**(b3 + x3 - 1))*((1-x-y-z)**(x4+b4-1))), 0, 1, lambda z: z, lambda z: (1-z)/2, lambda x,z: 0, lambda x,z: x)
This is throwing:
ValueError: negative number cannot be raised to a fractional power
I'm trying to integrate the function (x**(b1 + x1 - 1))*(y**(b2 + x2 - 1))*(z**(b3 + x3 - 1))*((1-x-y-z)**(x4+b4-1))
over the limits below:
first, y goes from 0 to x,
then, x goes from z to (1-z)/2,
and then, z goes from 0 to 1.
Can anyone tell me what I'm doing wrong?

I changed it a bit to debug it:
def func(x, y, z):
print(x, y, z)
print(b1 + x1 - 1)
print(b2 + x2 - 1)
print(b3 + x3 - 1)
print(1-x-y-z)
print(x4+b4-1)
return x**(b1 + x1 - 1) * y**(b2 + x2 - 1) * z**(b3 + x3 - 1) * (1-x-y-z)**(x4+b4-1)
So when running the integration:
tp1 = tplquad(func, 0, 1, lambda z: z, lambda z: (1-z)/2, lambda x, z: 0, lambda x,z: x)
I get the following:
0.25 0.375 0.5
61.8
63.2
5.0
-0.125
5.9999999999999964
The -0.125 from the 1-x-y-z is the problem. You're asking Python to calculate the fractional power of a negative number (1-x-y-z)**(x4+b4-1). The result would be complex in most cases.
Even if that wasn't a problem (in python-3.x the fractional power of a negative number isn't a problem!) it would still run into the float-requirement of triplequad:
>>> # Python-3.x
>>> tp1 = tplquad(func, 0, 1, lambda z: z, lambda z: (1-z)/2, lambda x, z: 0, lambda x,z: x)
TypeError: can't convert complex to float
So it seems like something is fundamentally wrong with your function (or the bounds are incorrect). That's something you have to adress. I can only guess what could be done.