Develop Reference

Maximising a function

experts. I'm trying to maximize a function my_obj with the Nelder-Mead algorithm to fit my data. For this i have taken help from the scipy's optimize.fmin . I think i am very close to the solutions but missing something and getting an error like:

As explained in the scipy.optimize.minimize documentation, you should be using a 1-D array (or a 1-D list because it is compatible) as input for your objective function instead of multiple parameters:
#!/usr/bin/env python
import numpy as np
from scipy.optimize import minimize
d1 = np.array([ 5.0, 10.0, 15.0, 20.0, 25.0])
h = np.array([10000720600.0, 10011506200.0, 10057741200.0, 10178305100.0,10415318500.0])
b = 2.0
cx = 2.0
#objective function
def obj_function(x): # EDIT: Input is a list
m,n,r= x
pw = 1/cx
c = b*cx
x1 = 1+(d1/n)**c
x2 = 1+(d1/m)**c
x3 = (x1/x2)**pw
dcal = (r)*x3
dobs = (h)
deld=((np.log10(dcal)-np.log10(dobs)))**2
return np.sum(deld)
print(obj_function([5.0,10.0,15.0])) # EDIT: Input is a list
x0 = [5.0,10.0,15.0]
print(obj_function(x0))
res = minimize(obj_function, x0, method='nelder-mead')
print(res)
Output:
% python3 script.py
432.6485766651165
432.6485766651165
final_simplex: (array([[7.76285924e+00, 3.02470699e-04, 1.93396980e+01],
[7.76286507e+00, 3.02555020e-04, 1.93397231e+01],
[7.76285178e+00, 3.01100639e-04, 1.93397381e+01],
[7.76286445e+00, 3.01025402e-04, 1.93397169e+01]]), array([0.12196442, 0.12196914, 0.12197448, 0.12198028]))
fun: 0.12196441986340725
message: 'Optimization terminated successfully.'
nfev: 130
nit: 67
status: 0
success: True
x: array([7.76285924e+00, 3.02470699e-04, 1.93396980e+01])

Trying to maximize this simple non linear problem using #gekko but getting this error

(#error: No solution found)
positions = ["AAPL", "NVDA", "MS","CI", "HON"]
cov = df_ret.cov()
ret = df_ret.mean().values
weights = np.array(np.random.random(len(positions)))
def maximize(weights):
std = np.sqrt(np.dot(np.dot(weights.T,cov),weights))
p_ret = np.dot(ret.T,weights)
sharpe = p_ret/std
return sharpe
a = GEKKO()
w1 = a.Var(value=0.2, lb=0, ub=1)
w2 = a.Var(value=0.2, lb=0, ub=1)
w3 = a.Var(value=0.2, lb=0, ub=1)
w4 = a.Var(value=0.2, lb=0, ub=1)
w5 = a.Var(value=0.2, lb=0, ub=1)
a.Equation(w1+w2+w3+w4+w5<=1)
weight = np.array([w1,w2,w3,w4,w5])
a.Obj(-maximize(weight))
a.solve(disp=False)
**** trying to figure out why is it giving no solution as an error
# df_ret is a data frame with returns (for stocks in position)
Df_ret looks like this
# trying to maximize the sharpe ratio
# w(1 to n) are the weights with sum less than or equal to 1****

Here is a solution with gekko:
from gekko import GEKKO
import numpy as np
import pandas as pd
a = GEKKO()
positions = ["AAPL", "NVDA", "MS","CI", "HON"]
df_ret = pd.DataFrame(np.array([[.001729, .014603, .036558, .016772, .001983],
[-0.015906, .006396, .012796, -.002163, 0],
[-0.001849, -.019598, .014484, .036856, .019292],
[.006648, .002161, -.020352, -.007580, 0.022083],
[-.008821, -.014016, -.006512, -.015802, .012583]]))
cov = df_ret.cov().values
ret = df_ret.mean().values
def obj(weights):
std = a.sqrt(np.dot(np.dot(weights.T,cov),weights))
p_ret = np.dot(ret.T,weights)
sharpe = p_ret/std
return sharpe
a = GEKKO()
w = a.Array(a.Var,len(positions),value=0.2,lb=1e-5, ub=1)
a.Equation(a.sum(w)<=1)
a.Maximize(obj(w))
a.solve(disp=False)
print(w)
A couple things that I've done to problem is to use the Array function to create the variable weights w. I also switched to using the gekko sqrt so that it does the automatic differentiation for the objective function. I also added a lower bound of 1e-5 to avoid sqrt(0) and divide by zero. The Obj() function minimizes so I removed the negative sign and use the Maximize() function to make it more readable. It produces this solution for w:
[[1e-05] [0.15810629919] [0.19423029287] [1e-05] [0.6476428726]]
Many are more familiar with scipy. Here is a benchmark problem where the same problem is solved with scipy.minimize.optimize and gekko. There is also a link for that same solution with MATLAB fmincon or gekko with MATLAB.

I am not familiar with GEKKO so I can't really help with that package, but incase someone doesn't answer how to do it using GEKKO, here's a potential solution with scipy.optimize.minimize:
from scipy.optimize import minimize
import numpy as np
import pandas as pd
def OF(weights, cov, ret, sign = 1.0):
std = np.sqrt(np.dot(np.dot(weights.T,cov),weights))
p_ret = np.dot(ret.T,weights)
sharpe = p_ret/std
return sign*sharpe
if __name__ == '__main__':
x0 = np.array([0.2,0.2,0.2,0.2,0.2])
df_ret = pd.DataFrame(np.array([[.001729, .014603, .036558, .016772, .001983],
[-0.015906, .006396, .012796, -.002163, 0],
[-0.001849, -.019598, .014484, .036856, .019292],
[.006648, .002161, -.020352, -.007580, 0.022083],
[-.008821, -.014016, -.006512, -.015802, .012583]]))
cov = df_ret.cov()
ret = df_ret.mean().values
minx0 = np.repeat(0, [len(x0)] , axis = 0)
maxx0 = np.repeat(1, [len(x0)] , axis = 0)
bounds = tuple(zip(minx0, maxx0))
cons = {'type':'ineq',
'fun':lambda weights: 1 - sum(weights)}
res_cons = minimize(OF, x0, (cov, ret, -1), bounds = bounds, constraints=cons, method='SLSQP')
print(res_cons)
print('Current value of objective function: ' + str(res_cons['fun']))
print('Current value of controls:')
print(res_cons['x'])
which outputs:
fun: -2.1048843911794486
jac: array([ 5.17067784e+00, -2.36839056e-04, -6.24716282e-04, 6.56819057e+00,
2.45392323e-04])
message: 'Optimization terminated successfully.'
nfev: 69
nit: 9
njev: 9
status: 0
success: True
x: array([5.47832097e-14, 1.52927443e-01, 1.87864415e-01, 5.32258098e-14,
6.26433468e-01])
Current value of objective function: -2.1048843911794486
Current value of controls:
[5.47832097e-14 1.52927443e-01 1.87864415e-01 5.32258098e-14
6.26433468e-01]
The sign parameter is added here because in order to maximize the objective function you just minimize OF*(-1). I set the default to 1 (minimize), but I pass -1 in args to change it.

using `fsolve` to solve m equations with n unknowns where n<m

Imagine I have two equations with one unknown and I want to use fsolve to solve it:
0 = 0.5*x[0]**2-2
0 = 2-x
Clearly the answer is x=2. I have tried this
import numpy as np; from scipy.optimize import fsolve
def f(x):
r = np.zeros(2)
r[0] = 0.5*x[0]**2-2
r[1] = 2-x[0]
return r
fsolve(f,[0.5])
The error message is "The array returned by a function changed size between calls"
I can't see what is going wrong here. How do I solve this problem?
In general, How do I solve equations where the number of variables is less than the number of equations.
Here is the full message
Traceback (most recent call last):
File "<ipython-input-37-e4f77791f3f6>", line 12, in <module>
fsolve(f,[0.5])
File "... anaconda3/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 148, in fsolve
res = _root_hybr(func, x0, args, jac=fprime, **options)
File ".... /anaconda3/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 227, in _root_hybr
ml, mu, epsfcn, factor, diag)
ValueError: The array returned by a function changed size between calls

In case of overdetermined system (the number of equations is greater than the number of variables)you need to use, for example, least squares approach. In such cases, there are usually no solution in traditional sense. And we need to define, what should we treat as a solution of the system.
Let you have a system of two equations with one scalar variable:
f(x) = 0
g(x) = 0
This system usually inconsistent and have no solution in traditional sense.
Lets add some values eps1 and eps2 to the right part of the system:
f(x) = 0 + eps1
g(x) = 0 + eps2
eps1 and eps2 are some values;
Now, lets find such x when eps1^2 + eps2^2 is rises its minimum value; that will be a solution of the system
in least squares sense.
To get such solution using scipy you can use least_square function.
Lets look at the following piece of code, that solves your system of equations:
import numpy as np
from scipy.optimize import fsolve, least_squares
def f(x):
r = np.zeros(2)
r[0] = 0.5*x**2-2
r[1] = 2-x
return r
least_squares(f, [0.0])
Result:
active_mask: array([0.])
cost: 5.175333019854869e-20
fun: array([ 2.87759150e-10, -1.43879575e-10])
grad: array([7.19397879e-10])
jac: array([[ 2.00000001],
[-1. ]])
message: '`gtol` termination condition is satisfied.'
nfev: 6
njev: 6
optimality: 7.193978788924559e-10
status: 1
success: True
x: array([2.])

Scipy: Minimize violates given bounds

I am trying to use scipy.optimize.minimize with simple a <= x <= b bounds. However, it often happens that my target function is evaluated just outside the bounds. To my understanding, this happens when minimize tries to determine the gradient of the target function at the boundary.
Minimal example:
import math
import numpy as np
from scipy.optimize import Bounds, minimize
constraint = Bounds([-1, -1], [1, 1], True)
def fun(x):
print(x)
return -math.exp(-np.dot(x,x))
result = minimize(fun, [-1, -1], bounds=constraint)
The output shows that the minimizer jumps to the point [1,1] and then tries to evaluate at [1.00000001, 1]:
[-1. -1.]
[-0.99999999 -1. ]
[-1. -0.99999999]
[-0.72932943 -0.72932943]
[-0.72932942 -0.72932943]
[-0.72932943 -0.72932942]
[-0.22590689 -0.22590689]
[-0.22590688 -0.22590689]
[-0.22590689 -0.22590688]
[1. 1.]
[1.00000001 1. ]
[1. 1.00000001]
[-0.03437328 -0.03437328]
...
Of course, there is no problem in this example, as fun can be evaluated also there. But that might not always be the case...
In my actual problem, the minimum can not be on the boundary and I have the easy workaround of adding an epsilon to the bounds.
But one would expect that there should be an easy solution to this issue which also works if the minimum can be at a boundary?
PS: It would be strange if I were the first to have this problem -- sorry if this question has been asked before somewhere, but I didn't find it anywhere.

As discussed here (thanks #"Welcome to Stack Overflow" for the comment directing me there), the problem is indeed that the gradient routine doesn't respect the bounds.
I wrote a new one that does the job:
import math
import numpy as np
from scipy.optimize import minimize
def gradient_respecting_bounds(bounds, fun, eps=1e-8):
"""bounds: list of tuples (lower, upper)"""
def gradient(x):
fx = fun(x)
grad = np.zeros(len(x))
for k in range(len(x)):
d = np.zeros(len(x))
d[k] = eps if x[k] + eps <= bounds[k][1] else -eps
grad[k] = (fun(x + d) - fx) / d[k]
return grad
return gradient
bounds = ((-1, 1), (-1, 1))
def fun(x):
print(x)
return -math.exp(-np.dot(x,x))
result = minimize(fun, [-1, -1], bounds=bounds,
jac=gradient_respecting_bounds(bounds, fun))
Note that this can be a bit less efficient, because fun(x) now gets evaluated twice at each point.
This seems to be unavoidable, relevant snippet from _minimize_lbfgsb in lbfgsb.py:
if jac is None:
def func_and_grad(x):
f = fun(x, *args)
g = _approx_fprime_helper(x, fun, epsilon, args=args, f0=f)
return f, g
else:
def func_and_grad(x):
f = fun(x, *args)
g = jac(x, *args)
return f, g
As you can see, the value of f can only be reused by the internal _approx_fprime_helper function.

Python3 Scipy: Desired error not necessarily achieved due to precision loss

I'm implementing Andrew Ng's Coursera course in Python and I'm doing Ex2 right now, Logistic Regression. I'm trying to use SciPy's optimize.minimize but I can't seem to get it to run correctly. I'll try to give as brief a summary of my code as possible while being thorough. I'm using Python3. Here is my variable setup, I move everything to numpy after using pandas to read in the csv file:
import numpy as np
import pandas as pd
from scipy.optimize import fmin_bfgs
from scipy import optimize as opt
from scipy.optimize import minimize
class Ex2:
def __init__(self):
self.pandas_data = pd.read_csv("ex2data1.txt", skipinitialspace=True)
self.data = self.pandas_data.values
self.data = np.insert(self.data, 0, 1, axis=1)
self.x = self.data[:, 0:3]
self.y = self.data[:, 3:]
self.theta = np.zeros(shape=(self.x.shape[1]))
x: (100, 3) numpy ndarray
y: (100, 1) numpy ndarray
theta: (3,) numpy ndarray (1-d)
Then, I define a sigmoid, cost and gradient function to give to Scipy's minimize:
#staticmethod
def sigmoid(x):
return 1/(1 + np.exp(x))
def cost(self, theta):
x = self.x
y = self.y
m = len(y)
h = self.sigmoid(x.dot(theta))
j = (1/m) * ((-y.T.dot(np.log(h))) - ((1-y).T.dot(np.log(1-h))))
return j[0]
def grad(self, theta):
x = self.x
y = self.y
theta = np.expand_dims(theta, axis=0)
m = len(y)
h = self.sigmoid(x.dot(theta.T))
grad = (1/m) * (x.T.dot(h-y))
grad = np.squeeze(grad)
return grad
These take theta, a 1-D numpy ndarray. Cost returns a scalar (the cost associated with the theta given) and gradient returns a 1-D numpy ndarray of updates for theta.
When I then run this code:
def run(self):
options = {'maxiter': 100}
print(minimize(self.cost, self.theta, jac=self.grad, options=options))
ex2 = Ex2()
ex2.run()
I get:
fun: 0.69314718055994529
hess_inv: array([[1, 0, 0],
[0, 1, 0],
[0, 0, 1]])
jac: array([ -0.1 , -12.00921659, -11.26284221])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 106
nit: 0
njev: 94
status: 2
success: False
x: array([ 0., 0., 0.])
Process finished with exit code 0
Can't quite get the formatting right on the output, apologies. That's the gist of what I'm doing, am I returning something from cost or gradient incorrectly? That seems most likely to me but I've been trying various combinations and formats of return values and nothing seems to work. Any help is greatly appreciated.
Edit: Among other things, to debug this I've made sure that cost and grad are returning what I expect, which they are (cost: float, grad: 1-D ndarray). Running both on an initial theta array of zeros gives me the same values as I get in Octave (which I know to be correct thanks to the provided code for the exercises). However, giving these values to the minimize function does not seem to be minimizing the theta values as expected.

If anyone stumbles across this and happens to have the same problem, I figured out that in my sigmoid function I should have had
return 1/(1 + np.exp(-x))
but had
return 1/(1 + np.exp(x))
After fixing that, the minimize function converged normally.