I would like to perform some tests on my optimisation routine using scipy.optimize.minimize, in particular graphing the convergence (or the rather objective function) each iteration, over multiple tests.
Suppose I have the following linearly constrained quadratic optimisation problem:
minimise: x_i Q_ij x_j + a|x_i|
subject to: sum(x_i) = 1
I can code this as:
def _fun(x, Q, a):
c = np.einsum('i,ij,j->', x, Q, x)
p = np.sum(a * np.abs(x))
return c + p
def _constr(x):
return np.sum(x) - 1
And I will implement the optimisation in scipy as:
x_0 = # some initial vector
x_soln = scipy.optimise.minimize(_fun, x_0, args=(Q, a), method='SLSQP',
constraints={'type': 'eq', 'fun': _constr})
I see that there is a callback argument but which only accepts a single argument of the parameter values at each iteration. How can I utilise this in a more esoteric case where I might have other arguments that need to be supplied to my callback function?
The way I solved this was use a generic callback cache object referenced each time from my callback function. Let's say you want to do 20 tests and plot the objective function after each iteration in the same chart. You will need an outer loop to run 20 tests, but we'll create that later.
First lets create a class that will store all iteration objective function values for us, and a couple extra bits and pieces:
class OpObj(object):
def __init__(self, Q, a):
self.Q, self.a = Q, a
rv = np.random.rand()
self.x_0 = np.array([rv, (1-rv)/2, (1-rv)/2])
self.f = np.full(shape=(500,), fill_value=np.NaN)
self.count = 0
def _fun(self, x):
return _fun(x, self.Q, self.a)
Also lets add a callback function that manipulates that class obj. Don't worry that it has more than one argument for now since we'll fix this later. Just make sure the first parameter is the solution variables.
def cb(xk, obj=None):
obj.f[obj.count] = obj._fun(xk)
obj.count += 1
All this does is use the object's functions and values to update itself, counting the number of iterations each time. This function will be called after each iteration.
Putting this all together all we need is two more things: 1) some matplotlib-ing to do the plot, and fixing the callback to have only one argument. We can do that with a decorator which is exactly what functools partial does. It returns a function with less arguments than the original. So the final code looks like this:
import matplotlib.pyplot as plt
import scipy.optimize as op
import numpy as np
from functools import partial
Q = np.array([[1.0, 0.75, 0.45], [0.75, 1.0, 0.60], [0.45, 0.60, 1.0]])
a = 1.0
def _fun(x, Q, a):
c = np.einsum('i,ij,j->', x, Q, x)
p = np.sum(a * np.abs(x))
return c + p
def _constr(x):
return np.sum(x) - 1
class OpObj(object):
def __init__(self, Q, a):
self.Q, self.a = Q, a
rv = np.random.rand()
self.x_0 = np.array([rv, (1-rv)/2, (1-rv)/2])
self.f = np.full(shape=(500,), fill_value=np.NaN)
self.count = 0
def _fun(self, x):
return _fun(x, self.Q, self.a)
def cb(xk, obj=None):
obj.f[obj.count] = obj._fun(xk)
obj.count += 1
fig, ax = plt.subplots(1,1)
x = np.linspace(1,500,500)
for test in range(20):
op_obj = OpObj(Q, a)
x_soln = op.minimize(_fun, op_obj.x_0, args=(Q, a), method='SLSQP',
constraints={'type': 'eq', 'fun': _constr},
callback=partial(cb, obj=op_obj))
ax.plot(x, op_obj.f)
ax.set_ylim((1.71,1.76))
plt.show()
Related
I am not sure about the best way to ask this question, but I am trying to find the long-term state of an ODE system with an arbitrary extra constraint that needs to be fulfilled.
Ex:
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
def f(t, X, a, b):
return a*X + b
N = 5
X0 = np.random.randn(N)
t_range = [0, 1]
a = 3
b = 4
sol = solve_ivp(f, t_range, X0, args=(a, b))
for i in range(N):
plt.plot(sol['t'], sol['y'][i, :])
In the code above, f is the right-hand-side of my ODE model. N is the dimension of my state vector X, t_range is the interval I want to view the values of X over, X0 is my initial condition, and a and b are just arbitrary model parameters.
The code written produces the following chart:
However, let's say I didn't want to evaluate X over some fixed t_range. Let's say I wanted to keep solving the ODE until some condition was satisfied:
def bc(X) -> bool:
""" True means we stop solving. """
return (X < 0).any()
In other words, I want to do something like this:
X = X0
while not bc(X):
X = new value from ODE solver
Is there an easy way to do this in scipy? If not, is there a straightforward way to implement this?
Please let me know if I can clarify anything better.
#Lutz Lehmann answered my question in the comments.
If I wanted the solver to stop when one entry in X became < 0:
event = lambda t, X, a, b : min(X) # account for when args are passed to event func
event.terminal # tells it to stop
sol = solve_ivp(f, t_range, X0, args=(a, b), events = event)
I've got a little project for my college and I need to write a method which fits an array to some function, here's it's part:
def Linear(self,x,a,b):
return a*x+b
def Quadratic(self, x, a,b,c):
return a*(x**2)+b*x+c
def Sinusoid(self,t, a, gam, omega, phi, offset):
return np.e ** (gam * t) * a * np.sin((2 * np.pi * omega * t) + phi) + offset
def Fit(self, name):
func= getattr(App, name)
self.fit_params, self.covariance_matrix = curve_fit(func, self.t, self.a, maxfev= 100000)
But it returns absolutely wrong values and also doesn't even work for Sinusoid function (ptimizeWarning: Covariance of the parameters could not be estimated warnings.warn('Covariance of the parameters could not be estimated'). I've already checked if it's not an issue with getattr function but it works correctly.
I'm running out of ideas where the issue is.
There are several problems here.
Linear, Quadratic and Sinusoid do not need self, you can define those as staticmethod:
#This is the decorator use to define staticmethod
#staticmethod
def Linear(x,a,b):
return a*x+b
The same applies to other methods (except Fit).
It will help when they are called in the future. When you call them with curve_fit(func,...), for example, if func is Sinusoid, what you are doing is
curve_fit(Sinusoid, ...) and not curve_fit(self.Sinusoid,...). When curve_fit use Sinusoid, the first argument may be understood as the self, therefore, you get and error.
If you define
#staticmethod
def Sinusoid(t, a, gam, omega, phi, offset):
return np.e ** (gam * t) * a * np.sin((2 * np.pi * omega * t) + phi) + offset
a, gam, omega, phi and offset are constants to fit. Therefore, when you call curve_fit, you need to pass this constants as params:
scipy.optimize.curve_fit(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False, check_finite=True, bounds=- inf, inf, method=None, jac=None, **kwargs)
To clarify: The first argument is the func, the second is your experimental data for x, and the third, is the data for y. And after that all the others kwargs, like maxfev. I don't know if self.a is the data for y, or if you want to set an initial value for a. I'm going to guess you want to set the initial value of a. You need to do it like this: p0=[x1,x2,x3,x4,..., xn] where xi is the initial value of the i-th constant. If you don't pass p0 as an argument to curve_fit, the default value for each constant is going to be 1. But here it comes the last problem:
The diferent functions use a diferent number arguments, so for the first one, Linear, you need to do p0 = [a,1]. For the second, p0 = [a, 1, 1]. And for the last one, the Sinusoid one, p0 = [a, 1, 1, 1, 1]
I don't really know if you can pass p0 = [a] to all of the functions and curve_fit will magically understand. But try so and if curve_fit doesn't let you, use condictionals:
def Fit(self, name):
if func = Linear:
p0 = [a,1]
elif func = Quadratic:
p0 = [a,1,1]
elif func = Sinusoid:
p0 = [a,1,1,1,1]
self.fit_params, self.covariance_matrix = curve_fit(func, self.t, self.y_values, p0=p0, maxfev= 100000)
Where self.y_values is the experimental data as an array-like element.
If self.a is the data for y you can cut all of this mambojambo and let Fit define as it currently is. But don't forguet the #staticmethod in the other functions.
Hope this solves your problem.
I have the following python function required inputs a and b.
def interimfunc(x,y,z):
#this is a dummy function - not a part of the actual question but included for completeness.
#the actual function involves some statistical treatment - but that is not a problem here.
sol = x*y+z
return sol
def finalfunc(a, b):
interimsol1 = interimfunc(0.4,a,b)
interimsol2 = interimfunc(0.8,a,b)
finalsol = interimsol1/interimsol2
return finalsol
if finalsol is a known value.
How do I find out the unknowns "a" and "b" by solving non-linear system of equations??
===
I got 4 down-votes after posting this. I am a mechanical engineer and learning computer science. I did try researching on the internet but need to refine my search - hence I asked a question to experts here.
===
In addition to above equations - there is one more information:
interimsol2 = interimfunc(0.8,a,b)
where interimsol2 = 10 #i.e. known value
How do we inclde this new information in our unknown finding?
===
Actual problem as requested below by #SergeyIvanov
def func(mu, sigma):
tenpercent = st.norm.ppf(2, mu, sigma)
ninetypercent = st.norm.ppf(2, mu, sigma)
rfs = tenpercent/ninetypercent
return rfs
I think you should solve a system of nonlinear equations. This code should solve your problem in case of two equations for two known solutions (of course you can extend it):
from scipy.optimize import fsolve
known_values = [3,5]
def interimfunc(x,y,z):
sol = x*y+z
return sol
def finalfunc(a, b):
interimsol1 = interimfunc(0.4,a,b)
interimsol2 = interimfunc(0.8,a,b)
finalsol = interimsol1/interimsol2
return finalsol
def equations(p):
a, b = p
return (finalfunc(a,b) - known_values[0], # finalfunc(a,b) == solution1
finalfunc(a,b) - known_values[1]) # finalfunc(a,b) == solution2
a, b = fsolve(equations, (1, 1))# solution
print(a,b)
# -6192.07497308 5779.26987919
print(equations((a, b)))
# (1.0000003476651482, -0.99999965233485177) <-- bad convergence beacause there is no free paremeter in finalfunc.
But it works only with equal known_values, that is pointless (solution will be a random combination of a and b). The problem is that you should have something to distinguish two equations finalfunc (e.g. additional parameter), because you can get diffrent solutions only with different arguments. So finally you should have something like this:
from scipy.optimize import fsolve
def interimfunc(x,y,z):
sol = x*y+z
return sol
def finalfunc(a, b, c ):
interimsol1 = interimfunc(0.4,a,b) + c
interimsol2 = interimfunc(0.8,a,b) + c
finalsol = interimsol1/interimsol2
return finalsol
known_values = [0.8260869565217391,0.8333333333333334]
def equations(p):
a, b = p
return (finalfunc(a,b,0) - known_values[0], # finalfunc(a,b,c) == solution1
finalfunc(a,b,1) - known_values[1]) # finalfunc(a,b,c) == solution2
a, b = fsolve(equations, (1, 1))# solution
print(a,b)
print(equations((a, b)))
# 10.0 15.0 <-- correct values
# (4.4408920985006262e-16, 2.2204460492503131e-16) <-- good convergence
For last example:
from scipy.optimize import fsolve
import scipy.stats as st
def equations(p):
mu, sigma = p
tenpercent = st.norm.ppf(2, mu, sigma)
ninetypercent = st.norm.ppf(2, mu, sigma)
return (ninetypercent - 500,
tenpercent / ninetypercent - 1.0)
mu, sigma = fsolve(equations,x0=(100, 10))# solution
print("mu, sigma:",mu, sigma)
print(equations((mu, sigma)))
The problem here is that ppf can generate nan and ruin an optimization process. So guess values should be proposed very carefully.
I am a python beginner, currently using scipy's odeint to compute a coupled ODE system, however, when I run, python shell always tell me that
>>>
Excess work done on this call (perhaps wrong Dfun type).
Run with full_output = 1 to get quantitative information.
>>>
So, I have to change my time step and final time, in order to make it integratable. To do this, I need to try a different combinations, which is quite a pain. Could anyone tell me how can I ask odeint to automatically vary the time step and final time to successfully integrate this ode system?
and here is part of the code which has called odeint:
def main(t, init_pop_a, init_pop_b, *args, **kwargs):
"""
solve the obe for a given set of parameters
"""
# construct initial condition
# initially, rho_ee = 0
rho_init = zeros((16,16))*1j ########
rho_init[1,1] = init_pop_a
rho_init[2,2] = init_pop_b
rho_init[0,0] = 1 - (init_pop_a + init_pop_b)########
rho_init_ravel, params = to_1d(rho_init)
# perform the integration
result = odeint(wrapped_bloch3, rho_init_ravel, t, args=args)
# BUG: need to pass kwargs
# rewrap the result
return from_1d(result, params, prepend=(len(t),))
things = [2*pi, 20*pi, 0,0, 0,0, 0.1,100]
Omega_a, Omega_b, Delta_a, Delta_b, \
init_pop_a, init_pop_b, tstep, tfinal = things
args = ( Delta_a, Delta_b, Omega_a, Omega_b )
t = arange(0, tfinal + tstep, tstep)
data = main(t, init_pop_a, init_pop_b, *args)
plt.plot(t,abs(data[:,4,4]))
where wrapped_bloch3 is the function compute dy/dt.
EDIT: I note you already got an answer here: complex ODE systems in scipy
odeint does not work with complex-valued equations. I get
from scipy.integrate import odeint
import numpy as np
def func(t, y):
return 1 + 1j
t = np.linspace(0, 1, 200)
y = odeint(func, 0, t)
# -> This outputs:
#
# TypeError: can't convert complex to float
# odepack.error: Result from function call is not a proper array of floats.
You can solve your equation by the other ode solver:
from scipy.integrate import ode
import numpy as np
def myodeint(func, y0, t):
y0 = np.array(y0, complex)
func2 = lambda t, y: func(y, t) # odeint has these the other way :/
sol = ode(func2).set_integrator('zvode').set_initial_value(y0, t=t[0])
y = [sol.integrate(tp) for tp in t[1:]]
y.insert(0, y0)
return np.array(y)
def func(y, t, alpha):
return 1j*alpha*y
alpha = 3.3
t = np.linspace(0, 1, 200)
y = myodeint(lambda y, t: func(y, t, alpha), [1, 0, 0], t)
I am trying to write a curve fitting function which returns the optimal parameters a, b and c, here is a simplified example:
import numpy
import scipy
from scipy.optimize import curve_fit
def f(x, a, b, c):
return x * 2*a + 4*b - 5*c
xdata = numpy.array([1,3,6,8,10])
ydata = numpy.array([ 0.91589774, 4.91589774, 10.91589774, 14.91589774, 18.91589774])
popt, pcov = scipy.optimize.curve_fit(f, xdata, ydata)
This works fine, but I want to give the user a chance to supply some (or none) of the parameters a, b or c, in which case they should be treated as constants and not estimated. How can I write f so that it fits only the parameters not supplied by the user?
Basically, I need to define f dynamically with the correct arguments. For instance if a was known by the user, f becomes:
def f(x, b, c):
a = global_version_of_a
return x * 2*a + 4*b - 5*c
Taking a page from the collections.namedtuple playbook, you can use exec to "dynamically" define func:
import numpy as np
import scipy.optimize as optimize
import textwrap
funcstr=textwrap.dedent('''\
def func(x, {p}):
return x * 2*a + 4*b - 5*c
''')
def make_model(**kwargs):
params=set(('a','b','c')).difference(kwargs.keys())
exec funcstr.format(p=','.join(params)) in kwargs
return kwargs['func']
func=make_model(a=3, b=1)
xdata = np.array([1,3,6,8,10])
ydata = np.array([ 0.91589774, 4.91589774, 10.91589774, 14.91589774, 18.91589774])
popt, pcov = optimize.curve_fit(func, xdata, ydata)
print(popt)
# [ 5.49682045]
Note the line
func=make_model(a=3, b=1)
You can pass whatever parameters you like to make_model. The parameters you pass to make_model become fixed constants in func. Whatever parameters remain become free parameters that optimize.curve_fit will try to fit.
For example, above, a=3 and b=1 become fixed constants in func. Actually, the exec statement places them in func's global namespace. func is thus defined as a function of x and the single parameter c. Note the return value for popt is an array of length 1 corresponding to the remaining free parameter c.
Regarding textwrap.dedent: In the above example, the call to textwrap.dedent is unnecessary. But in a "real-life" script, where funcstr is defined inside a function or at a deeper indentation level, textwrap.dedent allows you to write
def foo():
funcstr=textwrap.dedent('''\
def func(x, {p}):
return x * 2*a + 4*b - 5*c
''')
instead of the visually unappealing
def foo():
funcstr='''\
def func(x, {p}):
return x * 2*a + 4*b - 5*c
'''
Some people prefer
def foo():
funcstr=(
'def func(x, {p}):\n'
' return x * 2*a + 4*b - 5*c'
)
but I find quoting each line separately and adding explicit EOL characters a bit onerous. It does save you a function call however.
I usually use a lambda for this purpose.
user_b, user_c = get_user_vals()
opt_fun = lambda x, a: f(x, a, user_b, user_c)
popt, pcov = scipy.optimize.curve_fit(opt_fun, xdata, ydata)
If you want a simple solution based on curve_fit, I'd suggest that you wrap your function in a class. Minimal example:
import numpy
from scipy.optimize import curve_fit
class FitModel(object):
def f(self, x, a, b, c):
return x * 2*a + 4*b - 5*c
def f_a(self, x, b, c):
return self.f(x, self.a, b, c)
# user supplies a = 1.0
fitModel = FitModel()
fitModel.a = 1.0
xdata = numpy.array([1,3,6,8,10])
ydata = numpy.array([ 0.91589774, 4.91589774, 10.91589774, 14.91589774, 18.91589774])
initial = (1.0,2.0)
popt, pconv = curve_fit(fitModel.f_a, xdata, ydata, initial)
There is already a package that does this:
https://lmfit.github.io/lmfit-py/index.html
From the README:
"LMfit-py provides a Least-Squares Minimization routine and class
with a simple, flexible approach to parameterizing a model for
fitting to data. Named Parameters can be held fixed or freely
adjusted in the fit, or held between lower and upper bounds. In
addition, parameters can be constrained as a simple mathematical
expression of other Parameters."
def f(x, a = 10, b = 15, c = 25):
return x * 2*a + 4*b - 5*c
If the user doesn't supply an argument for the parameter in question, whatever you specified on the right-hand side of the = sign will be used:
e.g.:
f(5, b = 20) will evaluate to return 5 * 2*10 + 4*20 - 5*25 and
f(7) will evaluate to return 7 * 2*10 + 4*15 - 5*25