Develop Reference

LMFIT not properly fitting where scipy does with same starting parameter values

I have a complicated curve fitting function:
def corr_function(tau: np.ndarray, BG: float, avg_C: float, vz: float):
wxc = 8.3
wy = 2.5
wz = 3.35
D = 4.4e1
return 1/((math.pi)**(3/2)*wxc*wy*wz*avg_C)*(1 + 4*D*tau/(wxc**2))**(-1/2)*(1 + 4*D*tau/(wy**2))**(-1/2)*(1 + 4*D*tau/(wz**2))**(-1/2)*np.exp(-((vz*tau)**2/(wz**2 + 4*D*tau))) + BG
I tried to fit this with scipy:
popt, pcov = curve_fit(corr_function, tau, corr, [0, 1e-12, 2e5])
and lmfit
model = Model(corr_function, independent_vars=['tau'])
result = model.fit(
corr,
tau=tau,
BG=Parameter('BG', value=0, min=0),
avg_C=Parameter('avg_C', value=1e-12, min=0),
vz=Parameter('vz', value=2e5, min=0),
)
And while the scipy converges to a proper answer (blue), the lmfit doesn't (orange), where lmfit parameters don't change really at all during fitting
[[Fit Statistics]]
# fitting method = leastsq
# function evals = 61
# data points = 400
# variables = 3
chi-square = 1.5370e+12
reduced chi-square = 3.8714e+09
Akaike info crit = 8833.74620
Bayesian info crit = 8845.72059
## Warning: uncertainties could not be estimated:
BG_guess: at boundary
avg_C_guess: at initial value
avg_C_guess: at boundary
[[Variables]]
BG: 0.00000000 (init = 0)
avg_C: 3.9999e-12 (init = 4e-12)
vz: 8831416.63 (init = 200000)
I think I need lmfit to sample a larger parameter space (or more iterations), anyone know how to do this?
Also, note, I need to the input parameters to be static (can't bring them closer to proper fit), as I'll need to automate fitting for large parameter spaces

Scipy minimize / Scipy Curve fit / lmfit

log(VA) = gamma - (1/eta)log[alphaL^(-eta) + betaK^(-eta)]
I'm trying to estimate the above function with nonlinear least squares. I am using 3 different packages (Scipy-minimize, Scipy-curve_fit and lmfit - Model) for this but I find different parameter results in each one. I can't understand why. I would be very grateful if anyone can help with a solution or offer a different solution method.
SCIPY-MINIMIZE
import numpy as np
from scipy.optimize import minimize, curve_fit
from lmfit import Model, Parameters
L = np.array([0.299, 0.295, 0.290, 0.284, 0.279, 0.273, 0.268, 0.262, 0.256, 0.250])
K = np.array([2.954, 3.056, 3.119, 3.163, 3.215, 3.274, 3.351, 3.410, 3.446, 3.416])
VA = np.array([0.919, 0.727, 0.928, 0.629, 0.656, 0.854, 0.955, 0.981, 0.908, 0.794])
def f(param):
gamma = param[0]
alpha = param[1]
beta = param[2]
eta = param[3]
VA_est = gamma - (1/eta)*np.log(alpha*L**-eta + beta*K**-eta)
return np.sum((np.log(VA) - VA_est)**2)
bnds = [(1, np.inf), (0,1),(0,1),(-1, np.inf)]
x0 = (1,0.01,0.98, 1)
con = {"type":"eq", "fun":c}
result = minimize(f, x0, bounds = bnds)
print(result.fun)
print(result.message)
print(result.x[0],result.x[1],result.x[2],result.x[3])
SCIPY-MINIMIZE - OUT
0.30666062040617503
CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL
1.0 0.5587147011643757 0.9371430857380681 5.873041615873815
SCIPY-CURVE_FIT
def f(X, gamma, alpha, beta, eta):
L,K = X
return gamma - (1/eta) * np.log(alpha*L**-eta + beta*K**-eta)
p0 = 1,0.01,0.98, 1
res, cov = curve_fit(f, (L, K), np.log(VA), p0, bounds = ((1,0,0,-1),(np.inf,1,1,np.inf)) )
gamma, alpha, beta, eta = res[0],res[1],res[2],res[3]
gamma, alpha, beta, eta
SCIPY-CURVE_FIT - OUT
(1.000000000062141,
0.26366547263939205,
0.9804436474926481,
13.449747863921704)
LMFIT-MODEL
def f(x, gamma, alpha, beta, eta):
L = x[0]
K = x[1]
return gamma - (1/eta)*np.log(alpha*L**-eta + beta*K**-eta)
fmodel = Model(f)
params = Parameters()
params.add('gamma', value = 1, vary=True, min = 1)
params.add('alpha', value = 0.01, vary=True, max = 1, min = 0)
params.add('beta', value = 0.98, vary=True, max = 1, min = 0)
params.add('eta', value = 1, vary=True, min = -1)
result = fmodel.fit(np.log(VA), params, x=(L,K))
print(result.fit_report())
LMFIT-MODEL - OUT
[[Model]]
Model(f)
[[Fit Statistics]]
# fitting method = leastsq
# function evals = 103
# data points = 10
# variables = 4
chi-square = 0.31749840
reduced chi-square = 0.05291640
Akaike info crit = -26.4986758
Bayesian info crit = -25.2883354
## Warning: uncertainties could not be estimated:
gamma: at initial value
gamma: at boundary
alpha: at boundary
[[Variables]]
gamma: 1.00000000 (init = 1)
alpha: 1.3245e-13 (init = 0.01)
beta: 0.20130064 (init = 0.98)
eta: 447.960413 (init = 1)

A fitting algorithm always seeks for a local minimizer of the underlying least-squares problem. Note that your problem is convex but not strictly convex. Consequently, there's no unique global minimizer. But each local minimizer is a global one. By evaluating the first function f for each found solution, we can observe that they all have the same objective function value. Hence, each solution is a global minimizer.
Why does each method find a different minimizer? The reason is simple. Each one uses a different algorithm to solve the underlying nonlinear optimization problem, e.g. scipy.optimize.minimize uses the 'L-BFGS-B' algorithm while scipy.optimize.curve_fit uses scipy.optimize.least_squares with the Trust Region Reflective algorithm ('TRF'). In short, you can only expect the same solution for different algorithms for a strictly convex problem.

Iterative reconvolution fitting with measured irf using python and lmfit

I am trying to fit an exponential decay function with convolution to a measured instrument response using python and lmfit.
I am new to python and I am trying to follow the code in https://groups.google.com/group/lmfit-py/attach/90f51c25ebb39a52/deconvol_exp2.py?part=0.1&authuser=0.
import numpy as np
from lmfit import Model
import matplotlib.pyplot as plt
import requests
# Load data
url = requests.get('https://groups.google.com/group/lmfit-py/attach/73a983d40ad945b1/tcspcdatashifted.csv?part=0.1&authuser=0')
x,decay1,irf=np.loadtxt(url.iter_lines(),delimiter=',',unpack=True,dtype='float')
plt.figure(1)
plt.semilogy(x,decay1,x,irf)
plt.show()
# Define weights
wWeights=1/np.sqrt(decay1+1)
# define the double exponential model
def jumpexpmodel(x,tau1,ampl1,tau2,ampl2,y0,x0,args=(irf)):
ymodel=np.zeros(x.size)
t=x
c=x0
n=len(irf)
irf_s1=np.remainder(np.remainder(t-np.floor(c)-1, n)+n,n)
irf_s11=(1-c+np.floor(c))*irf[irf_s1.astype(int)]
irf_s2=np.remainder(np.remainder(t-np.ceil(c)-1,n)+n,n)
irf_s22=(c-np.floor(c))*irf[irf_s2.astype(int)]
irf_shift=irf_s11+irf_s22
irf_reshaped_norm=irf_shift/sum(irf_shift)
ymodel = ampl1*np.exp(-(x)/tau1)
ymodel+= ampl2*np.exp(-(x)/tau2)
z=Convol(ymodel,irf_reshaped_norm)
z+=y0
return z
# convolution using fft (x and h of equal length)
def Convol(x,h):
X=np.fft.fft(x)
H=np.fft.fft(h)
xch=np.real(np.fft.ifft(X*H))
return xch
# assign the model for fitting
mod=Model(jumpexpmodel)
When defining the initial parameters for the fit, I am getting the error.
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
#initialize the parameters - showing error
pars = mod.make_params(tau1=10,ampl1=1000,tau2=10,ampl2=1000,y0=0,x0=10,args=irf)
pars['x0'].vary =True
pars['y0'].vary =True
print(pars)
# fit this model with weights, initial parameters
result = mod.fit(decay1,params=pars,weights=wWeights,method='leastsq',x=x)
# print results
print(result.fit_report())
# plot results
plt.figure(5)
plt.subplot(2,1,1)
plt.semilogy(x,decay1,'r-',x,result.best_fit,'b')
plt.subplot(2,1,2)
plt.plot(x,result.residual)
plt.show()
Based on the documentation for lmfit.model, I suspect, this is because of how argument irf is defined in model as args=(irf). I have tried to pass irf to model instead of params. I have also tried to use **kwargs.
What is the correct way to incorporate irf into the model for convolution and fit the data?

I believe that you want to consider irf as an additional independent variable of the model function - a value that you pass in to the function but is not treated as a variable in the fit.
To do that, just modify the signature of your model function jumpexpmodel() to be the simpler
def jumpexpmodel(x, tau1, ampl1, tau2, ampl2, y0, x0, irf):
the body of the function is fine (in fact the args=(irf) would not have worked because you would have needed to unpack args -- the signature here is really what you wanted anyway).
Then tell lmfit.Model() that irf is an independent variable - the default is that the first argument is the only independent variable:
mod = Model(jumpexpmodel, independent_vars=('x', 'irf'))
Then, when making the parameters, do not include irf or args:
pars = mod.make_params(tau1=10, ampl1=1000, tau2=10, ampl2=1000, y0=0, x0=10)
but rather now pass in irf along with x to mod.fit():
result = mod.fit(decay1, params=pars, weights=wWeights, method='leastsq', x=x, irf=irf)
The rest of your program looks fine and the resulting fit will work reasonably well, giving a report of
[[Model]]
Model(jumpexpmodel)
[[Fit Statistics]]
# fitting method = leastsq
# function evals = 138
# data points = 2797
# variables = 6
chi-square = 3795.52585
reduced chi-square = 1.35991610
Akaike info crit = 865.855713
Bayesian info crit = 901.473529
[[Variables]]
tau1: 50.4330421 +/- 0.68246203 (1.35%) (init = 10)
ampl1: 2630.30664 +/- 20.1552948 (0.77%) (init = 1000)
tau2: 225.392872 +/- 2.75674753 (1.22%) (init = 10)
ampl2: 523.257894 +/- 12.4451921 (2.38%) (init = 1000)
y0: 20.7975212 +/- 0.14165429 (0.68%) (init = 0)
x0: -9.70588133 +/- 0.12597936 (1.30%) (init = 10)
[[Correlations]] (unreported correlations are < 0.100)
C(tau2, ampl2) = -0.947
C(tau1, ampl2) = -0.805
C(tau1, tau2) = 0.706
C(tau1, x0) = -0.562
C(ampl1, x0) = 0.514
C(tau1, ampl1) = -0.453
C(tau2, y0) = -0.426
C(ampl2, y0) = 0.314
C(ampl2, x0) = 0.291
C(tau2, x0) = -0.260
C(tau1, y0) = -0.212
C(ampl1, tau2) = 0.119
and a plot like this:

How to use python's lmfit for retrieving best constrained coefficient rates of ordinary differential equations (ODE)?

I am trying to minimize a loss function over an Ordinary differential equation (ODE) problem, using python.
This loss function is meant to retrieve the best coefficient rates of a user defined ODE.
Despite my efforts and examples found in the internet, all results ended up with ODEs whose solutions returned coefficient values which were outside of their respective bounds.
I know that this can happen for several reasons: choice of minimizer, bad ODE definition, integration step-size, among other things.
Nevertheless, my major problem is to find an example where the minimizer can support integration "Breaks" in the ODE integration.
For example, if I am interested in a simple ODE model, for example the Prey-Predator ODE (Lotka-Volterra), one rule that must be set is that no populations can achieve negative values.
Therefore, if the ODE minimizer for some given reason returns a negative value for any of the two populations modeled (Prey and Predator), the minimizer must stop its integration.
Nevertheless, as it appears, this kind of rule is not supported by common minimizations, at least for the Minimizer objects of the lmfit that I tested.
Even with the bounding rules setted for each Parameter object, the Minimizers continue to interpolate the data beyond the given thresholds (i.e.: negative populations).
For example:
Let's assume that I want to find the best coefficient rates of a Prey-Predator ODE with respect to some empirical data.
In this case, if I use the python's lmfit library for my problem, I could do something like below:
from lmfit import Parameters, report_fit, Minimizer
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import ode
end = '\n'*2 + '-'*50 + '\n'*2
def ode_f(t, xs, ps):
"""Lotka-Volterra predator-prey model."""
if isinstance(ps, Parameters):
Prey_Natality = ps[r'Prey_Natality'].value
Predation_rate = ps['Predation_rate'].value
Predator_Natality = ps['Predator_Natality'].value
Predator_Mortality = ps['Predator_Mortality'].value
Prey_natural_mortality = ps['Prey_natural_mortality'].value
else:
(Prey_Natality, Predation_rate, Predator_Natality,
Predator_Mortality, Prey_natural_mortality) = ps
Prey, Predator = xs
dPrey_dt = ( + Prey_Natality*Prey
- Predation_rate*(Prey)*Predator
- Prey_natural_mortality*Prey)
dPred_dt = ( + Predation_rate* (Prey ) *Predator
+ Predator_Natality*Predator
- Predator_Mortality*Predator)
return [dPrey_dt, dPred_dt]
def solout(y):
if np.any( y <=0):
return -1 # stop integration
else:
return 0
def ode_solver(t,
x0,
ps,
mode='vode',
nsteps=500,
method='bdf'):
"""
Solution to the ODE x'(t) = f(t,x,k) with initial condition x(0) = x0
"""
r = ode(ode_f).set_integrator(mode,
nsteps=nsteps,
method=method)
t0 = t.min()
tmax = t.max()
dt = np.diff(t)[0]
r.set_initial_value(x0, t0).set_f_params(ps)
y = []
times = []
integration_time = r.t
while r.successful() and integration_time < tmax:
r.integrate(integration_time + dt)
integration_time = r.t
yi = r.y
y.append(yi)
times.append(integration_time)
if solout(yi) == -1:
print('Time stoped at: {0}'.format(integration_time))
break
return (np.array(y)).astype(float), times
def ODE_solver_residual_evaluator(ps, ts, data):
x0 = [ps['Prey_Pop'].value, ps['Predator_Pop'].value]
model, times = ode_solver(ts, x0, ps)
# if data.shape[0] <= model.shape[0]:
# data.resize((model.shape[0],data.shape[1]),refcheck=False)
# else:
# model.resize((data.shape[0],data.shape[1]),refcheck=False)
return ( model[:len(data )] - data[:len(model)] )
def residual_dim_reducer(residual_array):
return np.square(residual_array).sum()
if '__main__' == __name__:
########
dt = 10**(-4)
t = np.arange(0, 100, dt)
Prey_initial_pop = 1200
Predator_initial_pop = 50
x0 = np.array([Prey_initial_pop,
Predator_initial_pop])
Prey_Natality = 2.6
Predation_rate = 0.12
Predator_Natality = 0.401
Predator_Mortality = 0.0025
Prey_natural_mortality = 0.001
true_params = np.array((Prey_Natality,
Predation_rate,
Predator_Natality,
Predator_Mortality,
Prey_natural_mortality))
data, times = ode_solver(t, x0, true_params)
data += np.random.lognormal(size=data.shape)*0.5
Param_Names = ['Prey_Natality',
'Predation',
'Predator_Natality',
'Predator_Mortality',
'Prey_natural_mortality']
Populations = ['Prey population',
'Predator population']
for i in range(data.shape[1]):
plt.plot(times, np.real(data[:,i]), 'o', label=r'original {0}'.format(Populations[i]))
plt.legend()
plt.show()
import os
to_save = os.path.join(os.getcwd(), 'original data.png')
plt.savefig(to_save)
print('Creating the minizer object', end=end)
################3
# set parameters incluing bounds
params = Parameters()
params.add('Prey_Pop', value= 100, min=1, max=600)
params.add('Predator_Pop', value=10, min=1, max=400)
params.add('Prey_Natality', value=0.03, min=0.000001, max=3.5)
params.add('Predation_rate', value=0.02, min=0.00003, max=3.5)
params.add('Predator_Natality', value=0.0004, min=0.00001, max=3.2)
params.add('Predator_Mortality', value=0.003, min=0.000001, max=3.2)
params.add('Prey_natural_mortality', value=0.001, min=0.0000001, max=0.2)
fitter = Minimizer(ODE_solver_residual_evaluator,
params,
fcn_args=(t, data),
iter_cb=None,
scale_covar=True,
nan_policy='propagate',
reduce_fcn=residual_dim_reducer,
calc_covar=True,
)
fitted_ODE = fitter.minimize(method='Nelder-Mead')
Optimum_params = fitted_ODE.params.valuesdict()
x0_optimum = np.array([Optimum_params.pop('Prey_Pop'),
Optimum_params.pop('Predator_Pop')])
Y_fitted, times_fitted = ode_solver(t, x0_optimum, Optimum_params.values())
Param_Names = list(params.keys())
print(end, 'Param_Names: {0}'.format(Param_Names))
##################
data = data[:len(Y_fitted)]
Y_fitted = Y_fitted[:len(data)]
from sklearn.metrics import (explained_variance_score,
r2_score,
mean_squared_error)
explained_variance_score = explained_variance_score(Y_fitted, data)
R2 = r2_score(Y_fitted, data)
RMSE = np.sqrt(mean_squared_error(Y_fitted, data))
print('Explained variance: {0} \n R2: {1} \n RMSE: {2}'.format(explained_variance_score,
R2, RMSE))
print(end, 'Fitting by Minizer is complete', end=end)
# display fitted statistics
report_fit(fitted_ODE)
print(end)
######################3
fig2 = plt.figure()
n_ref_markers = 12
markers_on = np.linspace(0, data[:,i].size-1, n_ref_markers).astype(int).tolist()
for i in range(Y_fitted.shape[1]):
plt.plot(times_fitted[:len(times)],
Y_fitted[:len(times),i],
'-',
linewidth=1.1,
label=r'fitted {0} '.format(Param_Names[i]))
plt.plot(np.arange(len(data)), data[:,i], 'o',
markevery=markers_on,
label=r'original {0}'.format(Param_Names[i]))
fig2.legend()
fig2.show()
The returned values are:
Two figures:
A) the original data
B) the original plus the ODE's modeled data according to the best coefficient rates that lmfit returned
A table of returns containing fitting measurements and parameters statistical descriptions.
Returned output:
--------------------------------------------------
Fitting by Minizer is complete
--------------------------------------------------
Models fitting error:
Explained variance: -90.1682809468072
R2: -3125.4358694840084
RMSE: 785.9581933129715
--------------------------------------------------
[[Fit Statistics]]
# fitting method = Nelder-Mead
# function evals = 66437
# data points = 364
# variables = 7
chi-square = 2.2485e+08
reduced chi-square = 629842.649
Akaike info crit = 4867.50583
Bayesian info crit = 4894.78590
## Warning: uncertainties could not be estimated:
[[Variables]]
Prey_Pop: 117.479436 +/- 16.3998313 (13.96%) (init = 100)
Predator_Pop: 397.552948 +/- nan (nan%) (init = 10)
Prey_Natality: 1.05567443 +/- nan (nan%) (init = 0.03)
Predation_rate: 3.46543190 +/- 0.05124925 (1.48%) (init = 0.02)
Predator_Natality: 0.48528830 +/- nan (nan%) (init = 0.0004)
Predator_Mortality: 2.76733581 +/- 0.06777831 (2.45%) (init = 0.003)
Prey_natural_mortality: 0.03928761 +/- 0.00503378 (12.81%) (init = 0.001)
[[Correlations]] (unreported correlations are < 0.100)
C(Predation_rate, Prey_natural_mortality) = -0.596
C(Prey_Pop, Predator_Mortality) = 0.179
C(Predation_rate, Predator_Mortality) = 0.141
C(Prey_Pop, Prey_natural_mortality) = 0.127
C(Predator_Mortality, Prey_natural_mortality) = -0.112
This problem came from a didactic example extracted from here:
Any help on the subject would be appreciated.
Sincerely,

Python fitting sinus cardinal and LMFIT library

I have a set of data from a physic experiment (simple-slit experiment) in university and i am trying to fit this data to a model that i build from the lmfit library.
I want a sinus cardinal square, in this form:
I(X)= I0.sinc²(pi.a.X/(lambda.D))
with a : the width of the slit,
lambda : wavelenght of the light
D : distance camera/slit
I0 : original intensity
import csv as csv
from math import pi
import matplotlib.pyplot as plt
import numpy as np
from lmfit import *
# create data to be fitted
with open('data_1.csv', 'r') as f:
values = list(csv.reader(f, delimiter=','))
values = np.array(values[1:], dtype=np.float)
position = values[:, 0]
intensity = values[:, 1]
#define function model
def fct(x, I0, a, D, b):
return I0 * np.square(np.sinc(pi * a * (x + b) / (0.00000063 * D)))
#b is for the horizontal shift because my experience
#was centered on 700 due to the camera
# do fit
vmodel = Model(fct)
vmodel.set_param_hint('I0', min=0., max=300.)
vmodel.set_param_hint('a', value=0.0005, min=0.0, max=1.)
vmodel.set_param_hint('D', value=0.53, min=0.0, max=1.)
vmodel.set_param_hint('b', min=0., max=2000.)
pars = vmodel.make_params()
result = vmodel.fit(intensity, pars, x=position)
# write report
print(result.fit_report())
#after we plot the data, with position on x and intensity on y
It returns totally wrong values and an error :
RuntimeWarning: invalid value encountered in double_scalars spercent =
'({0:.2%})'.format(abs(par.stderr/par.value))
[[Model]]
Model(fct)
[[Fit Statistics]]
# function evals = 7
# data points = 1280
# variables = 4
chi-square = 4058147.794
reduced chi-square = 3180.367
Akaike info crit = 10326.876
Bayesian info crit = 10347.494
[[Variables]]
I0: 0 +/- 0 (nan%) (init= 0)
a: 0.00050000 +/- 0 (0.00%) (init= 0.0005)
D: 0.50000000 +/- 0 (0.00%) (init= 0.5)
b: 400 +/- 0 (0.00%) (init= 400)
Could you help me please ? I tried lots of type models from this library but nothing work correctly and i really need it. I already solved 2D problems with a np.square, and other reading things, the major problem is the model.
Waiting for answers,
Thanks,

You probably want to provide reasonable starting values for all your parameter values. As you wrote it, there are no initial values for I0 or b but conveniently(?) these parameters have bounds set, so that initial values can be inferred (probably poorly) as the lower bound -- I don't know how b became 400. Maybe a typo?
Anyway, I would recommend trying
pars = vmodel.make_params(I0=150, b=400)
and then try the fit again.