I am trying to fit a Gaussian model onto gaussian distributed data (x,y) , using scipy's curve_fit. I am trying to tweak the parameters of the fitting, in order to get better fitting. I saw that curve_fit calls scipy.optimize.least_sq with the method LM (Levenberg-Marquardt method). It seems to me that it constructs a function that evaluates the least square criterion at each data point. In my example, I have 8 data points. In my comprehension and according to scipy's documentation gtol is "Orthogonality desired between the function vector and the columns of the Jacobian."
popt, pcov = optimize.curve_fit(parametrized_gaussian, patch_indexes * pixel_size, sub_sig,
p0=p0, jac=gaussian_derivative_wrt_param, maxfev=max_fev, gtol=1e-11, ftol=1e-11, xtol=1e-11)
parametrized_gaussian is simply :
def parametrized_gaussian(x, a, x0, sigma) :
res = a * np.exp(-(x - x0) ** 2 / (2 * sigma ** 2))
return res.astype('float64')
and gaussian_derivative_wrt_param is
def gaussian_derivative_wrt_param(x, a, x0, sigma):
return np.array([parametrized_gaussian(x, a, x0, sigma) / a,
2 * (x - x0) / (sigma ** 2) * parametrized_gaussian(x, a, x0, sigma),
(x - x0) ** 2 / (sigma ** 3) * parametrized_gaussian(x, a, x0, sigma)]). swapaxes(0, -1).astype('float64')
I wanted to check that the value of the jacobian at the resulting optimal parameters. I do not understand the values that I get. When curve_fit calls leastsq, it then uses :
retval = _minpack._lmder(func, Dfun, x0, args, full_output,
col_deriv, ftol, xtol, gtol, maxfev,
factor, diag)
I print Dfun(retval[0]), because retval[0] is the values of optimal parameters. This is what I get.
0.18634,-6175.62246,5660.31995
0.50737, -10685.47212, 6223.84575
0.88394, -7937.93400, 1971.45501
0.98540, 3054.98273, 261.93803
0.70291, 10670.53623, 4479.93075
0.32083, 8746.05579, 6594.01140
0.09370, 3686.25245, 4010.79420
0.01751, 900.40686, 1280.50557
How does this respect gtol ??
Results for Dfun(optimal parameters on the grid of 8 points
That is why I think I do not understand how gtol works.
From scipy/optimize/minpack/lmder.f, we find a more detailed description
c gtol is a nonnegative input variable. termination
c occurs when the cosine of the angle between fvec and
c any column of the jacobian is at most gtol in absolute
c value. therefore, gtol measures the orthogonality
c desired between the function vector and the columns
c of the jacobian.
This just means that if gtol=0, then f(x_optimal) and columns of the jacobian are perpendicular on convergence. If this is the case, then f'(x_optimal).T # f(x_optimal) is a zero matrix. Since this product is used as part of the iteration, it makes sense to stop when this is 0, because no more progress can made.
Related
I'm trying to fit a curve with a Gaussian plus a Lorentzian function, using the curve_fit function from scipy.
def gaussian(x, a, x0, sig):
return a * np.exp(-1/2 * (x - x0)**2 / sig**2)
def lorentzian(x, a, b, c):
return a*c**2/((x-b)**2+c**2)
def decompose(x, z, n, b, *par):
hb_n = gaussian(x, par[0], 4861.3*(1+z), n)
hb_b = lorentzian(x, par[1], 4861.3*(1+z), b)
return hb_b + hb_n
And when I set the p0 parameter, I can get a reasonable result, which fits the curve well.
guess = [0.0001, 2, 10, 3e-16, 3e-16]
p, c = curve_fit(decompose, wave, residual, guess)
fitting parameters
the fitting model and data figure when I set the p0 parameter
But if I set the p0 and bounds parameters simultaneously, the curve_fit function gives the initial guess as the final fitting result, which is rather deviated from the data.
guess = [0.0001, 2, 10, 3e-16, 3e-16]
p, c = curve_fit(decompose, wave, residual, guess, bounds=([-0.001, 0, 0, 0, 0], [0.001, 10, 100, 1e-15, 1e-15]))
fitting parameters
the fitting model and data figure when I set the p0 and bounds parameters simultaneously
I have tried many different combinations of boundaries for the parameters, but the fitting results invariably return the initial guess values. I've been stuck in this problem for a long time. I would be very grateful if anyone can give me some advice to solve this problem.
This happens due to a combination of the optimization algorithm and its parameters.
From the official documentation:
method{‘lm’, ‘trf’, ‘dogbox’}, optional
Method to use for optimization. See least_squares for more details.
Default is ‘lm’ for unconstrained problems and ‘trf’ if bounds are
provided. The method ‘lm’ won’t work when the number of observations
is less than the number of variables, use ‘trf’ or ‘dogbox’ in this
case.
So when you add bound constraints curve_fit will use different optimization algorithm (trust region instead of Levenberg-Marquardt).
To debug the problem you can try to set full_output=True as Warren Weckesser noted in the comments.
In the case of the fit with bounds you will see something similar to:
'nfev': 1
`gtol` termination condition is satisfied.
So the optimization stopped after the first iteration. That's why founed parameters similar to the initial guess.
To fix this you can specify lower gtol parameter. Full list of available parameters you can find here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html#scipy.optimize.least_squares
Example:
Code:
import numpy as np
from matplotlib import pyplot as plt
from scipy.optimize import curve_fit
def gaussian(x, a, x0, sig):
return a * np.exp(-1 / 2 * (x - x0) ** 2 / sig**2)
def lorentzian(x, a, b, c):
return a * c**2 / ((x - b) ** 2 + c**2)
def decompose(x, z, n, b, *par):
hb_n = gaussian(x, par[0], 4861.3 * (1 + z), n)
hb_b = lorentzian(x, par[1], 4861.3 * (1 + z), b)
return hb_b + hb_n
gt_parameters = [-2.42688295e-4, 2.3477827, 1.56977708e1, 4.47455820e-16, 2.2193466e-16]
wave = np.linspace(4750, 5000, num=400)
gt_curve = decompose(wave, *gt_parameters)
noisy_curve = gt_curve + np.random.normal(0, 2e-17, size=len(wave))
guess = [0.0001, 2, 10, 3e-16, 3e-16]
bounds = ([-0.001, 0, 0, 0, 0], [0.001, 10, 100, 1e-15, 1e-15])
options = [
("Levenberg-Marquardt without bounds", dict(method="lm")),
("Trust Region without bounds", dict(method="trf")),
("Trust Region with bounds", dict(method="trf", bounds=bounds)),
(
"Trust Region with bounds + fixed tolerance",
dict(method="trf", bounds=bounds, gtol=1e-36),
),
]
fig, axs = plt.subplots(len(options))
for (title, fit_params), ax in zip(options, axs):
ax.set_title(title)
p, c = curve_fit(decompose, wave, noisy_curve, guess, **fit_params)
fitted_curve = decompose(wave, *p)
ax.plot(wave, gt_curve, label="gt_curve")
ax.plot(wave, noisy_curve, label="noisy")
ax.plot(wave, fitted_curve, label="fitted_curve")
handles, labels = ax.get_legend_handles_labels()
fig.legend(handles, labels)
plt.show()
Dear StackOverflow community,
although the Python package SymPy provides functionality for various QM (e.g. wave functions and operators) and QFT (e.g. gamma matrices) operations, there is no support for functional derivatives.
I would like to implement the following analytically known functional derivative
D f(t) / D f(t') = delta(t - t') (delta distribution)
to compute more interesting results, e.g.,
D F[f] / D f(t)
where ordinary derivation rules apply until D f(t) / D f(t') has to be computed. I have included an example below. I am aware SymPy already supports taking derivatives with respect to functions, but that is not a functional derivative.
With best regards, XeLasar
Example:
F[f] := exp(integral integral f(x) G(x, y) f(y) dx dy)
D F[f] / D f(t) = (integral f(x) * G(x, y) * delta (t - y) dx dy
+ integral delta (t - x) * G(x, y) * f(y) dx dy) * F[f]
= (integral f(x) * G(x, t) dx
+ integral G(t, y) * f(y) dy) * F[f]
= 2 * (integral G(t, t') * f(t') dt') * F[f]
Note: The integral over dt' collapses due to the delta that arises from D f(t) / D f(t')! G(t, t') is an arbitrary symmetric function. The result can be further simplified as integration variables can be renamed.
I found a solution to the problem and I want to share it so others don't have to waste as much time as I did.
Rewriting the Derivative() function in SymPy is unfortunately not an option as derivation rules are defined within expression classes. Although the SymPy source feels like Spaghetti, it is well tested and should not be touched.
It is best to explain with one or two examples what I came up with, but roughly speaking: I found a convenient substitution to perform the functional derivative using the ordinary Derivative() function of SymPy.
Example:
Z[f] = Integral(f(x) G(x,y) f(y), dx, dy) , let f(x) -> A(z), f(y) -> B(z)
= Integral(A(z) G(x,y) B(z), dx, dy)
dZ/dz = Integral(dA/dz G(x,y) B(z) + A(z) G(x,y) dB/dz, x, y)
Re-substitute: A(z) -> f(x) , B(z) -> f(y)
Insert: dA/dz -> delta(t-x), dB/dz -> delta(t-y)
=> DZ[f]/Df(t) = Integral(delta(t-x) G(x,y) f(y) + f(x) G(x,y) delta(t-y), x, y)
= Integral(G(t,y) f(y), y) + Integral(f(x) G(x,t), x)
Although this is not a general proof, this substitution works in my usecases. Since I am using the normal SymPy Derivative() routine, everything works flawlessly.
With best regards, XeLasar
from scipy.optimize import curve_fit
def func(x, a, b):
return a * x + b
x = [0, 1, 2, 3]
y = [160, 317, 3302, 16002]
yerr = [0.0791, 0.0562, 0.0174, 0.0079]
curve_fit(func, x, y, sigma=yerr)
I am trying to do a weighed least square linear regression with the code above using scipy's curve_fit function. From this I need to get the slope and the error of the fit. I think I figured out how to get the slope using the below code, but I don't know how to extract the error of the fit.
# Code to get the slope (correct me if i'm wrong!!)
popt = curve_fit(func, x, y, sigma=yerr)
slope = popt[0]
Thank you!
======================================
Edit: I have been doing some research and I think I might have figured everything out for myself. I will do my best to explain it!
From the image below, I provide this function to curve_fit with a and b being my parameters corresponding to slope and intercept respectively.
When you use curve fit it returns a 1D array, popt, and a 2D array pcov.
popt contains the optimization of my provided parameters, so in this case, popt[0] is slope (green) and popt[1] is intercept (red).
Next, the pcov matrix represents covariance. What I'm mainly looking for here are the values on the diagonal, as these correspond to the variance of each parameter. Again I color coded these so that slope error is pcov[0,0] (green) and intercept error is pcov[1,1] (red).
From this I was able to get the slope of my line and its error. I hope this explanation can be helpful to someone else!!
As said in the comments:
curve_fit is a non linear fit that is definitively not necessary to make a linear regression.
If however used, your code would need to look like:
popt, pcov = curve_fit(func, x, y, sigma=yerr)
slope = popt[0]
That said, it is better to use the linear approach. One approach is given here, with the explanation going like this:
Use numpy.linalg.lstsq to minimize A x = b, where the math behind is using the minimization of ( A x - b ).T ( A x - b )
The weighted problem is ( A x - b ).T W ( A x - b )
With W being diagonal we can write it as W = V.T V so one has
( A x - b ).T W ( A x - b ) =
= ( A x - b ).T V.T V ( A x - b )
= (V A x - V b ).T (V A x - V b )
Hence, it is reduced to the original problem with A -> V A and b -> V b
Note,Wdoes not need to be diagonal, but will be real and symmetric and if an inverse exists we will be able to diagonalize it to W = Q D Q.T with Q being orthogonal and D diagonal.
I've got a little project for my college and I need to write a method which fits an array to some function, here's it's part:
def Linear(self,x,a,b):
return a*x+b
def Quadratic(self, x, a,b,c):
return a*(x**2)+b*x+c
def Sinusoid(self,t, a, gam, omega, phi, offset):
return np.e ** (gam * t) * a * np.sin((2 * np.pi * omega * t) + phi) + offset
def Fit(self, name):
func= getattr(App, name)
self.fit_params, self.covariance_matrix = curve_fit(func, self.t, self.a, maxfev= 100000)
But it returns absolutely wrong values and also doesn't even work for Sinusoid function (ptimizeWarning: Covariance of the parameters could not be estimated warnings.warn('Covariance of the parameters could not be estimated'). I've already checked if it's not an issue with getattr function but it works correctly.
I'm running out of ideas where the issue is.
There are several problems here.
Linear, Quadratic and Sinusoid do not need self, you can define those as staticmethod:
#This is the decorator use to define staticmethod
#staticmethod
def Linear(x,a,b):
return a*x+b
The same applies to other methods (except Fit).
It will help when they are called in the future. When you call them with curve_fit(func,...), for example, if func is Sinusoid, what you are doing is
curve_fit(Sinusoid, ...) and not curve_fit(self.Sinusoid,...). When curve_fit use Sinusoid, the first argument may be understood as the self, therefore, you get and error.
If you define
#staticmethod
def Sinusoid(t, a, gam, omega, phi, offset):
return np.e ** (gam * t) * a * np.sin((2 * np.pi * omega * t) + phi) + offset
a, gam, omega, phi and offset are constants to fit. Therefore, when you call curve_fit, you need to pass this constants as params:
scipy.optimize.curve_fit(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False, check_finite=True, bounds=- inf, inf, method=None, jac=None, **kwargs)
To clarify: The first argument is the func, the second is your experimental data for x, and the third, is the data for y. And after that all the others kwargs, like maxfev. I don't know if self.a is the data for y, or if you want to set an initial value for a. I'm going to guess you want to set the initial value of a. You need to do it like this: p0=[x1,x2,x3,x4,..., xn] where xi is the initial value of the i-th constant. If you don't pass p0 as an argument to curve_fit, the default value for each constant is going to be 1. But here it comes the last problem:
The diferent functions use a diferent number arguments, so for the first one, Linear, you need to do p0 = [a,1]. For the second, p0 = [a, 1, 1]. And for the last one, the Sinusoid one, p0 = [a, 1, 1, 1, 1]
I don't really know if you can pass p0 = [a] to all of the functions and curve_fit will magically understand. But try so and if curve_fit doesn't let you, use condictionals:
def Fit(self, name):
if func = Linear:
p0 = [a,1]
elif func = Quadratic:
p0 = [a,1,1]
elif func = Sinusoid:
p0 = [a,1,1,1,1]
self.fit_params, self.covariance_matrix = curve_fit(func, self.t, self.y_values, p0=p0, maxfev= 100000)
Where self.y_values is the experimental data as an array-like element.
If self.a is the data for y you can cut all of this mambojambo and let Fit define as it currently is. But don't forguet the #staticmethod in the other functions.
Hope this solves your problem.
I have a function, a gaussian, I have fitted this to my data from a data file. I now need to integrate the gaussian function to give the area under it.
This is my gaussian function
def I(theta,max_x,max_y,sigma):
return (max_y/(sigma*(math.sqrt(2*pi))))*np.exp(-((theta-max_x)**2)/(2*sigma**2))
COMPARING WITH GENERAL FORMULA
N(x | mu, sigma, n) := (n/(sigma*sqrt(2*pi))) * exp((-(x-mu)^2)/(2*sigma^2))
i.e n = max_y , MU = max_x , x = theta
this is what is given on another page:
If Phi(z) = integral(N(x|0,1,1), -inf, z); that is, Phi(z) is the integral of the standard normal distribution from >minus infinity up to z, then it's true by the definition of the error function that
Phi(z) = 0.5 + 0.5 * erf(z / sqrt(2)).
Likewise, if Phi(z | mu, sigma, n) = integral( N(x|sigma, mu, n),
-inf, z); that is, Phi(z | mu, sigma, n) is the integral of the normal distribution given parameters mu, sigma, and n from minus infinity up
to z, then it's true by the definition of the error function that
Phi(z | mu, sigma, n) = (n/2) * (1 + erf((x - mu) / (sigma *
sqrt(2)))).
I am unsure how this helps?? I just want to integrate my function over the plotted values under the curve. Is it saying this is the integral:
Phi(z | mu, sigma, n) = (n/2) * (1 + erf((x - mu) / (sigma * sqrt(2))))
The answer you have there is the indefinite integral. If you would like a numerical answer between two x limits, you can evaluate that function at two points and take the difference.
Your gaussian function is defined over all real numbers (−∞, +∞) but in practice, you are only interested in the middle part as the tails are very close to 0. To obtain a numerical estimate of the total area you can do as you say: evaluate the error function at two points suitably close to 0 on each side of the gaussian's peak and take the difference.
If Phi(z | mu, sigma, n) returns a function you could do:
integral = Phi(z | mu, sigma, n)
area = integral(X_HIGH) - integral(X_LOW)