I want to fit power-law model (x**m * c) for only two data points to find out the slope m. I am using the curve_fit function from scipy.optimize for this problem. Now when I run the following code
import numpy as np
from scipy.optimize import curve_fit
func = lambda x, m, c: x**m * c
xdata = np.array([235e6, 610e6])
ydata = np.array([0.077, 0.054])
err = np.array([0.0086, 0.0055])
coeff, var = curve_fit(func, xdata, ydata, sigma=err)
print(coeff, var)
It successfully returns the value of m i.e. coeff[0]. But the value of var is [[ inf inf] [ inf inf]]. Is there any problem because of just two data points? It cannot calculate covariance of best fit parameter values? Then how do I calculate error in m?
You have two free parameters and two data points, so the problem is under-constrained. Your fitted curve passes perfectly through the two data points with no error, and so the optimizer cannot calculate a covariance for the parameters.
Related
I'm currently comparing several traditional methods for selecting an exponential trend in a series of points. Trends are often selected in my industry without concern for measures of fit, and this means that a common method is to simply measure yr/yr change and average these results over a period of time. This produces a factor, but doesn't produce a fit, so it's difficult to compare it to an exponential regression or other approach. So my question:
If I have a pre-selected, fixed trend factor for an exponential curve, is there a simple method for optimizing the 'intercept' value which would minimize the squared error of the overall fit to a set of data? Consider the following:
import numpy as np
from sklearn.metrics import r2_score
from scipy.optimize import curve_fit
#Define exponential function
def ex(x,a,b):
return a*b**x
#Seed data with normally distributed error
x=np.linspace(1,20,20)
np.random.seed(100)
y=ex(x,100,1.01)+3*np.random.randn(20)
#Exponential regression for trend value, fitted values, and r_sq measure
popt, pcov = curve_fit(ex, x, y)
trend,fit,r_sq=(popt[1])-1, ex(x,*popt), r2_score(y,ex(x,*popt))
#Mean Yr/Yr change as an alternative measure of trend
trend_yryr=np.mean(np.diff(y)/y[1:])
print(trend)
print(trend_yryr)
The mean year/year change produces a different trend value for the data, and I'd like to compare it to the exponential regression's selected trend. My goal is to find the intercept which would minimize squared error for this alternative trend value over the data, and measure that squared error for comparison. Thanks for the help.
One way of fixing a parameter when using curve_fit is to pass a lambda function that hardcodes the parameters you want to fix. Here's how I did it:
# ... all your preamble, but importing matplotlib
new_b = trend_yryr + 1
popt2, pcov2 = curve_fit(lambda x, a: ex(x, a, new_b), x, y)
fit2, r_sq2 = ex(x, *popt2, new_b), r2_score(y, ex(x, *popt2, new_b))
popt is array([100.24989416, 1.00975864]) and popt2 is array([99.09513612]). Plotting the results gives me this:
You could use lmfit to do this perhaps a bit more elegantly, but it's essentially up to you.
from lmfit import Parameters, minimize
def residual(params, x, y):
a = params['a']
b = params['b']
model = ex(x, a, b)
return model - y
p1 = Parameters()
p1.add('a', value=1, vary=True)
p1.add('b', value=1, vary=True)
p2 = Parameters()
p2.add('a', value=1, vary=True)
p2.add('b', value=np.mean(np.diff(y)/y[1:]) + 1, vary=False) # important
out1 = minimize(residual, p1, args=(x, y))
out2 = minimize(residual, p2, args=(x, y))
The outputs of both methods are essentially the same so I won't post them again
As stated in title, scipy.optimize.minimize fails with "ValueError: setting an array element with a sequence." when calling minimize.
I'm applying scipy.optimize.minimize to a function that uses the variables coef (coefficients that I'm optimizing) and xData and yData (data variables).
I'll provide an example code below. I am aware through searching on how to use minimize that the error stems from the function being minimized returning an array when it should return a scalar. I'm not sure why it is returning an array, though.
Importantly, scipy.optimize.least_squares works and it seems to share the same syntax as scipy.optimize.minimize. scipy.optimize.fmin does not work either and it's included as well - it's the same as minimize with the Nelder-Mead method, which I'm calling.
Here is some generalized example code that has the error on Python 3:
import numpy as np
from scipy.optimize import least_squares
from scipy.optimize import minimize
from scipy.optimize import fmin
import matplotlib.pyplot as plt
xData = np.linspace(50,94,334);
yData = (xData-75)**2 + (np.random.random((334,))-.5)*600;
fun = lambda coef, x : coef[0] + coef[1]*x + coef[2]*x**2 ; #create a "lambda" function whatever that is that has a tuple for the polynomial coefficients in it
#function is y = coef0 + coef1*x + coef2*x^2 where y is lambda
funError = lambda coef, x, y: fun(coef,x) - y; #create a "lambda" function for the error between the real data y and the fit data y
#function is yError = y(coef,x) - yReal where yError is the lambda now
#expanded fully: yError = coef0 + coef1*x + coef2*x^2 - yReal
coef_init = (5,10,15); #initial coefficient guess
#coef0 is const (order 0)
#coef1 is order 1 coef
#coef2 is order 2 coef
coef = least_squares(funError,coef_init, args=(xData,yData) ); #calculate the polynomial coefficients to fit the data
yFit_lq = fun(coef.x,xData); #calc the guessed values
plt.figure();
plt.scatter( xData , yData , 20 , "r" );
plt.scatter( xData , yFit_lq , 20 );
plt.title("Least Squares");
plt.show();
coef = minimize(funError,coef_init, args=(xData,yData),method="Nelder-Mead" ); #calculate the polynomial coefficients to fit the data
yFit_min = fun(coef.x,xData); #calc the guessed values
plt.figure();
plt.scatter( xData , yData , 20 , "r" );
plt.scatter( xData , yFit_min , 20 );
plt.title("Minimize with Nelder-Mead");
plt.show();
coef = fmin(funError,coef_init, args=(xData,yData) ); #calculate the polynomial coefficients to fit the data
yFit_fmin = fun(coef.x,xData); #calc the guessed values
plt.figure();
plt.scatter( xData , yData , 20 , "r" );
plt.scatter( xData , yFit_fmin , 20 );
plt.title("fmin, equiv to min. w/ neldy");
plt.show();
I call least_squares, minimize, and fmin the same way and their pages just ask for args=(). I'm not sure what is going wrong in calling minimize and fmin that the "ValueError: setting an array element with a sequence." error occurs while least_squares is perfectly happy with the formatting.
I would also prefer to avoid excess function defs - the clean and simple lambda function should be able to handle this simple case.
least_squares and minimize have different requirements for the objective function.
least_squares expects your function to return a vector. The docstring describes this vector as the "vector of residuals". least_squares takes this vector and sums the squares of the elements to form the actual objective function that is minimized.
minimize expects your objective function to return a scalar. It tries to find the vector input that minimizes the scalar output of your function.
You can solve the least squares optimization problem with minimize by modifying your existing function so that it computes and returns the sum of the squared residuals:
def funError(coef, x, y):
residuals = fun(coef,x) - y
objective = (residuals**2).sum()
return objective
But then that function is not set up to use with least_squares. So instead, you could use two functions:
def funError(coef, x, y):
residuals = fun(coef,x) - y
return residuals
def funErrorSSR(coef, x, y):
residuals = funError(coef, x, y)
objective = (residuals**2).sum()
return objective
Use funError with least_squares, and funErrorSSR with minimize (or fmin).
I compare fitting with optimize.curve_fit and optimize.least_squares. With curve_fit I get the covariance matrix pcov as an output and I can calculate the standard deviation errors for my fitted variables by that:
perr = np.sqrt(np.diag(pcov))
If I do the fitting with least_squares, I do not get any covariance matrix output and I am not able to calculate the standard deviation errors for my variables.
Here's my example:
#import modules
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.optimize import least_squares
noise = 0.5
N = 100
t = np.linspace(0, 4*np.pi, N)
# generate data
def generate_data(t, freq, amplitude, phase, offset, noise=0, n_outliers=0, random_state=0):
#formula for data generation with noise and outliers
y = np.sin(t * freq + phase) * amplitude + offset
rnd = np.random.RandomState(random_state)
error = noise * rnd.randn(t.size)
outliers = rnd.randint(0, t.size, n_outliers)
error[outliers] *= 10
return y + error
#generate data
data = generate_data(t, 1, 3, 0.001, 0.5, noise, n_outliers=10)
#initial guesses
p0=np.ones(4)
x0=np.ones(4)
# create the function we want to fit
def my_sin(x, freq, amplitude, phase, offset):
return np.sin(x * freq + phase) * amplitude + offset
# create the function we want to fit for least-square
def my_sin_lsq(x, t, y):
# freq=x[0]
# phase=x[1]
# amplitude=x[2]
# offset=x[3]
return (np.sin(t*x[0]+x[2])*x[1]+ x[3]) - y
# now do the fit for curve_fit
fit = curve_fit(my_sin, t, data, p0=p0)
print 'Curve fit output:'+str(fit[0])
#now do the fit for least_square
res_lsq = least_squares(my_sin_lsq, x0, args=(t, data))
print 'Least_squares output:'+str(res_lsq.x)
# we'll use this to plot our first estimate. This might already be good enough for you
data_first_guess = my_sin(t, *p0)
#data_first_guess_lsq = x0[2]*np.sin(t*x0[0]+x0[1])+x0[3]
data_first_guess_lsq = my_sin(t, *x0)
# recreate the fitted curve using the optimized parameters
data_fit = my_sin(t, *fit[0])
data_fit_lsq = my_sin(t, *res_lsq.x)
#calculation of residuals
residuals = data - data_fit
residuals_lsq = data - data_fit_lsq
ss_res = np.sum(residuals**2)
ss_tot = np.sum((data-np.mean(data))**2)
ss_res_lsq = np.sum(residuals_lsq**2)
ss_tot_lsq = np.sum((data-np.mean(data))**2)
#R squared
r_squared = 1 - (ss_res/ss_tot)
r_squared_lsq = 1 - (ss_res_lsq/ss_tot_lsq)
print 'R squared curve_fit is:'+str(r_squared)
print 'R squared least_squares is:'+str(r_squared_lsq)
plt.figure()
plt.plot(t, data)
plt.title('curve_fit')
plt.plot(t, data_first_guess)
plt.plot(t, data_fit)
plt.plot(t, residuals)
plt.figure()
plt.plot(t, data)
plt.title('lsq')
plt.plot(t, data_first_guess_lsq)
plt.plot(t, data_fit_lsq)
plt.plot(t, residuals_lsq)
#error
perr = np.sqrt(np.diag(fit[1]))
print 'The standard deviation errors for curve_fit are:' +str(perr)
I would be very thankful for any help, best wishes
ps: I got a lot of input from this source and used part of the code Robust regression
The result of optimize.least_squares has a parameter inside of it called jac. From the documentation:
jac : ndarray, sparse matrix or LinearOperator, shape (m, n)
Modified Jacobian matrix at the solution, in the sense that J^T J is a Gauss-Newton approximation of the Hessian of the cost function. The type is the same as the one used by the algorithm.
This can be used to estimate the Covariance Matrix of the parameters using the following formula: Sigma = (J'J)^-1.
J = res_lsq.jac
cov = np.linalg.inv(J.T.dot(J))
To find the variance of the parameters one can then use:
var = np.sqrt(np.diagonal(cov))
The SciPy program optimize.least_squares requires the user to provide in input a function fun(...) which returns a vector of residuals. This is typically defined as
residuals = (data - model)/sigma
where data and model are vectors with the data to fit and the corresponding model predictions for each data point, while sigma is the 1σ uncertainty in each data value.
In this situation, and assuming one can trust the input sigma uncertainties, one can use the output Jacobian matrix jac returned by least_squares to estimate the covariance matrix. Moreover, assuming the covariance matrix is diagonal, or simply ignoring non-diagonal terms, one can also obtain the 1σ uncertainty perr in the model parameters (often called "formal errors") as follows (see Section 15.4.2 of Numerical Recipes 3rd ed.)
import numpy as np
from scipy import linalg, optimize
res = optimize.least_squares(...)
U, s, Vh = linalg.svd(res.jac, full_matrices=False)
tol = np.finfo(float).eps*s[0]*max(res.jac.shape)
w = s > tol
cov = (Vh[w].T/s[w]**2) # Vh[w] # robust covariance matrix
perr = np.sqrt(np.diag(cov)) # 1sigma uncertainty on fitted parameters
The above code to obtain the covariance matrix is formally the same as the following simpler one (as suggested by Alex), but the above has the major advantage that it works even when the Jacobian is close to degenerate, which is a common occurrence in real-world least-squares fits
cov = linalg.inv(res.jac.T # res.jac) # covariance matrix when jac not degenerate
If one does not trust the input uncertainties sigma, one can still assume that the fit is good, to estimate the data uncertainties from the fit itself. This corresponds to assuming chi**2/DOF=1, where DOF is the number of degrees of freedom. In this case, one can use the following lines to rescale the covariance matrix before computing the uncertainties
chi2dof = np.sum(res.fun**2)/(res.fun.size - res.x.size)
cov *= chi2dof
perr = np.sqrt(np.diag(cov)) # 1sigma uncertainty on fitted parameters
I'm fitting an experimental spectrum to a theoretical expectation using LeastSq from SciPy. There are of course errors associated with the experimental values. How can I feed these to the LeastSq or do I need a different routine? I find nothing in the documentation.
The scipy.optimize.leastsq function does not have a built-in way to incorporate weights. However, the scipy.optimize.curve_fit function does have a sigma parameter which can be used to indicate the variance of each y-data point.
curve_fit uses 1.0/sigma as the weight, where sigma can be an array of length N, (the same length as ydata).
So somehow you have to surmise the variance of each ydata point based on the size of the error bar and use that to determine sigma.
For example, if you declare that half the length of the error bar represents 1 standard deviation, then the variance (what curve_fit calls sigma) would be the square of the standard deviation.
sigma = (length_of_error_bar/2)**2
Reference:
Wikipedia page on Weighted Least-Squares
I'm in the middle of doing this myself so I will share what I've done and perhaps we can get some comments from the community. I have a collection of data points taken at definite time intervals from which I've calculated standard deviations. I would like to fit these points with a sin function. Leastsq does this by minimizing the residual, or the difference between your data points and the fit function based on a set of parameters, p. We may weight our residuals by dividing them by the variance, or the square of the standard deviation.
As follows:
from scipy.optimize import leastsq
import numpy as np
from matplotlib import pyplot as plt
def sin_func(t, p):
""" Returns the sin function for the parameters:
p[0] := amplitude
p[1] := period/wavelength
p[2] := phase offset
p[3] := amplitude offset
"""
y = p[0]*np.sin(2*np.pi/p[1]*t+p[2])+p[3]
return y
def sin_residuals(p, y, t, std):
err = (y - p[0]*np.sin(2*np.pi/p[1]*t+p[2])-p[3])/std**2
return err
def sin_fit(t, ydata, std, p0):
""" Fits a set of data, ydata, on a domain, t, with individual standard
deviations, std, to a sin curve given the initial parameters, p0, of the form:
p[0] := amplitude
p[1] := period/wavelength
p[2] := phase offset
p[3] := amplitude offset
"""
# optimization #
pbest = leastsq(sin_residuals, p0, args=(ydata, t, std), full_output=1)
p_fit = pbest[0]
# fit to data #
fit = p_fit[0]*np.sin(2*np.pi/p_fit[1]*t+p_fit[2])+p_fit[3]
return p_fit
Hi I want to calculate errors in slope and intercept which are calculated by scipy.polyfit function. I have (+/-) uncertainty for ydata so how can I include it for calculating uncertainty into slope and intercept? My code is,
from scipy import polyfit
import pylab as plt
from numpy import *
data = loadtxt("data.txt")
xdata,ydata = data[:,0],data[:,1]
x_d,y_d = log10(xdata),log10(ydata)
polycoef = polyfit(x_d, y_d, 1)
yfit = 10**( polycoef[0]*x_d+polycoef[1] )
plt.subplot(111)
plt.loglog(xdata,ydata,'.k',xdata,yfit,'-r')
plt.show()
Thanks a lot
You could use scipy.optimize.curve_fit instead of polyfit. It has a parameter sigma for errors of ydata. If you have your error for every y value in a sequence yerror (so that yerror has the same length as your y_d sequence) you can do:
polycoef, _ = scipy.optimize.curve_fit(lambda x, a, b: a*x+b, x_d, y_d, sigma=yerror)
For an alternative see the paragraph Fitting a power-law to data with errors in the Scipy Cookbook.