My question is about fitting parameters of a complicated model composed of different parametric functions.
More precisely, I want to describe a complicated experiment.
The experiment produces a one-dimensional array of measured data data, where each its entries corresponds to (a set of) experimental control variables x.
I now a theoretical model (actually multiple models, see below) model(x,pars), which takes x and a lot of parameters pars to give a prediction for data. However, not all parameters are known and I need to fit them.
Moreover, some details of the model are not yet certain. Because of that, I actually have a family of multiple models which are in some parts very similar, but where some internal component of the model is different (but a large part of the model is the same).
Unfortunately, switching one component for another might introduce new (unknown) parameters, that is we now have modelA(x,parsA) and modelB(x,parsB)
which have different parameters.
Basically, the model is composed of functions f(x, pars, vals_of_subfuncs) where x is the independent variable, pars are some explicit parameters of f, and vals_of_subfuncs are the results of evaluating some lower-level functions, which themselves depend on their own parameters (and maybe the results of their own lower-level functions etc.)
Obviously, there are no recursions possible, and at there is a lowest level of functions that do not rely on the value of other functions.
The situation is best illustrated in this picture:
Modular model architecture
The independent variable is x (blue), parameters are a,b,c,d (red), and the values of subfunctions appear as green arrows into nodes that represent functions.
In (1), we have a lowest-level function G(x; (a,b); {}) with no sub-functions and a higher-level function F(x; c; G(x; (a,b)) whose evaluation gives the model result, which depends on x and pars=(a,b,c).
In (2) and (3) we change a component of the model, (F->F') and (G->G'), respectively. This changes the parameter dependence of the final model.
Now I am looking for a most pythonic/modular way to approach the problem of implementing parameter fitting in this situation, without having to re-write the fit function everytime I swap/change a component of my model, thereby possibly introducing a new parameter.
At the moment, I am trying to find solutions to this problem using lmfit. I also thought about maybe trying to use sympy to work with symbolic "parameters", but I don't think all the functions that appear can be easily written as expressions that can be evaluated by asteval.
Does anyone know of a natural way to approach such a situation?
I think this question would definitely be improved with a more concrete example, (that is, with actual code). If I understand correctly, you have a general model
def model_func(x, a, b, c, d):
gresult = G(x, a, b, d)
return F(x, b, c, gresult)
but you also want to control whether d and b are really variables, and whether c gets passed to F. Is that correct?
If that is correct (or at least captures the spirit), then I think you can do this with lmfit (disclaimer: I'm a lead author) with a combination of adding keyword arguments to the model function and setting some Parameter values as fixed.
For example, you might do some rearranging like this:
def G(x, a, b=None, d=None):
if b is not None and d is None:
return calc_g_without_d(x, a, b)
return calc_g_with_d(x, a, d)
def F(x, gresult, b, c=None):
if c is None:
return calc_f_without_c(x, gresult, b)
return calc_f_with_c(x, gresult, b, c)
def model_func(x, a, b, c, d, g_with_d=True, f_with_c=True):
if g_with_d:
gresult = G(x, a, d)
else:
gresult = G(x, a, b)
if f_with_c:
return F(x, gresult, b, c=c)
else:
return F(x, gresult, b)
Now, when you make your model you can override the default values f_with_c and/or g_with_d:
import lmfit
mymodel = lmfit.Model(model_func, f_with_c=False)
params = mymodel.make_params(a=100, b=0.2201, c=2.110, d=0)
and then evaluate the model with mymodel.eval() or run a fit with mymodel.fit() and passing in explicit values for the keyword arguments f_with_c and/or g_with_d, like
test = mymodel.eval(params, x=np.linspace(-1, 1, 41),
f_with_c=False, g_with_d=False)
or
result = mymodel.fit(ydata, params, x=xdata, g_with_d=False)
I think the way you have it specified, you'd want to make sure that d was not a variable in the fit when g_with_d=False, and there are cases where you would want b to not vary in the fit. You can do that with
params['b'].vary = False
params['d'].vary = False
as needed. I can imagine your actual problem is slightly more involved than that, but I hope that helps get you started in the right direction.
Thanks for the answers.
I think lmfit might be able to do what I want, but I will have to implement the "modularity" myself.
The example I have was just conceptional and a miminal model. In general, the "networks" of function and their dependencies are much more intricate that what I do in the example.
My current plan is as follows:
I will write a class Network for the "network", which contains certain Nodes.
Notes specify their possible "symbolic" dependence on subNodes, explicit parameters and independent variables.
The Network class will have routines to check that the such constructed network is consisten. Moreover, it will have a (lmfit) Parameters object (i.e. the unification of all the parameters that the nodes explicitly depend on) and provide some method to generate an lmfit Model from that.
Then I will use lmfit for the fitting.
At least this is the plan.
If I succeed in building this, I will publish update this post with my code.
Since you brought up sympy, I think you should take a look at symfit, which does precisely what you ask for in the last paragraph. With symfit you can write symbolic expressions, which are then fitted with scipy. It will make it very easy for you to combine your different submodels willy-nilly.
Let me implement your second example using symfit:
from symfit import variables, parameters, Fit, Model
a, b, c = parameters('a, b, c')
x, G, F = variables('x, G, F')
model_dict = {
G: a * x + b,
F: b * G + c * x
}
model = Model(model_dict)
print(model.connectivity_mapping)
I choose these rather trivial functions, but you can obviously choose whatever you want. To see that this model matches your illustration, this is what connectivity_mapping prints:
{F: {b, G, x, c}, G: {b, a, x}}
So you see that this is really a mapping representing what you drew. (The arguments are in no particular order within each set, but they will be evaluated in the right order, e.g. G before F.) To then fit to your data, simply do
fit = Fit(model, x=xdata, F=Fdata)
fit_results = fit.execute()
And that's it! I hope this makes it clearer why I think symfit does fit your use case. I'm sorry I couldn't clarify that earlier, I was still finalizing this feature into the API so up to now it only existed in the development branch. But I made a release with this and many other features just now :).
Disclaimer: I'm the author of symfit.
Related
I have a problem, I have 2 lists with x and y values, and I would like to create a function based on these. But the problem is that I would like to build a function like this one:
f(x) = a * (x-b)**c
I already know scipy.interpolate but I couldn't find anything to return a function like this one.
is there a quite easy way to try to create the best function I can by searching which values of a,b and c match the most?
thanks for your help!
Edit:
here is what my current values of x and y look like:
I created this function :
def problem(values):
s = sum((y - values[0]*(x-values[1])**values[2])**2 for x,y in zip(X,Y))
return(s)
and I tried to find the best values of a,b and c with scipy.optimize.minimize but I don't know with which values of a,b and c I should start...
values = minimize(problem,(a,b,c))
(Edited to account for the OP's added code and sub-question.)
The general idea is to use a least-squares minimization to find the "best" values of a, b, and c. First define a function whose parameters are a, b, c that returns the sum of the squares of the differences between the given y values and the calculated values of a * (x-b)**c. (That function can be done as a one-liner.) Then use an optimization routine, such as one found in scipy, to minimize the value of that function value. Those values of a, b, c are what you want--use them to define your desired function.
There are a few details to examine, such as restrictions on the allowed values of a, b, c, but those depend somewhat on your lists of x and y values.
Now that you have shown a graph of your x and y values, I see that your values are all positive and the function is generally increasing. For that common situation I would use the initial values
a = 1.0
b = 0.0
c = 1.0
That gives a straight line through the origin, in fact the line y = x, which is often a decent first guess. In your case the x and y values have a very different scale, with y about a hundred times larger than x, so you would probably get better results with changing the value of a:
a = 100.0
b = 0.0
c = 1.0
I can see even better values and some restrictions on the end values but I would prefer to keep this answer more general and useful for other similar problems.
Your function problem() looks correct to me, though I would have written it a little differently for better clarity. Be sure to test it.
def problem (a , b, c, d):
return a * (x[d]-b)**c
I guess is what you are after. With D being what value of the X array. Not sure where Y comes into it.
when I read the documents about Creating a new Op, I can't understand the grad() in the examples http://deeplearning.net/software/theano/extending/extending_theano.html#example-op-definition. Why do they return output_grads[0] * 2 not 2? and what's output_grads[0] represent for?
If output_grads[0] represent a chain derivative with respect to the input x, in the next example http://deeplearning.net/software/theano/extending/extending_theano.html#example-props-definition, why the grad() return a * output_grads[0] + b (it should be self.a * output_grads[0] + self.b) not self.a * output_grads[0]?
How about a more complicated custom Op? Like y = exp(x1)/(a*(x1**3)+log(x2)), how to write its grad()? And furthermore, if the inputs are vectors or matrix, how to write the grad()?
As the extended .grad() documentation points out, the output_grads argument is
(where f is one of your Op's outputs and C is the cost on which you called theano.tensor.grad(...))
The page also says that the .grad(...) method of an Op must return
(where x is an input to your Op)
I think the ax+b example is just wrong. If you look at the actual code, for example, the Sigmoid or the XLogX,
it seems to just implement the chain rule.
Disclaimer: I haven't implemented a custom Op so far and was looking into this myself and this is how I understood it.
I am writing a script to calculate the definite integral of an equation. I'm writing a helper function that would take in coefficients as parameters and return a function of x.
def eqn(x, k, c, a):
return ((k*x + c**(1-a))
Next, I define a function that calculates the definite integral, using quad imported from scipy:
from scipy.integrate import quad
def integral(eqn, c_i, y_i):
integral_i, integral_err = quad(eqn, c_i, y_i)
print integral_i
Then I call the function by passing in parameters
k = calc_k(7511675,1282474,0,38,2)
eqn = carbon_path_eqn(x, k, 7511675, 2)
carbon_path_def_int(eqn,0,38)
However, I get an error saying that 'name x is not defined'. I understand that x isn't defined globally, but I'm wondering how I can write a helper function, that takes in parameters, and still returns a function of x that can be used in quad?
Thank you!
PS -
#bpachev, this is a follow up from the other post
The mistake here is that the function 'eqn' does not return a function, it returns the value of a function at some point x, given parameters k,c,a.
quad should be passed a function (in your case, eqn) where the first argument (in your case, x) is assumed to be the variable over which the function is integrated. You also need to pass quad a tuple of the remaining parameters (in your case (k,c,a)) and two limits (in your case, c_i,y_i). In other words, call quad like this:
quad(eqn,c_i,y_i,args=(k,c,a))
This is all explained in the scipy documentation http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.quad.html.
This is not what you asked. However, as someone else has mentioned, sympy would probably make your life much easier. For instance, suppose you need to be able to evaluate integrals of functions of x such as f in the code below, where a and b are arbitrary constants. Here's how you can use sympy to do that.
Define the function, integrate it with respect to x and save the result, then evaluate the result for values of a, b and x.
I want to integrate a function using python, where the output is a new function rather than a numerical value. For example, I have the equation (from Arnett 1982 -- analytical description of a supernova):
def A(z,tm,tni):
y=tm/(2*tni)
tm=8.8 # diffusion parameter
tni=8.77 # efolding time of Ni56
return 2*z*np.exp((-2*z*y)+(z**2))
I want to then find the integral of A, and then plot the results. First, I naively tried scipy.quad:
def Arnett(t,z,tm,tni,tco,Mni,Eni,Eco):
x=t/tm
Eni=3.90e+10 # Heating from Ni56 decay
Eco=6.78e+09 # Heating from Co56 decay
tni=8.77 # efolding time of Ni56
tco=111.3 # efolding time of Co56
tm=8.8 # diffusion parameter
f=integrate.quad(A(z,tm,tni),0,x) #integral of A
h=integrate.quad(B(z,tm,tni,tco),0,x) #integral of B
g=np.exp((-(x/tm)**2))
return Mni*g*((Eni-Eco)*f+Eco*h)
Where B is also a pre-defined function (not presented here). Both A and B are functions of z, however the final equation is a function of time, t. (I believe that it is herein I am causing my code to fail.)
The integrals of A and B run from zero to x, where x is a function of time t. Attempting to run the code as it stands gives me an error: "ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()".
So after a short search I thought that maybe sympy would be the way to go. However I am failing with this as well.
I wonder if anyone has a helpful suggestion how to complete this task please?
Many thanks,
Zach
You can integrate A analytically. Assuming I'm not missing something silly due to being up way too late, does the following help?
import sympy as sy
sys.displayhook = sy.pprint
A, y, z, tm, t, tni = sy.symbols('A, y, z, tm, t, tni')
A = 2*z*sy.exp(-2*z*y + z**2)
expr = sy.integrate(A, (z,0,t)) # patience - this takes a while
expr
# check:
(sy.diff(expr,t).simplify() - A.replace(z,t)).simplify()
# thus, the result:
expr.replace(y,tm/(2*tni)).replace(t,t/tm)
The last line yields the integral of your A function in analytic form, though it does require evaluating the imaginary error function (which you can do with scipy.special.erfi()).
I think what you are looking for are lambda expression (if i understood correctly what you said.. see here for extra information and some examples on lambda functions).
What they allow you to do is define an anonymous function in A and return it so that you get your B function, should work something like this:
def A(parameters):
return lambda x: x * parameters # for simplicity i applied a multiplication
# but you can apply anything you want to x
B = A(args)
x = B(2)
Hope I could provide you with a decent response!
I think the error you get comes from an incorrect call to scipy.integrate.quad:
The first argument needs to be just the function name, integration is then performed over the first variable of this function. The values of the other variables can be passed to the function via the args keyword.
The output of scipy.integrate.quad contains not only the value of the integral, but also an error estimate. So a tuple of 2 values is returned!
In the end the following function should work:
def Arnett(t, z, Mni, tm=8.8, tni=8.77, tco=111.3, Eni=3.90e+10,
Eco=6.78e+09):
x=t/tm
f,err=integrate.quad(A,0,x,args=(tm,tni)) #integral of A
h,err=integrate.quad(B,0,x,args=(tm,tni,tco)) #integral of B
g=np.exp((-(x/tm)**2))
return Mni*g*((Eni-Eco)*f+Eco*h)
But an even better solution would probably be integrating A and B analytically and then evaluating the expression as murison suggested.
I have a number of methods that are independent of each other but are needed collectively to compute an output. Thus, when a variable in any of the methods changes all the methods are called in the computation which is slow and expensive. Here is a quick pesudo-code of what I have:
# o represents an origin variable
# valueA represents a variable which can change
def a (o, valueA):
# calculations
return resultA
def b (o, valueB):
# calculations
return resultA
def c (o, valueC1, valueC2):
# calculations
return resultA
def compute (A, B, C1, C2):
one = self.a(o, A)
two = self.b(one,B)
three = self.c(two, C1, C2)
return img
For example when the value of C1 changes, when calling compute all the methods are calculated despite a & b having no change. What I would like is some way of checking which of the values of A,B,C1,C2 have changed between each call to compute.
I have considered defining a list of the values then on the next call comparing it to the new values being pass to compute. Eg; 1st call: list=[1,2,3,4] on 2nd call list=[1,3,4,5] so b & c need calculating but a is the same. However, I am unsure as to how to go from the comparison to defining which method to call?
Some background on my particular application in case it is of use. I have a wxPython window with sliders that determine values for image processing and an image is drawn on each change of these sliders.
What is the best way to compare each call to compute and remove these wasted repeated computations?
If i have to solve this, I would use a Dictionary, where the key is the valueX (or a list of it if have more than one, in your example C) and the value should be the result of the function.
So, you should have something like that:
{ valueA: resultA, valueB: resultB, [valueC1, valueC2]: resultC }
To do that, in the functions you will have to add it:
def a(o, valueA):
[calcs]
dic[valueA] = resultA
return resultA
[...]
def c(o, valueC1, valueC2)
[calcs]
dic[[valueC1, valueC2]] = resultC
return resultC
And, in the function that computes, you can try to get the value for the parameters and if not get the value, calculate it
def compute (A, B, C1, C2):
one = dic.get(A) if dic.get(A) else self.a(o, A)
two = dic.get(B) if dic.get(B) else self.b(one,B)
three = dic.get([C1,C2]) if dic.get([C1,C1]) else self.c(two, C1, C2)
return img
P.D: this is the "crude" implementation of memoize functions that #holdenweb pointed in his comment.
You could consider making the methods memoizing functions that use a dict to look up the results of previously stored computations (probably best in the class namespace to allow memoizing to optimize across all instances).
The memory requirements could be quite severe, however, if the methods are called with many arguments, in which case you might want to adopt a "publish and subscribe" pattern to try and make your computation more "systolic" (driven by changes in the data, loosely).
That' a couple of approaches. I'm sure SO will think of more.