We are using scipy.optimize.differential_evolution through lmfit for an optimization problem with major parameter magnitudes differences.
We know that leastsq does handle this type of problems but we are wondering if differential_evolution does as well or if we should normalize them in advance?
Best regards
I think the answer to both of your 2 options is No.
First No: scipy (and so lmfit) differential_evolution() does not automatically rescale parameter values to have similar orders of magnitude.
Second No: you should not have to rescale the values.
Basically, for some methods (say, leastsq) it is important to rescale because the algorithm constructs and uses derivative matrices that include all of the variables. If there are widely varying scales, the manipulation of the matrices can be numerically unstable.
Differential Evolution does not use such derivatives (even where they would be helpful, hence its incredible slowness). So, it will not have that problem.
That said, just from a stability-of-numerical-calculations perspective, I would recommend against some variables that are on the scale of 1.e-12 with other on the scale 1e12. If you know the scale is outside 1e-6 to 1e6, rescale.
Related
I'm trying to fit my data and have so far used sp.optimize.leastsq. I changed to sp.optimize.least_squares to add bounds to the parameters, but both when I use bounds and when I don't the search doesn't converge, even in data sets sp.optimize.leastsq fitted just fine.
Shouldn't these functions work the same?
What could be the difference between them that makes the newer one not to find solutions the older one did?
leastsq
is a wrapper around MINPACK’s lmdif and lmder algorithms.
least_squares implements other methods in addition to the MINPACK algorithm.
method{‘trf’, ‘dogbox’, ‘lm’}, optional
Algorithm to perform minimization.
‘trf’ : Trust Region Reflective algorithm, particularly suitable for large sparse problems with bounds. Generally robust method.
‘dogbox’ : dogleg algorithm with rectangular trust regions, typical use case is small problems with bounds. Not recommended for problems with rank-deficient Jacobian.
‘lm’ : Levenberg-Marquardt algorithm as implemented in MINPACK. Doesn’t handle bounds and sparse Jacobians. Usually the most efficient method for small unconstrained problems.
Default is ‘trf’. See Notes for more information.
It is possible for some problems that lm method does not converge while trf converges.
Consider for the sake of simplicity the following equation (Burgers equation):
Let's solve it using scipy (in my case scipy.integrate.ode.set_integrator("zvode", ..).integrate(T)) with a variable time-step solver.
The issue is the following: if we use the naïve implementation in Fourier space
then the viscosity term nu * d2x(u[t]) can cause an overshoot if the time step is too big. This can lead to a fair amount of noise in the solutions, or even to (fake) diverging solutions (even with stiff solvers, on slightly more complex version of this equation).
One way to regularize this is to evaluate the viscosity term at step t+dt, and the update step becomes
This solution works well when programmed explicitly. How can I use scipy's variable-step ode solver to implement it ? To my surprise I haven't found any documentation on this fairly elementary thorny issue...
You actually can't, or on the other extreme, odeint or ode->zvode already does that to any given problem.
To the first, you would need to give the two parts of the equation separately. Obviously, that is not part of the solver interface. Look at DDE and SDE solvers where such a partition of the equation is actually required.
To the second, odeint and ode->zvode use implicit multi-step methods, which means that the values of u(t+dt) and the right side there enter the computation and the underlying local approximation.
You could still try to hack your original approach into the solver by providing a Jacobian function that only contains the second derivative term, but quite probably you will not achieve an improvement.
You could operator-partition the ODE and solve the linear part separately introducing
vhat(k,t) = exp(nu*k^2*t)*uhat(k,t)
so that
d/dt vhat(k,t) = -i*k*exp(nu*k^2*t)*conv(uhat(.,t),uhat(.,t))(k)
I want to find the minimum of a function in python y = f(x)
Problem : the solver tries to compute the gradient with super close x values (delta x around 1e-8), and my function f is not sensitive to such a small step (ie we can see y vary when delta x around 1e-1).
Hence gradient is 0 to the solver, and can not find the proper solution.
I've tried following solvers from scipy, I can't find the option I'm looking for..
scipy.optimize.minimize
scipy.optimize.fmin
In Matlab fmincon , there is an option that does the job 'DiffMinChange' : Minimum change in variables for finite-difference gradients (a positive scalar).
You may want to try and use L-BFGS-B from scipy:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html
And provide the “epsilon” parameter to be around 0.1/0.05 and see if it makes it better. I am of course assuming that you will let the solver compute the gradient for you by numerical differentiation (I.e., you pass fprime=None and approx_grad=True) to the routine.
I personally despise the “minimize” interface to various solvers so I prefer to deal with the actual solvers themselves.
my problem at hand: I am using scipy curve_fit to fit a curve (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html) but in many occasions, the parameters estimated for such curve refer to one of the local many "local" minima and not "global" minimum. Now this is to be expected given how curve_fit was designed. Still, I really need my global minimum.
In order to find it, my initial hunch would be to multiply initial starting points, run multiple curve_fit instances and choose the one with the lowest fit error but I would suffer from a number of biases in my personal initial guess estimates (also potentially the number of combinations could be considerable and this would be detrimental to performance).
Do you happen to know better, faster and/or methodologically sounder methods on how to proceed? (they do not need to pass for least squares, I can build ad hoc stuff if necessary)
There are a couple possible approaches. One would be to do a "brute force" search through your parameter space to find candidate starting points for the local solver in curve_fit. Another would be to use a global solver such as differential evolution. For sure, both of these can be much slower than a single curve_fit, but they do aim at finding "global minima". Within scipy.optimize, these methods are brute and differential_evolution, respectively. It should be noted neither of these is actually global optimizers, as they both require upper and lower bounds for the search space of all parameters. Still, within those boundaries, they do attempt to find the best result, not just a local minimum close to your starting values.
A common approach is to use brute with medium-sized steps for each parameter, then take the best ten of those and use Levenberg-Marquardt (from leastsq, as used in curve_fit) starting from each of these. Similarly, you can use differential_evolution and then refine.
You might find lmfit (https://lmfit.github.io/lmfit-py) helpful, as it allows you to set up the model once and switch between solvers, including
brute, differential_evolution, and leastsq. Lmfit also makes it easy to fix some parameters or place upper/lower bounds on some parameters. It also provides a higher-level interface to model building and curve-fitting, and methods to explore the confidence intervals of parameters.
I'm trying to perform a constrained least-squares estimation using Scipy such that all of the coefficients are in the range (0,1) and sum to 1 (this functionality is implemented in Matlab's LSQLIN function).
Does anybody have tips for setting up this calculation using Python/Scipy. I believe I should be using scipy.optimize.fmin_slsqp(), but am not entirely sure what parameters I should be passing to it.[1]
Many thanks for the help,
Nick
[1] The one example in the documentation for fmin_slsqp is a bit difficult for me to parse without the referenced text -- and I'm new to using Scipy.
scipy-optimize-leastsq-with-bound-constraints on SO givesleastsq_bounds, which is
leastsq
with bound constraints such as 0 <= x_i <= 1.
The constraint that they sum to 1 can be added in the same way.
(I've found leastsq_bounds / MINPACK to be good on synthetic test functions in 5d, 10d, 20d;
how many variables do you have ?)
Have a look at this tutorial, it seems pretty clear.
Since MATLAB's lsqlin is a bounded linear least squares solver, you would want to check out scipy.optimize.lsq_linear.
Non-negative least squares optimization using scipy.optimize.nnls is a robust way of doing it. Note that, if the coefficients are constrained to be positive and sum to unity, they are automatically limited to interval [0,1], that is one need not additionally constrain them from above.
scipy.optimize.nnls automatically makes variables positive using Lawson and Hanson algorithm, whereas the sum constraint can be taken care of as discussed in this thread and this one.
Scipy nnls uses an old fortran backend, which is apparently widely used in equivalent implementations of nnls by other software.