I am using scipy.optimize.basinhopping for finding the minima of a scalar function. I wonder whether it is possible to disable the local minimization part of scipy.optimize.basinhopping? As we can see from the output message below, minimization_failures and nit are nearly the same, indicating that the local minimization part may be useless for the global optimization process of basinhopping --- reason why I would like to disable the local minimization part, for the sake of efficiency.
You can avoid running the minimizer by using a custom minimizer that does nothing.
See the discussion on "Custom minimizers" in the documentation of minimize():
**Custom minimizers**
It may be useful to pass a custom minimization method, for example
when using a frontend to this method such as `scipy.optimize.basinhopping`
or a different library. You can simply pass a callable as the ``method``
parameter.
The callable is called as ``method(fun, x0, args, **kwargs, **options)``
where ``kwargs`` corresponds to any other parameters passed to `minimize`
(such as `callback`, `hess`, etc.), except the `options` dict, which has
its contents also passed as `method` parameters pair by pair. Also, if
`jac` has been passed as a bool type, `jac` and `fun` are mangled so that
`fun` returns just the function values and `jac` is converted to a function
returning the Jacobian. The method shall return an ``OptimizeResult``
object.
The provided `method` callable must be able to accept (and possibly ignore)
arbitrary parameters; the set of parameters accepted by `minimize` may
expand in future versions and then these parameters will be passed to
the method. You can find an example in the scipy.optimize tutorial.
Basically, you need to write a custom function that returns an OptimizeResult and pass it to basinhopping via the method part of minimizer_kwargs, for example
from scipy.optimize import OptimizeResult
def noop_min(fun, x0, args, **options):
return OptimizeResult(x=x0, fun=fun(x0), success=True, nfev=1)
...
sol = basinhopping(..., minimizer_kwargs=dict(method=noop_min))
Note: I don't know how skipping local minimization affects the convergence properties of the basinhopping algorithm.
You can use minimizer_kwargs to specify to minimize() what options your prefer to the local minimization step. See the dedicated part of the docs.
It is then up to what type of solver you ask minimize for. You can try setting a larger tol to make the local minimization step terminate earlier.
EDIT, in reply to the comment "What if I want to disable the local minimization part completely?"
The basinhopping algorithm from the docs works like:
The algorithm is iterative with each cycle composed of the following
features
random perturbation of the coordinates
local minimization accept or
reject the new coordinates based on the minimized function value
If the above is accurate there is no way to skip the local minimization step entirely, because its output is required by the algorithm to proceed further, i.e. keep or discard the new coordinate. However, I am not an expert of this algorithm.
Related
I found in the release notes from scipy to version 1.9.0 the following about the optimisation module, in the section "scipy.optimize improvements", 4th point:
Add a vectorized parameter to call a vectorized objective function only once per iteration.
However, I already checked out the documentation for such parameters (for the minisation function and minize_sclar) and couldn't find any hint for such a parameter. While searching in the internet I only found some posts concerning some suggestions or GitHub-issuses to implement such a thing (or workarounds for that).
Where is this parameter to find and can I use it?
Those notes have a more specific note for scipy.optimize.differential_evolution. That parameter is explained there. I've also come across it in other SO questions, but I don't recall which functions use it.
Basically for functions that allow it, you can write an objective function, or other callable (jacobian, boundary?), in a way that takes a 2d array of values. Normally the function just takes a 1d array, the current "state". But with "vectorized=True", the function should be prepared to accept a set of "state" array, and return a value for each.
So instead of calling the objective k times to get a range of value, such as when calculating a gradient, it can call it one, with a (n,k) argument, and get back all k results with one call.
I tried to explain how solve_ivp uses this at
scipy.integrate.solve_ivp vectorized
I'm relatively new to using SciPy; I'm currently using it to minimize a cost function for a multi-layer-perceptron model. I can't use scikit-learn because I need to have the ability to set the coefficients (they are read-only in the MLPClassifer) and add random permutations and noise to any and all parameters. I haven't finished the implementation quite yet, but I am confused about the parameters required for the minimize function.
For example, I have a function that I have written to calculate the "cost" (energy to minimize) of the function, and it calculates the gradient at the same time. That's nothing special as it's common practice. However, when calling scipy.optimize.minimize, it asks for two different functions: one that returns the scalar that is to be minimized (i.e., the cost in my case) and one that calculates the gradient of the current state. Example:
j,grad = myCostFunction(X,y)
Unless I am mistaken, it seems that it would need to call my function twice, with each call needing to be specified to return either the cost or the gradient, like so:
opt = scipy.optimize.minimize(fun=myJFunction, jac=myGradFunction, args = args,...)
Isn't this a waste of computation time? My data set will be > 1 million samples and 10ish features, so reducing redundant computation would be preferred since I will be training and retraining this thing tens of thousands of times for my project.
Another point of confusion is with the args input. Are the arguments passed like this:
# This is what I expect happens
myJFunction(x0,*args)
myGradFunction(x0,*args)
or like this:
# This is what I wish it did
myJFunction(x0,arg0,arg1,arg2)
myGradFunction(x0,arg3,arg4,arg5)
Thanks in advance!
After doing some experimentation and searching, I found the answers to my own questions.
While I can't say for sure about the scipy.optimize.minimize function, using other optimization functions (for example, scipy.optimize.fmin_tnc) explicitly states that the callable function func can either (1) return both the energy and the gradient, (2) return the energy and specify the gradient function for that parameter fprime (slower), or (3) return only the energy and have the function estimate the gradient through perturbation (much slower).
See the docs here: https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.fmin_tnc.html
I was very happy to see that I could use only one function to return both parameters. I assume it is the same case for the minimize function, but I have not tested it to be sure (See Edit 1)
As for my second question, if you specify two different functions, the *args parameters are passed to both functions the same; you cannot specify individual parameters for both.
EDIT 1: Reading through the minimize documentation more, if the parameter jac is set to True, then the optimizer assumes that the func returns energy and gradient. Reading the docs thoroughly is helpful, it seems.
From what I have seen, odeint seems to automatically decide what sort of algorithm it wants to use. However, the math that I am using is sufficiently finicky in converging that I want to have more control over which algorithm gets used. I've tried using ode (specifically, vode), but I'm having a hard time debugging it because of all the moving parts. I'm really not interested in taking apart all the details of the ode solver I'm using: I just want to be able to tell odeint to use one specific alogrithm and call it a day. Is this possible, or will I need to debug ode?
LSODA method used by odeint automatically switches between Adam's method (non-stiff) and BDF (stiff) and as far as I see it does not allow to change this behaviour.
If you are looking for a solver with a similar interface but where you can explicitly choose an integration method, have a look at scipy's solve_ivp.
It has an optional argument method that allows you to choose an integration method. Make sure not to choose 'LSODA' or you will get the same problem again.
How do i pass tolerances and other parameters through CVXPY when using the CPLEX solver?
from cvxpy import Problem, Minimize
from cvxpy.settings import CPLEX
costs = ...
constraints = ...
prob = Problem(Minimize(costs), constraints)
prob.solve(solver=CPLEX, ...)
I see a page of CPLEX Parameters though it is unclear which ones apply to my quadratic problem. Also, the CVXPY documentation has pass through options for other solvers but not CPLEX.
This will change in the future (see this pull request), but with cvxpy 1.0.6, you can do the following (NOTE: this is undocumented behavior; see below for more):
prob.solve(solver=CPLEX, advance=0)
The advance=0 will turn "off" the advanced start switch parameter. So, if the parameter name is parameters.advance in the CPLEX Python API, you would pass in the part after parameters. (i.e., advance) and the value as a keyword argument. Any extra keyword arguments that are passed to the solve method are interpreted this way. For debugging, you should probably set verbose=True (one of the standard keyword arguments to solve) to turn on the engine log; the parameter settings will be displayed at the top of the log.
This behavior was not documented for good reason. It doesn't allow you to set parameters like data consistency checking and modeling assistance. That parameter name in the CPLEX Python API is parameters.read.datacheck but read.datacheck cannot be used as a keyword argument in Python (it would result in a syntax error).
As a workaround, consider using the ILOG_CPLEX_PARAMETER_FILE environment variable, which is documented here.
EDIT: the workaround above is no longer necessary with cvxpy 1.0.8. That is, you should be able to set all of the parameters now regardless of where they are in the parameter hierarchy. You need to use the optional cplex_params argument, though. It's nice to combine this with verbose=True so that you can see the parameter settings in the engine log. For example:
prob.solve(solver=cvxpy.CPLEX,
verbose=True,
cplex_params={"mip.tolerances.absmipgap": 1e-07,
"benders.strategy": 3})
I'm having troubles to minimize a complex non linear function in python. This function is actually the chiSquare of a fitting model used to fit experimental data. In order to get the global minimum, I'm using the basinhopping function in scipy. This function is a wrapper of the minimize() function that adds some perturbation to look for different local minima. Right now my problem is that it has troubles to find the local minima.
There are a bunch of solvers that can be used in minimize(), and since I'm using bounds I chose between 'L-BFGS-B', 'SLSQP' and 'TNC'. None of them really find local minima. Is there a method based on the popular Levenberg-Marquardt algorithm that can be use to minimize? Maybe this does not make sense otherwise it would be already implemented, but I can't understand why.
My original idea was actually to use the leastsqbound function(https://pypi.python.org/pypi/leastsqbound) that I know is very good at providing accurate covariance matrix despide the bounds, and include it in a larger algorithm that would look for global minima (like the basinhopping function). Do you know if something like this already exist?
Thanks a lot for you advices!
Scipy has a Levenberg-Marquardt implementation: scipy.optimize.leastsq. It does not have the right return type to use with minimize (and therefore basin_hopping). However, it appears this could be remedied fairly straightforwardly.
Though I have not run it, this should do the trick:
def leastsq_for_minimize( *args, **kwargs ):
results = leastsq( *args, **kwargs )
optimize_results = scipy.optimize.OptimizeResult()
# Some code here to correctly copy results to optimize results
return optimize_results
scipy.optimize.basinhopping(
# your arguments here
minimizer_kwargs=dict(method=leastsq_for_minimize),
)
minimize documentation
basinhopping documentation:
OptimizeResult documentation