I am trying to solve a large-scale nonlinear system using the exact Newton method in SciPy. In my application, the Jacobian is easy to assemble (and factorize) as a sparse matrix.
It seems that all methods available in scipy.optimize.root approximate the Jacobian in one way or another, and I can't find a way to use Newton's method using the API that is discussed in SciPy's documentation.
Nonetheless, using the internal API, I have managed to use Newton's method with the following code:
from scipy.optimize.nonlin import nonlin_solve
x, info = nonlin_solve(f, x0, jac, line_search=False)
where f(x) is the residual and jac(x) is a callable that returns the Jacobian at x as a sparse matrix.
However, I am not sure whether this function is meant to be used outside SciPy and is subject to changes without notice.
Would this be recommended approach?
It is meant to be used.
Scipy's private functions that are not meant to be used from the outside start with a _.
This was confirmed by the scipy's team in an issue I raised recently: cf https://github.com/scipy/scipy/issues/17510
Related
Is there any optimization method/solver in mystic or scipy.optimization library that solves the problem in the matrix domain. In other words, is there any optimization method/solver that accepts matrix as an argument and minimizes its trace?
The trace of a matrix is:
tr(A) = sum(i, a[i,i])
So, depending on the rest of the model, (almost) any solver will allow you to use an objective like this. It is linear and is as easy and well-behaved as it gets. It is about the best objective you will ever see.
I've written a Python script to solve the Time Difference of Arrival (TDoA) angular reconstruction problem in 3-dimensions. To do so, I'm using SciPy's scipy.optimize.root root finding algorithm to solve a system of nonlinear equations. I find that the Levenberg-Marquardt method is the only supported method capable of reliably producing accurate results (most others simply fail).
I'd like to assess the uncertainty in the resulting solution. For most methods (including the default hybr method), SciPy returns the inverse Hessian of the objective function (i.e. the covariance matrix), from which one may begin to calculate the uncertainty(ies) in the found roots. Unfortunately this is not the case for the Levenberg-Marquardt method (which I'm admittedly much less familiar with on a mathematical method than the other methods... it just seems to work).
How (in general) can I estimate the uncertainties in the solution returned by scipy.optimize.root when using the lm method?
I am having trouble solving an optimisation problem in python, involving ~20,000 decision variables. The problem is non-linear and I wish to apply both bounds and constraints to the problem. In addition to this, the gradient with respect to each of the decision variables may be calculated.
The bounds are simply that each decision variable must lie in the interval [0, 1] and there is a monotonic constraint placed upon the variables, i.e each decision variable must be greater than the previous one.
I initially intended to use the L-BFGS-B method provided by the scipy.optimize package however I found out that, while it supports bounds, it does not support constraints.
I then tried using the SQLSP method which does support both constraints and bounds. However, because it requires more memory than L-BFGS-B and I have a large number of decision variables, I ran into memory errors fairly quickly.
The paper which this problem comes from used the fmincon solver in Matlab to optimise the function, which, to my knowledge, supports the application of both bounds and constraints in addition to being more memory efficient than the SQLSP method provided by scipy. I do not have access to Matlab however.
Does anyone know of an alternative I could use to solve this problem?
Any help would be much appreciated.
I'm trying to solve a first-order ODE in Python:
where Gamma and u are square matrices.
I don't explicitly know u(t) at all times, but I do know it at discrete timesteps from doing an earlier calculation.
Every example I found of Python's solvers online (e.g. this one for scipy.integrate.odeint and scipy.integrate.ode) know the expression for the derivative analytically as a function of time.
Is there a way to call these (or other differential equation solvers) without knowing an analytic expression for the derivative?
For now, I've written my own Runge-Kutta solver and jitted it with numba.
You can use any of the SciPy interpolation methods, such as interp1d, to create a callable function based on your discrete data, and pass it to odeint. Cubic spline interpolation,
f = interp1d(x, y, kind='cubic')
should be good enough.
Is there a way to call these (or other differential equation solvers) without knowing an analytic expression for the derivative?
Yes, none of the solvers you mentioned (nor most other solvers) require an analytic expression for the derivative. Instead they call a function you supply that has to evaluate the derivative for a given time and state. So, your code would roughly look something like:
def my_derivative(time,flat_Gamma):
Gamma = flat_Gamma.reshape(dim_1,dim_2)
u = get_u_from_time(time)
dGamma_dt = u.dot(Gamma)
return dGamma_dt.flatten()
from scipy.integrate import ode
my_integrator = ode(my_derivative)
…
The difficulty in your situation is rather that you have to ensure that get_u_from_time provides an appropriate result for every time with which it is called. Probably the most robust and easy solution is to use interpolation (see the other answer).
You can also try to match your integration steps to the data you have, but at least for scipy.integrate.odeint and scipy.integrate.ode this will be very tedious as all the integrators use internal steps that are inconvenient for this purpose. For example, the fifth-order Dormand–Prince method (DoPri5) uses internal steps of 1/5, 3/10, 4/5, 8/9, and 1. This means that if you have temporally equidistant data for u, you would need 90 data points for each integration step (as 1/90 is the greatest common divisor of the internal steps). The only integrator that could make this remotely feasible is the Bogacki–Shampine integrator (RK23) from cipy.integrate.solve_ivp with internal steps of 1/2, 3/4, and 1.
I'm trying to solve a set of nonlinear equations using the dog-leg trust-region algorithm in Matlab and Python.
In Matlab there is fsolve where this algorithm is the default, whereas for Python we specify 'dogleg' in scipy.optimize.minimize. I won't need to specify a Jacobian or Hessian for the Matlab whereas Python needs either one to solve the problem.
I don't have the Jacobian/Hessian so is there a way around this issue for Python? Or is there another function that performs the equivalent of Matlab's dog-leg method in fsolve?
In newer versions of scipy there is the approx_fprime function. It computes a numerical approximation of the jacobian of function f at position xk using the foward step finite difference. It returns an ndarray with the partial derivate of f at positions xk.
If you can't upgrade your version of scipy, you can always copy the implementation from scipy's source.
Edit:
scipy.optimize.minimize calls approx_fprime internally if the input jac=False. So in your case, it should be enough to do the following:
scipy.optimize.minimize(fun, x0, args, method='dogleg', jac=False)
Edit
scipy does not seem to handle the jac=False condition properly so it is necessary to build a callable jac using approx_fprime as follows
jac = lambda x,*args: scipy.optimize.approx_fprime(x,fun,epsilon,*args)
scipy.optimize.minimize(fun, x0, args, method='dogleg', jac=jac)