Develop Reference

How do I use fsolve to solve a quadratic equation?

How do I use fsolve to calculate the value of y for the following non-linear equation in Python
y=x^3 -√y
(when x = 0, 1, 2.3611, 2.9033, 3.2859, 3.5915)
I have tried by solving the problem on paper and then using a function to calculate the value of y. But I am unable to use fsolve to do the same for me.
def func(x):return np.round(((-1+np.sqrt(1+(4*x**3)))/2)**2,4)

One should avoid the root functions if applying a numerical algorithm, at zero arguments the root function is non-smooth. Thus use y=z², hoping that the function z^2+z is convex enough that starting at 1.0 the root-finding iteration stays with positive values for z, and take the square root of the result,
y = [ fsolve(lambda z: z**2+z-x**3, 1.0)[0]**2 for x in [0, 1, 2.3611, 2.9033, 3.2859, 3.5915] ]
This gets the solutions
[0.0, 0.3819660112501052, 10.000316539128024, 20.000195919522547, 30.00100142437062, 40.00161656606038]

You can try like this:
import math
from scipy.optimize import fsolve
y = [0.000, 0.3820, 10.00, 20.00, 30.00, 40.00]
def func(x):
for i in y:
return x**3-(math.sqrt(i))
x0 = fsolve(func, [0, 1, 2.3611, 2.9033, 3.2859, 3.5915])
OutPut:
[0.00000000e+000 2.24279573e-109 5.29546580e-109 6.51151078e-109 7.36960349e-109 8.05500243e-109]
It is scientific notation. If 1e-5 It means 1 × 10−5. In other words, 0.00001.
Convert scientific notation to decimals:
Now, x0 = [0.00000000e+000 2.24279573e-109 5.29546580e-109 6.51151078e-109 7.36960349e-109 8.05500243e-109]
for i in x0:
data = float("{:.8f}".format(float(str(i))))
print(data)

Improve performance of autograd jacobian

I'm wondering how the following code could be faster. At the moment, it seems unreasonably slow, and I suspect I may be using the autograd API wrong. The output I expect is each element of timeline evaluated at the jacobian of f, which I do get, but it takes a long time:
import numpy as np
from autograd import jacobian
def f(params):
mu_, log_sigma_ = params
Z = timeline * mu_ / log_sigma_
return Z
timeline = np.linspace(1, 100, 40000)
gradient_at_mle = jacobian(f)(np.array([1.0, 1.0]))
I would expect the following:
jacobian(f) returns an function that represents the gradient vector w.r.t. the parameters.
jacobian(f)(np.array([1.0, 1.0])) is the Jacobian evaluated at the point (1, 1). To me, this should be like a vectorized numpy function, so it should execute very fast, even for 40k length arrays. However, this is not what is happening.
Even something like the following has the same poor performance:
import numpy as np
from autograd import jacobian
def f(params, t):
mu_, log_sigma_ = params
Z = t * mu_ / log_sigma_
return Z
timeline = np.linspace(1, 100, 40000)
gradient_at_mle = jacobian(f)(np.array([1.0, 1.0]), timeline)

From https://github.com/HIPS/autograd/issues/439 I gathered that there is an undocumented function autograd.make_jvp which calculates the jacobian with a fast forward mode.
The link states:
Given a function f, vectors x and v in the domain of f, make_jvp(f)(x)(v) computes both f(x) and the Jacobian of f evaluated at x, right multiplied by the vector v.
To get the full Jacobian of f you just need to write a loop to evaluate make_jvp(f)(x)(v) for each v in the standard basis of f's domain. Our reverse mode Jacobian operator works in the same way.
From your example:
import autograd.numpy as np
from autograd import make_jvp
def f(params):
mu_, log_sigma_ = params
Z = timeline * mu_ / log_sigma_
return Z
timeline = np.linspace(1, 100, 40000)
gradient_at_mle = make_jvp(f)(np.array([1.0, 1.0]))
# loop through each basis
# [1, 0] evaluates (f(0), first column of jacobian)
# [0, 1] evaluates (f(0), second column of jacobian)
for basis in (np.array([1, 0]), np.array([0, 1])):
val_of_f, col_of_jacobian = gradient_at_mle(basis)
print(col_of_jacobian)
Output:
[ 1. 1.00247506 1.00495012 ... 99.99504988 99.99752494
100. ]
[ -1. -1.00247506 -1.00495012 ... -99.99504988 -99.99752494
-100. ]
This runs in ~ 0.005 seconds on google collab.
Edit:
Functions like cdf aren't defined for the regular jvp yet but you can use another undocumented function make_jvp_reversemode where it is defined. Usage is similar except that the output is only the column and not the value of the function:
import autograd.numpy as np
from autograd.scipy.stats.norm import cdf
from autograd.differential_operators import make_jvp_reversemode
def f(params):
mu_, log_sigma_ = params
Z = timeline * cdf(mu_ / log_sigma_)
return Z
timeline = np.linspace(1, 100, 40000)
gradient_at_mle = make_jvp_reversemode(f)(np.array([1.0, 1.0]))
# loop through each basis
# [1, 0] evaluates first column of jacobian
# [0, 1] evaluates second column of jacobian
for basis in (np.array([1, 0]), np.array([0, 1])):
col_of_jacobian = gradient_at_mle(basis)
print(col_of_jacobian)
Output:
[0.05399097 0.0541246 0.05425823 ... 5.39882939 5.39896302 5.39909665]
[-0.05399097 -0.0541246 -0.05425823 ... -5.39882939 -5.39896302 -5.39909665]
Note that make_jvp_reversemode will be slightly faster than make_jvp by a constant factor due to it's use of caching.

Scipy: Minimize violates given bounds

I am trying to use scipy.optimize.minimize with simple a <= x <= b bounds. However, it often happens that my target function is evaluated just outside the bounds. To my understanding, this happens when minimize tries to determine the gradient of the target function at the boundary.
Minimal example:
import math
import numpy as np
from scipy.optimize import Bounds, minimize
constraint = Bounds([-1, -1], [1, 1], True)
def fun(x):
print(x)
return -math.exp(-np.dot(x,x))
result = minimize(fun, [-1, -1], bounds=constraint)
The output shows that the minimizer jumps to the point [1,1] and then tries to evaluate at [1.00000001, 1]:
[-1. -1.]
[-0.99999999 -1. ]
[-1. -0.99999999]
[-0.72932943 -0.72932943]
[-0.72932942 -0.72932943]
[-0.72932943 -0.72932942]
[-0.22590689 -0.22590689]
[-0.22590688 -0.22590689]
[-0.22590689 -0.22590688]
[1. 1.]
[1.00000001 1. ]
[1. 1.00000001]
[-0.03437328 -0.03437328]
...
Of course, there is no problem in this example, as fun can be evaluated also there. But that might not always be the case...
In my actual problem, the minimum can not be on the boundary and I have the easy workaround of adding an epsilon to the bounds.
But one would expect that there should be an easy solution to this issue which also works if the minimum can be at a boundary?
PS: It would be strange if I were the first to have this problem -- sorry if this question has been asked before somewhere, but I didn't find it anywhere.

As discussed here (thanks #"Welcome to Stack Overflow" for the comment directing me there), the problem is indeed that the gradient routine doesn't respect the bounds.
I wrote a new one that does the job:
import math
import numpy as np
from scipy.optimize import minimize
def gradient_respecting_bounds(bounds, fun, eps=1e-8):
"""bounds: list of tuples (lower, upper)"""
def gradient(x):
fx = fun(x)
grad = np.zeros(len(x))
for k in range(len(x)):
d = np.zeros(len(x))
d[k] = eps if x[k] + eps <= bounds[k][1] else -eps
grad[k] = (fun(x + d) - fx) / d[k]
return grad
return gradient
bounds = ((-1, 1), (-1, 1))
def fun(x):
print(x)
return -math.exp(-np.dot(x,x))
result = minimize(fun, [-1, -1], bounds=bounds,
jac=gradient_respecting_bounds(bounds, fun))
Note that this can be a bit less efficient, because fun(x) now gets evaluated twice at each point.
This seems to be unavoidable, relevant snippet from _minimize_lbfgsb in lbfgsb.py:
if jac is None:
def func_and_grad(x):
f = fun(x, *args)
g = _approx_fprime_helper(x, fun, epsilon, args=args, f0=f)
return f, g
else:
def func_and_grad(x):
f = fun(x, *args)
g = jac(x, *args)
return f, g
As you can see, the value of f can only be reused by the internal _approx_fprime_helper function.

nonlinear optimization with vectors, scalars and inequality constraints

I have set of equation in form: Y=aA+bB
where Y-is know vector of floats (only this one is known!); a, b are unkown scalar (float) and A, B are unknown vectors of floats. Each equation have it own Y, a, b, whereas all equation share the same unknow vectors A and B.
I have set of such equation so my problem is to minimize function:
(Y-aA-bB)+(Y'-a'A-b'B)+....
I have also many inequality constrains of type: Ai>Aj (Ai i-th element of vector A), Bi>= Bk, Bi>0, a>a', ...
Is there any software or library (ideally for python) which can handle this problem?

General remarks
This is a linear problem (at least in the linear least-squares sense, continue reading)!
It's also incompletely specified as it's not clear if there should be always a feasible solution in your case or if you want to minimize some given loss in general. Your text sounds like the latter, but in this case one has to chose the loss (which makes a difference in regards to possible algorithms). Let's take the euclidean-norm (probably the best pick here)!
Ignoring constraints for a moment, we can view this problem as basic least-squares solution to a linear matrix equation problem (euclidean-norm vs. squared euclidean-norm does not make a difference!).
min || b - Ax ||^2
Here:
M = number of Y's
N = size of Y
b = (Y0,
Y1,
...) -> shape: M*N (flattened: Y_x = (y_x_0, y_x_1).T)
A = ((a0, 0, 0, ..., b0, 0, 0, ...),
(0, a0, 0, ..., 0, b0, 0, ...),
(0, 0, a0, ..., 0, 0, b0, ...),
...
(a1, 0, 0, ..., b1, 0, 0, ...)) -> shape: (M*N, N*2)
x = (A0, A1, A2, ... B0, B1, B2, ...) -> shape: N*2 (one for A, one for B)
What you should do
If unconstrained:
Convert to standard-form and use numpy's lstsq
If constrained:
Either use customized optimization algorithms, or:
Linear-programming (if minimizing absolute-differences / l1-norm)
I'm too lazy to formulate it for scipy's linprog
Not that hard, but l1-norm is non-trivial using scipy's API
Much easier to formulate with cvxpy (obj=cvxpy.norm(X, 1))
Quadratic-programming / Second-order-cone-programming (if minimizing euclidean norm / l2-norm)
Again, too lazy to formuate it; no special solver available at scipy yet
Could be easily formulated with cvxpy (obj=cvxpy.norm(X, 2))
Emergency: use general-purpose constrained nonlinear-optimization algorithms like SLSQP -> see code
Some hacky code (not the best approach!)
This code:
Is just a demo!
Uses general nonlinear optimization algorithms from scipy
Therefore:
easier to formulate
Less fast & robust than LP, QP, SOCP
But will achieve approximately the same result as convergence on convex optimization problems is guaranteed
Uses automatic-differentiation whenever needed
(author too lazy to add gradients)
this can really hurt if performance is important
Is really ugly in terms of np.repeat vs. broadcasting!
Code:
import numpy as np
from scipy.optimize import minimize
np.random.seed(1)
""" Fake-problem (usually the job of the question-author!) """
def get_partial(N=10):
Y = np.random.uniform(size=N)
a, b = np.random.uniform(size=2)
return Y, a, b
""" Optimization """
def optimize(list_partials, N, M):
""" General approach:
This is a linear system of equations (with constraints)
Basic (unconstrained) form: min || b - Ax ||^2
"""
Y_all = np.vstack(map(lambda x: x[0], list_partials)).ravel() # flat 1d
a_all = np.hstack(map(lambda x: np.repeat(x[1], N), list_partials)) # repeat to be of same shape
b_all = np.hstack(map(lambda x: np.repeat(x[2], N), list_partials)) # """
def func(x):
A = x[:N]
B = x[N:]
return np.linalg.norm(Y_all - a_all * np.repeat(A, M) - b_all * np.repeat(B, M))
""" Example constraints: A >= B element-wise """
cons = ({'type': 'ineq',
'fun' : lambda x: x[:N] - x[N:]})
res = minimize(func, np.zeros(N*2), constraints=cons, method='SLSQP', options={'disp': True})
print(res)
print(Y_all - a_all * np.repeat(res.x[:N], M) - b_all * np.repeat(res.x[N:], M))
""" Test """
M = 4
N = 3
list_partials = [get_partial(N) for i in range(M)]
optimize(list_partials, N, M)
Output:
Optimization terminated successfully. (Exit mode 0)
Current function value: 0.9019356096498999
Iterations: 12
Function evaluations: 96
Gradient evaluations: 12
fun: 0.9019356096498999
jac: array([ 1.03786588e-04, 4.84041870e-04, 2.08129734e-01,
1.57609582e-04, 2.87599862e-04, -2.07959406e-01])
message: 'Optimization terminated successfully.'
nfev: 96
nit: 12
njev: 12
status: 0
success: True
x: array([ 1.82177105, 0.62803449, 0.63815278, -1.16960281, 0.03147683,
0.63815278])
[ 3.78873785e-02 3.41189867e-01 -3.79020251e-01 -2.79338679e-04
-7.98836875e-02 7.94168282e-02 -1.33155595e-01 1.32869391e-01
-3.73398306e-01 4.54460178e-01 2.01297470e-01 3.42682496e-01]
I did not check the result! If there is an error it's an implementation-error, not a conceptional one (my opinion)!

I agree with sascha that this is a linear problem. As I do not like constrains very much, I prefer, actually, to make it a non-linear without constrains. I do so by setting the vector A=(a1**2, a1**2+a2**2, a1**2+a2**2+a3**2, ...) like this it is ensured that it is all positive and A_i > A_j for i>j. That makes errors a bit problematic, as you now have to consider error propagation to get A1, A2, etc. including correlation, but I will have an important point on that at the end. The "simple" solution would look as follows:
import numpy as np
from scipy.optimize import leastsq
from random import random
np.set_printoptions(linewidth=190)
def generate_random_vector(n, sortIt=True):
out=np.fromiter( (random() for x in range(n) ),np.float)
if sortIt:
out.sort()
return out
def residuals(parameters,dataVec,dataLength,vecDims):
aParams=parameters[:dataLength]
bParams=parameters[dataLength:2*dataLength]
AParams=parameters[-2*vecDims:-vecDims]
BParams=parameters[-vecDims:]
YList=dataVec
AVec=[a**2 for a in AParams]##assures A_i > 0
BVec=[b**2 for b in BParams]
AAVec=np.cumsum(AVec)##assures A_i>A_j for i>j
BBVec=np.cumsum(BVec)
dist=[ np.array(Y)-a*np.array(AAVec)-b*np.array(BBVec) for Y,a,b in zip(YList,aParams,bParams) ]
dist=np.ravel(dist)
return dist
if __name__=="__main__":
aList=generate_random_vector(20, sortIt=False)
bList=generate_random_vector(20, sortIt=False)
AVec=generate_random_vector(5)
BVec=generate_random_vector(5)
YList=[a*AVec+b*BVec for a,b in zip(aList,bList)]
aGuess=20*[.2]
bGuess=20*[.3]
AGuess=5*[.4]
BGuess=5*[.5]
bestFitValues, covMX, infoDict, messages ,ier = leastsq(residuals, aGuess+bGuess+AGuess+BGuess ,args=(YList,20,5) ,full_output=True)
print "a"
print aList
besta = bestFitValues[:20]
print besta
print "b"
print bList
bestb = bestFitValues[20:40]
print bestb
print "A"
print AVec
bestA = bestFitValues[-2*5:-5]
realBestA = np.cumsum([x**2 for x in bestA])
print realBestA
print "B"
print BVec
bestB = bestFitValues[-5:]
realBestB = np.cumsum([x**2 for x in bestB])
print realBestB
print covMX
The problem on errors and correlation is that the solution to the problem is not unique. If Y = a A + b B is a solution and we, e.g., rotate such that A = c E + s F and B = -s E + c F then also Y = (ac-bs) E + (as+bc) F =e E + f F is a solution. The parameter space is, hence, completely flat at "the solution" resulting in huge errors and apocalyptic correlations.

Is it possible to compute double integral using scipy.integrate.fixed_quad?

I am trying to compute a double integral (over a triangle with nodes at (0,0), (0,1), (1,0)) using Gaussian quadrature of order n. However, running
import scipy.integrate as integrate
f = lambda x,y: x+y
inside = lambda x: integrate.fixed_quad(f, 0, 1-x, args=(x,), n=5)[0]
outside = integrate.fixed_quad(inside, 0, 1, n=5)
gives
Traceback (most recent call last):
File "", line 1, in
File "/Users/username/anaconda/lib/python3.5/site-packages/scipy/integrate/quadrature.py", line 82, in fixed_quad
return (b-a)/2.0 * np.sum(w*func(y, *args), axis=0), None
File "", line 1, in
File "/Users/username/anaconda/lib/python3.5/site-packages/scipy/integrate/quadrature.py", line 78, in fixed_quad
if np.isinf(a) or np.isinf(b):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
This is the second part of the question Can scipy.integrate.fixed_quad compute integral with functional boundaries?.

The answer to your question is, yes, under certain conditions.
For demonstration purposes, I first choose different bounds than you (11 instead of 1 - x).
Generally, one can solve these types of integrals using dblquad:
area_dblquad = integrate.dblquad(lambda x, y: x + y, 0, 1, lambda x: 0, lambda x: 11)[0]
which here returns 66. That is not an option as you mentioned in the comments.
One can now do this integration stepwise and it works fine for quad as well as fixed_quad:
def integrand(x, y):
return x + y
def fint_quad(x):
return integrate.quad(integrand, 0, 11, args=(x, ))[0]
def fint_fixed_quad(x):
return integrate.fixed_quad(integrand, 0, 11, args=(x, ), n=5)[0]
res_quad = integrate.quad(lambda x: fint_quad(x), 0, 1)
res_fixed_quad = integrate.fixed_quad(lambda x: fint_fixed_quad(x), 0, 1, n=5)
They both return 66 as well, as expected. That shows that it can work to compute double integrals using scipy.integrate.fixed_quad.
However, when one now changes the upper bound back to the one you had (from 11 to 1 - x), it still works for quad but crashes for fixed_quad:
area_dblquad = integrate.dblquad(lambda x, y: x + y, 0, 1, lambda x: 0, lambda x: 1 - x)[0]
res_quad = integrate.quad(lambda x: fint_quad(x), 0, 1)
both return 0.333333..., the call with fixed_quad results in the error you received. One can understand the error by looking on the source code:
x, w = _cached_roots_legendre(n)
x = np.real(x)
if np.isinf(a) or np.isinf(b):
raise ValueError("Gaussian quadrature is only available for "
"finite limits.")
y = (b-a)*(x+1)/2.0 + a
return (b-a)/2.0 * np.sum(w*func(y, *args), axis=-1), None
When one prints a and b one gets:
a: 0
b: 1
a: 0
b: [ 0.95308992 0.76923466 0.5 0.23076534 0.04691008]
So for the call with 1-x, b is actually a numpy array with n elements and one cannot compare an array to infinity, that's why it crashes. Whether that is an intended behavior or a bug, I can't answer; might be worth opening an issue on github.

fixed_quad requires f to accept vector inputs. And the result should be the mapped values for the inputs (i.e. something like map(f, xs)). With this in mind, just ensure your inside function returns mapped values, and you're ready to go.
import scipy.integrate as integrate
f = lambda y,x: x+y
inside = lambda xs, n: np.array([integrate.fixed_quad(f, 0, 1-x, args=(x,), n=n)[0]
for x in np.array(xs)])
order = 5
outside = integrate.fixed_quad(inside, 0, 1, n=order, args=(order,))
Also, be careful with the order of arguments for your integrand. Judging from your code arg=(x,), it seems that you want the inner integral to be done along y dimension. The first argument of the integrand is the dimension along which it is integrated. So it should be lambda y,x instead (note that this is also the shape of integrand expected by dblquad).