I have an issue: I'm trying to find the minimum of a function which depends on several parameters that I'd like to change as well. let's take as a simplified example:
import numpy as np
import scipy.optimize as opt
def f(x, a, b, c):
f = a * x**2 + b * x + c
return f
I'd like to find the x which minimizes the function for different set of values of a, b, c, let's say for
a = [-1, 0, 1]
b = [0, 1, 2]
c = [0, 1]
ATM I have three nested loops and a minimization:
for p1 in a:
for p2 in b:
for p3 in c:
y = opt.minimize(f, x0=[0, ], args=(p1, p2, p3, ))
print(y)
which is really slow for the calculation I'm doing, but I haven't found any better so far. So, does anyone know a way or a package that would allow me to improve the efficiency?
You could use a combination of different techniques to improve the efficiency of your script:
Use itertools.product to generate every possible combination in the list a, b, c
Use multiprocessingto execute the minimizations in parallel.
Other than this, i can't think of a way to optimize the efficiency of the code. As was pointed out in the comment, the constant value c has no influence on the minimization. But i'm sure the quadratic function is just an example.
I took the code of the multiprocessing part from here.
Here's the working code.
import numpy as np
import scipy.optimize as opt
import itertools
from multiprocessing import Pool
def f(x, a, b, c):
f = a * x**2 + b * x + c
return f
def mini(args):
res = opt.minimize(f, x0=np.array([0]), args=args)
return res.x
if __name__=="__main__":
a = np.linspace(-1,2,100)
b = np.linspace(0,2,100)
c = [0, 1]
args = list(itertools.product(a,b,c))
print("Number of combos:" + str(len(args)))
p = Pool(4)
import time
t0 = time.time()
res = p.map(mini, args)
print(time.time()-t0)
Even these 20000 combinations only need 5,28 seconds on my average laptop.
scipy.optimize.newton can do this.
Related
I am not sure about the best way to ask this question, but I am trying to find the long-term state of an ODE system with an arbitrary extra constraint that needs to be fulfilled.
Ex:
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
def f(t, X, a, b):
return a*X + b
N = 5
X0 = np.random.randn(N)
t_range = [0, 1]
a = 3
b = 4
sol = solve_ivp(f, t_range, X0, args=(a, b))
for i in range(N):
plt.plot(sol['t'], sol['y'][i, :])
In the code above, f is the right-hand-side of my ODE model. N is the dimension of my state vector X, t_range is the interval I want to view the values of X over, X0 is my initial condition, and a and b are just arbitrary model parameters.
The code written produces the following chart:
However, let's say I didn't want to evaluate X over some fixed t_range. Let's say I wanted to keep solving the ODE until some condition was satisfied:
def bc(X) -> bool:
""" True means we stop solving. """
return (X < 0).any()
In other words, I want to do something like this:
X = X0
while not bc(X):
X = new value from ODE solver
Is there an easy way to do this in scipy? If not, is there a straightforward way to implement this?
Please let me know if I can clarify anything better.
#Lutz Lehmann answered my question in the comments.
If I wanted the solver to stop when one entry in X became < 0:
event = lambda t, X, a, b : min(X) # account for when args are passed to event func
event.terminal # tells it to stop
sol = solve_ivp(f, t_range, X0, args=(a, b), events = event)
I have the following python function required inputs a and b.
def interimfunc(x,y,z):
#this is a dummy function - not a part of the actual question but included for completeness.
#the actual function involves some statistical treatment - but that is not a problem here.
sol = x*y+z
return sol
def finalfunc(a, b):
interimsol1 = interimfunc(0.4,a,b)
interimsol2 = interimfunc(0.8,a,b)
finalsol = interimsol1/interimsol2
return finalsol
if finalsol is a known value.
How do I find out the unknowns "a" and "b" by solving non-linear system of equations??
===
I got 4 down-votes after posting this. I am a mechanical engineer and learning computer science. I did try researching on the internet but need to refine my search - hence I asked a question to experts here.
===
In addition to above equations - there is one more information:
interimsol2 = interimfunc(0.8,a,b)
where interimsol2 = 10 #i.e. known value
How do we inclde this new information in our unknown finding?
===
Actual problem as requested below by #SergeyIvanov
def func(mu, sigma):
tenpercent = st.norm.ppf(2, mu, sigma)
ninetypercent = st.norm.ppf(2, mu, sigma)
rfs = tenpercent/ninetypercent
return rfs
I think you should solve a system of nonlinear equations. This code should solve your problem in case of two equations for two known solutions (of course you can extend it):
from scipy.optimize import fsolve
known_values = [3,5]
def interimfunc(x,y,z):
sol = x*y+z
return sol
def finalfunc(a, b):
interimsol1 = interimfunc(0.4,a,b)
interimsol2 = interimfunc(0.8,a,b)
finalsol = interimsol1/interimsol2
return finalsol
def equations(p):
a, b = p
return (finalfunc(a,b) - known_values[0], # finalfunc(a,b) == solution1
finalfunc(a,b) - known_values[1]) # finalfunc(a,b) == solution2
a, b = fsolve(equations, (1, 1))# solution
print(a,b)
# -6192.07497308 5779.26987919
print(equations((a, b)))
# (1.0000003476651482, -0.99999965233485177) <-- bad convergence beacause there is no free paremeter in finalfunc.
But it works only with equal known_values, that is pointless (solution will be a random combination of a and b). The problem is that you should have something to distinguish two equations finalfunc (e.g. additional parameter), because you can get diffrent solutions only with different arguments. So finally you should have something like this:
from scipy.optimize import fsolve
def interimfunc(x,y,z):
sol = x*y+z
return sol
def finalfunc(a, b, c ):
interimsol1 = interimfunc(0.4,a,b) + c
interimsol2 = interimfunc(0.8,a,b) + c
finalsol = interimsol1/interimsol2
return finalsol
known_values = [0.8260869565217391,0.8333333333333334]
def equations(p):
a, b = p
return (finalfunc(a,b,0) - known_values[0], # finalfunc(a,b,c) == solution1
finalfunc(a,b,1) - known_values[1]) # finalfunc(a,b,c) == solution2
a, b = fsolve(equations, (1, 1))# solution
print(a,b)
print(equations((a, b)))
# 10.0 15.0 <-- correct values
# (4.4408920985006262e-16, 2.2204460492503131e-16) <-- good convergence
For last example:
from scipy.optimize import fsolve
import scipy.stats as st
def equations(p):
mu, sigma = p
tenpercent = st.norm.ppf(2, mu, sigma)
ninetypercent = st.norm.ppf(2, mu, sigma)
return (ninetypercent - 500,
tenpercent / ninetypercent - 1.0)
mu, sigma = fsolve(equations,x0=(100, 10))# solution
print("mu, sigma:",mu, sigma)
print(equations((mu, sigma)))
The problem here is that ppf can generate nan and ruin an optimization process. So guess values should be proposed very carefully.
I have some large arrays each with i elements, call them X, Y, Z, for which I need to find some values a, b--where a and b are real numbers between 0 and 1--such that, for the following functions,
r = X - a*Y - b*Z
r_av = Sum(r)/i
rms = Sum((r - r_av)^2), summing over the i pixels
I want to minimize the rms. Basically I'm looking to minimize the scatter in r, and thus need to find the right a and b to do that. So far I have thought to do this in nested loops in one of two ways: either 1)just looping through a range of possible a,b and then selecting out the smallest rms, or 2)inserting a while statement so that the loop will terminate once rms stops decreasing with decreasing a,b for instance. Here's some pseudocode for these:
1) List
for a = 1
for b = 1
calculate m
b = b - .001
a = a - .001
loop 1000 times
sort m values, from smallest
print (a,b) corresponding to smallest m
2) Terminate
for a = 1
for b = 1
calculate m
while m > previous step,
b = b - .001
a = a - .001
Is one of these preferable? Or is there yet another, better way to go about this? Any tips would be greatly appreciated.
There is already a handy formula for least squares fitting.
I came up with two different ways to solve your problem.
For the first one, consider the matrix K:
L = len(X)
K = np.identity(L) - np.ones((L, L)) / L
In your case, A and B are defined as:
A = K.dot(np.array([Y, Z]).transpose())
B = K.dot(np.array([X]).transpose())
Apply the formula to find C that minimizes the error A * C - B:
C = np.linalg.inv(np.transpose(A).dot(A))
C = C.dot(np.transpose(A)).dot(B)
Then the result is:
a, b = C.reshape(2)
Also, note that numpy already provides linalg.lstsq that does the exact same thing:
a, b = np.linalg.lstsq(A, B)[0].reshape(2)
A simpler way is to define A as:
A = np.array([Y, Z, [1]*len(X)]).transpose()
Then solve it against X to get the coefficients and the mean:
a, b, mean = np.linalg.lstsq(A, X)[0]
If you need a proof of this result, have a look at this post.
Example:
>>> import numpy as np
>>> X = [5, 7, 9, 5]
>>> Y = [2, 0, 4, 1]
>>> Z = [7, 2, 4, 6]
>>> A = np.array([Y, Z, [1] * len(X)]).transpose()
>>> a, b, mean = np.linalg.lstsq(A, X)[0]
>>> print(a, b, mean)
0.860082304527 -0.736625514403 8.49382716049
This is my first attempt to use JIT for python and this is the use case I want to speed up. I read a bit about numba and it seemed simple enough but the following code didn't provide any speedup. Please excuse any obvious mistakes I may be making.
I also tried to do what the basic tutorial of cython suggests but again no difference in time.
http://docs.cython.org/src/tutorial/cython_tutorial.html
I'm guessing I have to do something like declare variables? Use other libraries? Use for loops exclusively for everything? I'd appreciate any guidance or examples I can refer to.
For example I know from a previous question Elementwise operations in mpmath slow compared to numpy and its solution that using gmpy instead of mpmath was significantly faster.
import numpy as np
from scipy.special import eval_genlaguerre
from sympy import mpmath as mp
from sympy.mpmath import laguerre as genlag2
import collections
from numba import jit
import time
def len2(x):
return len(x) if isinstance(x, collections.Sized) else 1
#jit # <-- removing this doesn't change the output time if anything it's slower with this
def laguerre(a, b, x):
fun = np.vectorize(genlag2)
return fun(a, b, x)
def f1( a, b, c ):
t = time.time()
M = np.ones( [ len2(a), len2(b), len2(c) ] )
A, B, C = np.meshgrid( a, b, c, indexing = 'ij' )
temp = laguerre(A, B, C)
M *= temp
print 'part1: ', time.time() - t
t = time.time()
A, B = np.meshgrid( a, b, indexing= 'ij' )
temp = np.array( [[ mp.fac(x1)/mp.fac(y1) for x1,y1 in zip(x2,y2)] for x2,y2 in zip(A, B)] )
temp = np.reshape( temp, [ len(a), len(b), 1 ] )
temp = np.repeat( temp, len(c), axis = 2 )
print 'part2 so far:', time.time() - t
M *= temp
print 'part2 finally', time.time() - t
t = time.time()
a = mp.arange( 30 )
b = mp.arange( 10 )
c = mp.linspace( 0, 100, 100 )
M = f1( a, b, c)
Better to use numba with vectorize with self-defined decorators, if not defined lazy action will execute, which may lead to slow down the process.
Jit is slow compared to vectorize in my opinion.
Is there a way to speed up this code:
import mpmath as mp
import numpy as np
from time import time as epochTime
def func(E):
f = lambda theta: mp.sin(theta) * mp.exp(E * (mp.cos(theta**2) + \
mp.cos(theta)**2))
return f
start = epochTime()
mp.mp.dps = 15
mp.mp.pretty = True
E = np.linspace(0, 10, 200)
ints = [mp.quadgl(func(e), [0, mp.pi]) for e in E] # Main Job
print ('Took:{:.3}s'.format(epochTime() - start))
Running your code, I timed it to 5.84s
using Memoize and simplifying expressions:
cos = Memoize(mp.cos)
sin = Memoize(mp.sin)
def func(E):
def f(t):
cost = cos(t)
return sin(t) * mp.exp(E * (cos(t*t) + cost*cost))
return f
I got it down to 3.25s first time, and ~2.8s in the next iterations.
(An even better approach might be using lru_cache from the standard library, but I did not try to time it).
If you are running similar code many times, it may be sensible to Memoize() both func and f, so the computations become trivial ( ~0.364s ).
Replacing mp with math for cos/sin/exp, I got down to ~1.3s, and now memoizing make the performance worse, for some reason (~1.5s, I guess the lookup time became dominant).
In general, you want to avoid calls to transcendent functions like sin, cos, exp, ln as much as possible, especially in a "hot" function like an integrand.
Replace x**2 by x*x (often x**2 calls a generic=slow exponentiation function)
use variables for "expensive" intermediate terms which are used more than once
transform your equation to reduce or eliminate transcendent functions
special-case for typical parameter values. Integer exponents are a frequent candidate.
precompute everything that is constant, espc. in parameterized functions
For the particular example you can substitute z=cos(theta). It is dz = -sin(theta)dtheta. Your integrand becomes
-exp(E*(z^2 + cos(arccos(z)^2))
saving you some of the transcendent function calls. The boundaries [0, pi] become [1, -1]. Also avoid x**2, better use x*x.
Complete code:
import mpmath as mp
import numpy as np
from time import time as epochTime
def func(E):
def f(z):
acz = mp.acos(z)
return -mp.exp(E * (mp.cos(acz*acz) + z*z))
return f
start = epochTime()
mp.mp.dps = 15
mp.mp.pretty = True
E = np.linspace(0, 10, 200)
ints = [mp.quadgl(func(e), [1.0, -1.0]) for e in E] # Main Job
print ('Took:{:.3}s'.format(epochTime() - start))