Develop Reference

Python solve ODE

How to solve a ODE，I have tried to use scipy.integrate.odeint, but what should I do if I don't know the initial value，
That's what I've defined：
alpha=0.204
beta=0.196
b=5.853
c=241.38
def Left_equation(x):
return alpha * x
def Right_equation(x):
return (b * (x ** 2)) + c
def diff_equation(x, t):
return (t ** beta) * (Right_equation(x) - Left_equation(x))
I also want to get the graph of the result,I don't know if I need to get the analytic solution first.

If the initial condition is not known, then the integration would need to be done symbolically. The Python package sympy can be used for symbolic integration of ordinary differential equations, as follows (using the function sympy.dsolve):
"""How to integrate symbolically an ordinary differential equation."""
import sympy
def main():
alpha = 0.204
beta = 0.196
b = 5.853
c = 241.38
t = sympy.symbols('t')
x = sympy.Function('x')(t)
dxdt = sympy.Derivative(x, t)
e = (t**beta) * ((b * (x**2)) + c - alpha * x)
x_eq = sympy.dsolve(dxdt - e)
if __name__ == '__main__':
main()
For this example, the function sympy.solvers.ode.ode.dsolve raises the exception PolynomialDivisionFailed. But the above code shows how to do symbolic integration.
An alternative is to numerically solve the differential equation (for specific initial condition), and compute solutions for a range of initial conditions, to explore how the solution depends on the initial condition. Using the function scipy.integrate.odepack.odeint of the Python package scipy, and the Python packages matplotlib and numpy, this can be done as follows:
"""How to integrate numerically an ordinary differential equation."""
from matplotlib import pyplot as plt
import numpy as np
import scipy.integrate
def main():
x0 = np.linspace(0, 0.2, 10) # multiple alternative initial conditions
t = np.linspace(0, 0.01, 100) # where to solve
x = scipy.integrate.odeint(deriv, x0, t)
# plot results
plt.plot(t, x)
plt.xlabel('t')
plt.ylabel('x')
plt.grid(True)
plt.show()
def deriv(x, t):
alpha = 0.204
beta = 0.196
b = 5.853
c = 241.38
return (t ** beta) * ((b * (x**2)) + c - alpha * x)
if __name__ == '__main__':
main()
As noted though in the documentation of the function scipy.integrate.odeint:
For new code, use scipy.integrate.solve_ivp to solve a differential equation.
The function scipy.integrate.solve_ivp can be used as follows:
"""How to integrate numerically an ordinary differential equation."""
from matplotlib import pyplot as plt
import numpy as np
import scipy.integrate
def main():
x0 = np.linspace(0, 0.2, 10) # multiple alternative initial conditions
t = (0.0, 0.01) # where to solve: (start, end)
res = scipy.integrate.solve_ivp(deriv, t, x0) # different order of
# arguments than for the function `scipy.integrate.odeint`
# plot results
plt.plot(res.t, res.y.T)
plt.xlabel('t')
plt.ylabel('x')
plt.grid(True)
plt.show()
def deriv(t, x): # not x, t as for the function `scipy.integrate.odeint`
alpha = 0.204
beta = 0.196
b = 5.853
c = 241.38
return (t ** beta) * ((b * (x**2)) + c - alpha * x)
if __name__ == '__main__':
main()
Please note the differences in arguments between the function scipy.integrate.odeint and the function scipy.integrate.solve_ivp.
The above code produces the following plot:

Improve speed of gradient descent

I am trying to maximize a target function f(x) with function scipy.optimize.minimum. But it usually takes 4-5 hrs to run the code because the function f(x) involves a lot of computation of complex matrix. To improve its speed, I want to use gpu. And I've already tried tensorflow package. Since I use numpy to define f(x), I have to convert it into tensorflow's format. However, it doesn't support the computation of complex matrix. What else package or means I can use? Any suggestions?
To specific my problem, I will show calculate scheme below:
Calculate the expectation :
-where H=x*H_0, x is the parameter
Let \phi go through the dynamics of Schrödinger equation
-Different H is correspond to a different \phi_end. Thus, parameter x determines the expectation
Change x, calculate the corresponding expectation
Find a specific x that minimize the expectation
Here is a simple example of part of my code:
import numpy as np
import cmath
from scipy.linalg import expm
import scipy.optimize as opt
# create initial complex matrixes
N = 2 # Dimension of matrix
H = np.array([[1.0 + 1.0j] * N] * N) # a complex matrix with shape(N, N)
A = np.array([[0.0j] * N] * N)
A[0][0] = 1.0 + 1j
# calculate the expectation
def value(phi):
exp_H = expm(H) # put the matrix in the exp function
new_phi = np.linalg.linalg.matmul(exp_H, phi)
# calculate the expectation of the matrix
x = np.linalg.linalg.matmul(H, new_phi)
expectation = np.inner(np.conj(phi), x)
return expectation
# Contants
tmax = 1
dt = 0.1
nstep = int(tmax/dt)
phi_init = [1.0 + 1.0j] * N
# 1st derivative of Schrödinger equation
def dXdt(t, phi, H): # 1st derivative of the function
return -1j * np.linalg.linalg.matmul(H, phi)
def f(X):
phi = [[0j] * N] * nstep # store every time's phi
phi[0] = phi_init
# phi go through the dynamics of Schrödinger equation
for i in range(nstep - 1):
phi[i + 1] = phi[i] - dXdt(i * dt, X[i] * H, phi[i]) * dt
# calculate the corresponding value
f_result = value(phi[-1])
return f_result
# Initialize the parameter
X0 = np.array(np.ones(nstep))
results = opt.minimize(f, X0) # minimize the target function
opt_x = results.x
PS:
Python Version: 3.7
Operation System: Win 10

Plotting Fourier Series coefficients in Python using Simpson's Rule

I want to 1. express Simpson's Rule as a general function for integration in python and 2. use it to compute and plot the Fourier Series coefficients of the function .
I've stolen and adapted this code for Simpson's Rule, which seems to work fine for integrating simple functions such as ,
or
Given period , the Fourier Series coefficients are computed as:
where k = 1,2,3,...
I am having difficulty figuring out how to express . I'm aware that since this function is odd, but I would like to be able to compute it in general for other functions.
Here's my attempt so far:
import matplotlib.pyplot as plt
from numpy import *
def f(t):
k = 1
for k in range(1,10000): #to give some representation of k's span
k += 1
return sin(t)*sin(k*t)
def trapezoid(f, a, b, n):
h = float(b - a) / n
s = 0.0
s += f(a)/2.0
for j in range(1, n):
s += f(a + j*h)
s += f(b)/2.0
return s * h
print trapezoid(f, 0, 2*pi, 100)
This doesn't give the correct answer of 0 at all since it increases as k increases and I'm sure I'm approaching it with tunnel vision in terms of the for loop. My difficulty in particular is with stating the function so that k is read as k = 1,2,3,...
The problem I've been given unfortunately doesn't specify what the coefficients are to be plotted against, but I am assuming it's meant to be against k.

Here's one way to do it, if you want to run your own integration or fourier coefficient determination instead of using numpy or scipy's built in methods:
import numpy as np
def integrate(f, a, b, n):
t = np.linspace(a, b, n)
return (b - a) * np.sum(f(t)) / n
def a_k(f, k):
def ker(t): return f(t) * np.cos(k * t)
return integrate(ker, 0, 2*np.pi, 2**10+1) / np.pi
def b_k(f, k):
def ker(t): return f(t) * np.sin(k * t)
return integrate(ker, 0, 2*np.pi, 2**10+1) / np.pi
print(b_k(np.sin, 0))
This gives the result
0.0
On a side note, trapezoid integration is not very useful for uniform time intervals. But if you desire:
def trap_integrate(f, a, b, n):
t = np.linspace(a, b, n)
f_t = f(t)
dt = t[1:] - t[:-1]
f_ab = f_t[:-1] + f_t[1:]
return 0.5 * np.sum(dt * f_ab)
There's also np.trapz if you want to use pre-builtin functionality. Similarly, there's also scipy.integrate.trapz

python divide by zero encountered in log - logistic regression

I'm trying to implement a multiclass logistic regression classifier that distinguishes between k different classes.
This is my code.
import numpy as np
from scipy.special import expit
def cost(X,y,theta,regTerm):
(m,n) = X.shape
J = (np.dot(-(y.T),np.log(expit(np.dot(X,theta))))-np.dot((np.ones((m,1))-y).T,np.log(np.ones((m,1)) - (expit(np.dot(X,theta))).reshape((m,1))))) / m + (regTerm / (2 * m)) * np.linalg.norm(theta[1:])
return J
def gradient(X,y,theta,regTerm):
(m,n) = X.shape
grad = np.dot(((expit(np.dot(X,theta))).reshape(m,1) - y).T,X)/m + (np.concatenate(([0],theta[1:].T),axis=0)).reshape(1,n)
return np.asarray(grad)
def train(X,y,regTerm,learnRate,epsilon,k):
(m,n) = X.shape
theta = np.zeros((k,n))
for i in range(0,k):
previousCost = 0;
currentCost = cost(X,y,theta[i,:],regTerm)
while(np.abs(currentCost-previousCost) > epsilon):
print(theta[i,:])
theta[i,:] = theta[i,:] - learnRate*gradient(X,y,theta[i,:],regTerm)
print(theta[i,:])
previousCost = currentCost
currentCost = cost(X,y,theta[i,:],regTerm)
return theta
trX = np.load('trX.npy')
trY = np.load('trY.npy')
theta = train(trX,trY,2,0.1,0.1,4)
I can verify that cost and gradient are returning values that are in the right dimension (cost returns a scalar, and gradient returns a 1 by n row vector), but i get the error
RuntimeWarning: divide by zero encountered in log
J = (np.dot(-(y.T),np.log(expit(np.dot(X,theta))))-np.dot((np.ones((m,1))-y).T,np.log(np.ones((m,1)) - (expit(np.dot(X,theta))).reshape((m,1))))) / m + (regTerm / (2 * m)) * np.linalg.norm(theta[1:])
why is this happening and how can i avoid this?

The proper solution here is to add some small epsilon to the argument of log function. What worked for me was
epsilon = 1e-5
def cost(X, y, theta):
m = X.shape[0]
yp = expit(X # theta)
cost = - np.average(y * np.log(yp + epsilon) + (1 - y) * np.log(1 - yp + epsilon))
return cost

You can clean up the formula by appropriately using broadcasting, the operator * for dot products of vectors, and the operator # for matrix multiplication — and breaking it up as suggested in the comments.
Here is your cost function:
def cost(X, y, theta, regTerm):
m = X.shape[0] # or y.shape, or even p.shape after the next line, number of training set
p = expit(X # theta)
log_loss = -np.average(y*np.log(p) + (1-y)*np.log(1-p))
J = log_loss + regTerm * np.linalg.norm(theta[1:]) / (2*m)
return J
You can clean up your gradient function along the same lines.
By the way, are you sure you want np.linalg.norm(theta[1:]). If you're trying to do L2-regularization, the term should be np.linalg.norm(theta[1:]) ** 2.

Cause:
This is happening because in some cases, whenever y[i] is equal to 1, the value of the Sigmoid function (theta) also becomes equal to 1.
Cost function:
J = (np.dot(-(y.T),np.log(expit(np.dot(X,theta))))-np.dot((np.ones((m,1))-y).T,np.log(np.ones((m,1)) - (expit(np.dot(X,theta))).reshape((m,1))))) / m + (regTerm / (2 * m)) * np.linalg.norm(theta[1:])
Now, consider the following part in the above code snippet:
np.log(np.ones((m,1)) - (expit(np.dot(X,theta))).reshape((m,1)))
Here, you are performing (1 - theta) when the value of theta is 1. So, that will effectively become log (1 - 1) = log (0) which is undefined.

I'm guessing your data has negative values in it. You can't log a negative.
import numpy as np
np.log(2)
> 0.69314718055994529
np.log(-2)
> nan
There are a lot of different ways to transform your data that should help, if this is the case.

def cost(X, y, theta):
yp = expit(X # theta)
cost = - np.average(y * np.log(yp) + (1 - y) * np.log(1 - yp))
return cost
The warning originates from np.log(yp) when yp==0 and in np.log(1 - yp) when yp==1. One option is to filter out these values, and not to pass them into np.log. The other option is to add a small constant to prevent the value from being exactly 0 (as suggested in one of the comments above)

Add epsilon value[which is a miniature value] to the log value so that it won't be a problem at all.
But i am not sure if it will give accurate results or not .

Integrating a multidimensional integral in scipy

Motivation: I have a multidimensional integral, which for completeness I have reproduced below. It comes from the computation of the second virial coefficient when there is significant anisotropy:
Here W is a function of all the variables. It is a known function, one which I can define a python function for.
Programming Question: How do I get scipy to integrate this expression? I was thinking of chaining two triple quads (scipy.integrate.tplquad) together, but I'm worried about performance and accuracy. Is there a higher dimensional integrator in scipy, one that can handle an arbitrary number of nested integrals? If not, what is the best way to do this?

With a higher-dimensional integral like this, monte carlo methods are often a useful technique - they converge on the answer as the inverse square root of the number of function evaluations, which is better for higher dimension then you'll generally get out of even fairly sophisticated adaptive methods (unless you know something very specific about your integrand - symmetries that can be exploited, etc.)
The mcint package performs a monte carlo integration: running with a non-trivial W that is nonetheless integrable so we know the answer we get (note that I've truncated r to be from [0,1); you'll have to do some sort of log transform or something to get that semi-unbounded domain into something tractable for most numerical integrators):
import mcint
import random
import math
def w(r, theta, phi, alpha, beta, gamma):
return(-math.log(theta * beta))
def integrand(x):
r = x[0]
theta = x[1]
alpha = x[2]
beta = x[3]
gamma = x[4]
phi = x[5]
k = 1.
T = 1.
ww = w(r, theta, phi, alpha, beta, gamma)
return (math.exp(-ww/(k*T)) - 1.)*r*r*math.sin(beta)*math.sin(theta)
def sampler():
while True:
r = random.uniform(0.,1.)
theta = random.uniform(0.,2.*math.pi)
alpha = random.uniform(0.,2.*math.pi)
beta = random.uniform(0.,2.*math.pi)
gamma = random.uniform(0.,2.*math.pi)
phi = random.uniform(0.,math.pi)
yield (r, theta, alpha, beta, gamma, phi)
domainsize = math.pow(2*math.pi,4)*math.pi*1
expected = 16*math.pow(math.pi,5)/3.
for nmc in [1000, 10000, 100000, 1000000, 10000000, 100000000]:
random.seed(1)
result, error = mcint.integrate(integrand, sampler(), measure=domainsize, n=nmc)
diff = abs(result - expected)
print "Using n = ", nmc
print "Result = ", result, "estimated error = ", error
print "Known result = ", expected, " error = ", diff, " = ", 100.*diff/expected, "%"
print " "
Running gives
Using n = 1000
Result = 1654.19633236 estimated error = 399.360391622
Known result = 1632.10498552 error = 22.0913468345 = 1.35354937522 %
Using n = 10000
Result = 1634.88583778 estimated error = 128.824988953
Known result = 1632.10498552 error = 2.78085225405 = 0.170384397984 %
Using n = 100000
Result = 1646.72936 estimated error = 41.3384733174
Known result = 1632.10498552 error = 14.6243744747 = 0.8960437352 %
Using n = 1000000
Result = 1640.67189792 estimated error = 13.0282663003
Known result = 1632.10498552 error = 8.56691239895 = 0.524899591322 %
Using n = 10000000
Result = 1635.52135088 estimated error = 4.12131562436
Known result = 1632.10498552 error = 3.41636536248 = 0.209322647304 %
Using n = 100000000
Result = 1631.5982799 estimated error = 1.30214644297
Known result = 1632.10498552 error = 0.506705620147 = 0.0310461413109 %
You could greatly speed this up by vectorizing the random number generation, etc.
Of course, you can chain the triple integrals as you suggest:
import numpy
import scipy.integrate
import math
def w(r, theta, phi, alpha, beta, gamma):
return(-math.log(theta * beta))
def integrand(phi, alpha, gamma, r, theta, beta):
ww = w(r, theta, phi, alpha, beta, gamma)
k = 1.
T = 1.
return (math.exp(-ww/(k*T)) - 1.)*r*r*math.sin(beta)*math.sin(theta)
# limits of integration
def zero(x, y=0):
return 0.
def one(x, y=0):
return 1.
def pi(x, y=0):
return math.pi
def twopi(x, y=0):
return 2.*math.pi
# integrate over phi [0, Pi), alpha [0, 2 Pi), gamma [0, 2 Pi)
def secondIntegrals(r, theta, beta):
res, err = scipy.integrate.tplquad(integrand, 0., 2.*math.pi, zero, twopi, zero, pi, args=(r, theta, beta))
return res
# integrate over r [0, 1), beta [0, 2 Pi), theta [0, 2 Pi)
def integral():
return scipy.integrate.tplquad(secondIntegrals, 0., 2.*math.pi, zero, twopi, zero, one)
expected = 16*math.pow(math.pi,5)/3.
result, err = integral()
diff = abs(result - expected)
print "Result = ", result, " estimated error = ", err
print "Known result = ", expected, " error = ", diff, " = ", 100.*diff/expected, "%"
which is slow but gives very good results for this simple case. Which is better is going to come down to how complicated your W is and what your accuracy requirements are. Simple (fast to evaluate) W with high accuracy will push you to this sort of method; complicated (slow to evaluate) W with moderate accuracy requirements will push you towards MC techniques.
Result = 1632.10498552 estimated error = 3.59054059995e-11
Known result = 1632.10498552 error = 4.54747350886e-13 = 2.7862628625e-14 %

Jonathan Dursi has made a very good answer. I will just add on to his answer.
Now scipy.integrate has a function named nquad that one can perform a multi-dimensional integral without hassle. See this link for more information. Below we compute the integral using nquad with Jonathan's example:
from scipy import integrate
import math
import numpy as np
def w(r, theta, phi, alpha, beta, gamma):
return(-math.log(theta * beta))
def integrand(r, theta, phi, alpha, beta, gamma):
ww = w(r, theta, phi, alpha, beta, gamma)
k = 1.
T = 1.
return (math.exp(-ww/(k*T)) - 1.)*r*r*math.sin(beta)*math.sin(theta)
result, error = integrate.nquad(integrand, [[0, 1], # r
[0, 2 * math.pi], # theta
[0, math.pi], # phi
[0, 2 * math.pi], # alpha
[0, 2 * math.pi], # beta
[0, 2 * math.pi]]) # gamma
expected = 16*math.pow(math.pi,5)/3
diff = abs(result - expected)
The result is more accurate than the chained tplquad:
>>> print(diff)
0.0

I'll just make a couple of general comments about how to accurately do this sort of integral, but this advice is not specific to scipy (too long for a comment, even though it is not an answer).
I don't know your use case, i.e. whether you are satisfied with a `good' answer with a few digits of accuracy which could be obtained straightforwardly using Monte Carlo as outlined in Jonathan Dursi's answer, or whether you really want to push the numerical accuracy as far as possible.
I've performed analytic, Monte Carlo and quadrature calculations of virial coefficients myself. If you want to do the integrals accurately, then there are a few things you should do:
Attempt to perform as many of the integrals exactly as possible; it may well be that integration in some of your coordinates is quite simple.
Consider transforming your variables of integration so that the integrand is as smooth as possible. (This helps for both Monte Carlo and quadrature).
For Monte Carlo, use importance sampling for best convergence.
For quadrature, with 7 integrals it may just be possible to get really fast convergence using tanh-sinh quadrature. If you can get it down to 5 integrals then you should be able to get 10's of digits of precision for your integral. I highly recommend mathtool / ARPREC for this purpose, available from David Bailey's homepage: http://www.davidhbailey.com/

First to say that I am not that good in math so please be kind. Anyway, here is my try:
Note that in your question there are 6 variables but 7 integrals!?
In Python using Sympy:
>>> r,theta,phi,alpha,beta,gamma,W,k,T = symbols('r theta phi alpha beta gamma W k T')
>>> W = r+theta+phi+alpha+beta+gamma
>>> Integral((exp(-W/(k*T))-1)*r**2*sin(beta)*sin(theta),(r,(0,2*pi)),(theta,(0,pi)),(phi,(0,2*pi)),(alpha,(0,2*pi)),(beta,(0,pi)),(gamma,(0,pi)))
>>> integrate((exp(-W)-1)*r**2*sin(beta)*sin(theta),(r,(0,2*pi)),(theta,(0,pi)),(phi,(0,2*pi)),(alpha,(0,2*pi)),(beta,(0,pi)),(gamma,(0,pi)))
and here is the result: [LateX code]
\begin{equation*}- \frac{128}{3} \pi^{6} - \frac{\pi^{2}}{e^{2 \pi}} - \frac{\pi}{e^{2 \pi}} - \frac{2}{e^{2 \pi}} - \frac{\pi^{2}}{e^{3 \pi}} - \frac{\pi}{e^{3 \pi}} - \frac{2}{e^{3 \pi}} - 3 \frac{\pi^{2}}{e^{6 \pi}} - 3 \frac{\pi}{e^{6 \pi}} - \frac{2}{e^{6 \pi}} - 3 \frac{\pi^{2}}{e^{7 \pi}} - 3 \frac{\pi}{e^{7 \pi}} - \frac{2}{e^{7 \pi}} + \frac{1}{2 e^{9 \pi}} + \frac{\pi}{e^{9 \pi}} + \frac{\pi^{2}}{e^{9 \pi}} + \frac{1}{2 e^{8 \pi}} + \frac{\pi}{e^{8 \pi}} + \frac{\pi^{2}}{e^{8 \pi}} + \frac{3}{e^{5 \pi}} + 3 \frac{\pi}{e^{5 \pi}} + 3 \frac{\pi^{2}}{e^{5 \pi}} + \frac{3}{e^{4 \pi}} + 3 \frac{\pi}{e^{4 \pi}} + 3 \frac{\pi^{2}}{e^{4 \pi}} + \frac{1}{2 e^{\pi}} + \frac{1}{2}\end{equation*}
You may play a bit more for your question ;)