Develop Reference

How to use Gradient Descent to solve this multiple terms trigonometry function?

Question is like this:
f(x) = A sin(2π * L * x) + B cos(2π * M * x) + C sin(2π * N * x)
and L,M,N are constants integer, 0 <= L,M,N <= 100
and A,B,C can be any possible integers.
Here is the given data:
x = [0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11,0.12,0.13,0.14,0.15,0.16,0.17,0.18,0.19,0.2,0.21,0.22,0.23,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5,0.51,0.52,0.53,0.54,0.55,0.56,0.57,0.58,0.59,0.6,0.61,0.62,0.63,0.64,0.65,0.66,0.67,0.68,0.69,0.7,0.71,0.72,0.73,0.74,0.75,0.76,0.77,0.78,0.79,0.8,0.81,0.82,0.83,0.84,0.85,0.86,0.87,0.88,0.89,0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99]
y = [4,1.240062433,-0.7829654986,-1.332487982,-0.3337640721,1.618033989,3.512512389,4.341307895,3.515268061,1.118929599,-2.097886967,-4.990538967,-6.450324073,-5.831575611,-3.211486891,0.6180339887,4.425660706,6.980842552,7.493970785,5.891593744,2.824429495,-0.5926374511,-3.207870455,-4.263694544,-3.667432785,-2,-0.2617162175,0.5445886005,-0.169441247,-2.323237059,-5.175570505,-7.59471091,-8.488730333,-7.23200463,-3.924327772,0.6180339887,5.138501587,8.38127157,9.532377045,8.495765687,5.902113033,2.849529206,0.4768388529,-0.46697525,0.106795821,1.618033989,3.071952496,3.475795162,2.255463709,-0.4905371745,-4,-7.117914956,-8.727599664,-8.178077181,-5.544088451,-1.618033989,2.365340134,5.169257268,5.995297102,4.758922924,2.097886967,-0.8873135564,-3.06024109,-3.678989552,-2.666365632,-0.6180339887,1.452191817,2.529722611,2.016594378,-0.01374122059,-2.824429495,-5.285215072,-6.302694708,-5.246870619,-2.210419738,2,6.13956874,8.965976562,9.68000641,8.201089581,5.175570505,1.716858387,-1.02183483,-2.278560533,-1.953524751,-0.6180339887,0.7393509358,1.129293593,-0.02181188158,-2.617913164,-5.902113033,-8.727381729,-9.987404016,-9.043589913,-5.984648344,-1.618033989,2.805900027,6.034770001,7.255101454,6.368389697]
enter image description here
How to use Gradient Descent to solve this multiple terms trigonometry function?

Gradient descent is not well suited for optimisation over integers. You can try a navie relaxation where you solve in floats, and hope the rounded solution is still ok.
from autograd import grad, numpy as jnp
import numpy as np
def cast(params):
[A, B, C, L, M, N] = params
L = jnp.minimum(jnp.abs(L), 100)
M = jnp.minimum(jnp.abs(M), 100)
N = jnp.minimum(jnp.abs(N), 100)
return A, B, C, L, M, N
def pred(params, x):
[A, B, C, L, M, N] = cast(params)
return A *jnp.sin(2 * jnp.pi * L * x) + B*jnp.cos(2*jnp.pi * M * x) + C * jnp.sin(2 * jnp.pi * N * x)
x = [0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11,0.12,0.13,0.14,0.15,0.16,0.17,0.18,0.19,0.2,0.21,0.22,0.23,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5,0.51,0.52,0.53,0.54,0.55,0.56,0.57,0.58,0.59,0.6,0.61,0.62,0.63,0.64,0.65,0.66,0.67,0.68,0.69,0.7,0.71,0.72,0.73,0.74,0.75,0.76,0.77,0.78,0.79,0.8,0.81,0.82,0.83,0.84,0.85,0.86,0.87,0.88,0.89,0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99]
y = [4,1.240062433,-0.7829654986,-1.332487982,-0.3337640721,1.618033989,3.512512389,4.341307895,3.515268061,1.118929599,-2.097886967,-4.990538967,-6.450324073,-5.831575611,-3.211486891,0.6180339887,4.425660706,6.980842552,7.493970785,5.891593744,2.824429495,-0.5926374511,-3.207870455,-4.263694544,-3.667432785,-2,-0.2617162175,0.5445886005,-0.169441247,-2.323237059,-5.175570505,-7.59471091,-8.488730333,-7.23200463,-3.924327772,0.6180339887,5.138501587,8.38127157,9.532377045,8.495765687,5.902113033,2.849529206,0.4768388529,-0.46697525,0.106795821,1.618033989,3.071952496,3.475795162,2.255463709,-0.4905371745,-4,-7.117914956,-8.727599664,-8.178077181,-5.544088451,-1.618033989,2.365340134,5.169257268,5.995297102,4.758922924,2.097886967,-0.8873135564,-3.06024109,-3.678989552,-2.666365632,-0.6180339887,1.452191817,2.529722611,2.016594378,-0.01374122059,-2.824429495,-5.285215072,-6.302694708,-5.246870619,-2.210419738,2,6.13956874,8.965976562,9.68000641,8.201089581,5.175570505,1.716858387,-1.02183483,-2.278560533,-1.953524751,-0.6180339887,0.7393509358,1.129293593,-0.02181188158,-2.617913164,-5.902113033,-8.727381729,-9.987404016,-9.043589913,-5.984648344,-1.618033989,2.805900027,6.034770001,7.255101454,6.368389697]
def loss(params):
p = pred(params, np.array(x))
return jnp.mean((np.array(y)-p)**2)
params = np.array([np.random.random()*100 for _ in range(6)])
for _ in range(10000):
g = grad(loss)
params = params - 0.001*g(params)
print("Relaxed solution", cast(params), "loss=", loss(params))
constrained_params = np.round(cast(params))
print("Integer solution", constrained_params, "loss=", loss(constrained_params))
print()
Since the problem will have a lot of local minima, you might need to run it multiple times.

It's quite hard to use gradient descent to find a solution to this problem, because it tends to get stuck when changing the L, M, or N parameters. The gradients for those can push it away from the right solution, unless it is very close to an optimal solution already.
There are ways to get around this, such as basinhopping or random search, but because of the function you're trying to learn, you have a better alternative.
Since you're trying to learn a sinusoid function, you can use an FFT to find the frequencies of the sine waves. Once you have those frequencies, you can find the amplitudes and phases used to generate the same sine wave.
Pardon the messiness of this code, this is my first time using an FFT.
import scipy.fft
import numpy as np
import math
import matplotlib.pyplot as plt
def get_top_frequencies(x, y, num_freqs):
x = np.array(x)
y = np.array(y)
# Find timestep (assume constant timestep)
dt = abs(x[0] - x[-1]) / (len(x) - 1)
# Take discrete FFT of y
spectral = scipy.fft.fft(y)
freq = scipy.fft.fftfreq(y.shape[0], d=dt)
# Cut off top half of frequencies. Assumes input signal is real, and not complex.
spectral = spectral[:int(spectral.shape[0] / 2)]
# Double amplitudes to correct for cutting off top half.
spectral *= 2
# Adjust amplitude by sampling timestep
spectral *= dt
# Get ampitudes for all frequencies. This is taking the magnitude of the complex number
spectral_amplitude = np.abs(spectral)
# Pick frequencies with highest amplitudes
highest_idx = np.argsort(spectral_amplitude)[::-1][:num_freqs]
# Find amplitude, frequency, and phase components of each term
highest_amplitude = spectral_amplitude[highest_idx]
highest_freq = freq[highest_idx]
highest_phase = np.angle(spectral[highest_idx]) / math.pi
# Convert it into a Python function
function = ["def func(x):", "return ("]
for i, components in enumerate(zip(highest_amplitude, highest_freq, highest_phase)):
amplitude, freq, phase = components
plus_sign = " +" if i != (num_freqs - 1) else ""
term = f"{amplitude:.2f} * math.cos(2 * math.pi * {freq:.2f} * x + math.pi * {phase:.2f}){plus_sign}"
function.append(" " + term)
function.append(")")
return "\n ".join(function)
x = [0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11,0.12,0.13,0.14,0.15,0.16,0.17,0.18,0.19,0.2,0.21,0.22,0.23,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5,0.51,0.52,0.53,0.54,0.55,0.56,0.57,0.58,0.59,0.6,0.61,0.62,0.63,0.64,0.65,0.66,0.67,0.68,0.69,0.7,0.71,0.72,0.73,0.74,0.75,0.76,0.77,0.78,0.79,0.8,0.81,0.82,0.83,0.84,0.85,0.86,0.87,0.88,0.89,0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99]
y = [4,1.240062433,-0.7829654986,-1.332487982,-0.3337640721,1.618033989,3.512512389,4.341307895,3.515268061,1.118929599,-2.097886967,-4.990538967,-6.450324073,-5.831575611,-3.211486891,0.6180339887,4.425660706,6.980842552,7.493970785,5.891593744,2.824429495,-0.5926374511,-3.207870455,-4.263694544,-3.667432785,-2,-0.2617162175,0.5445886005,-0.169441247,-2.323237059,-5.175570505,-7.59471091,-8.488730333,-7.23200463,-3.924327772,0.6180339887,5.138501587,8.38127157,9.532377045,8.495765687,5.902113033,2.849529206,0.4768388529,-0.46697525,0.106795821,1.618033989,3.071952496,3.475795162,2.255463709,-0.4905371745,-4,-7.117914956,-8.727599664,-8.178077181,-5.544088451,-1.618033989,2.365340134,5.169257268,5.995297102,4.758922924,2.097886967,-0.8873135564,-3.06024109,-3.678989552,-2.666365632,-0.6180339887,1.452191817,2.529722611,2.016594378,-0.01374122059,-2.824429495,-5.285215072,-6.302694708,-5.246870619,-2.210419738,2,6.13956874,8.965976562,9.68000641,8.201089581,5.175570505,1.716858387,-1.02183483,-2.278560533,-1.953524751,-0.6180339887,0.7393509358,1.129293593,-0.02181188158,-2.617913164,-5.902113033,-8.727381729,-9.987404016,-9.043589913,-5.984648344,-1.618033989,2.805900027,6.034770001,7.255101454,6.368389697]
print(get_top_frequencies(x, y, 3))
That produces this function:
def func(x):
return (
5.00 * math.cos(2 * math.pi * 10.00 * x + math.pi * 0.50) +
4.00 * math.cos(2 * math.pi * 5.00 * x + math.pi * -0.00) +
2.00 * math.cos(2 * math.pi * 3.00 * x + math.pi * -0.50)
)
Which is not quite the format you specified - you asked for two sins and one cos, and for no phase parameter. However, using the trigonometric identity cos(x) = sin(pi/2 - x), you can convert this into an equivalent expression that matches what you want:
def func(x):
return (
5.00 * math.sin(2 * math.pi * -10.00 * x) +
4.00 * math.cos(2 * math.pi * 5.00 * x) +
2.00 * math.sin(2 * math.pi * 3.00 * x)
)
And there's the original function!

How to use multiprocessing pool for a multivariable function python

I'm trying to use the library multiprocessing, to run the next code:
import matplotlib.pyplot as plt
import numpy as np
from multiprocessing import Pool
def newtalt(fun,x0,err,mit):
xnew = x0.copy()
F, dF = fun(x0)
r = F.copy()
M = dF.copy()
sigma = np.linalg.norm(r)
for k in range(mit):
if sigma < err:
break
d = np.linalg.solve(M, -r)
xnew = xnew + d
r, M = fun(xnew)
sigma = np.linalg.norm(r)
return xnew, k
def fun(x):
f_r = x[0] ** 3 - 3 * x[0] * (x[1] ** 2) - 1
f_i = 3 * (x[0]**2) * x[1] - (x[1] ** 3)
f = np.array([f_r,f_i])
df_rx = 3 * (x[0] ** 2) - 3 * (x[1] ** 2)
df_ry = -6 * x[0] * x[1]
df_ix = 6 * x[0] * x[1]
df_iy = 3 * (x[0] ** 2) - 3 * (x[1] ** 2)
df = np.array([[df_rx,df_ry],[df_ix,df_iy]])
return f, df
if __name__=='__main__':
pool = Pool(processes=10)
err = 1e-5
mit = 300
N = 50
x = np.linspace(-2.5, 2.5, N)
y = np.linspace(-2.5, 2.5, N)
A = np.zeros((N, N))
B = np.zeros([N, N, 4], dtype=int)
for i in range(N):
for j in range(N):
z = np.array([x[i], y[j]])
xsol, it = pool.map(newtonalt,(fun,z,err,mit))
pool.close()
A[i,j] = it
plt.imshow(A,cmap='Set1')
plt.show()
Cleary it doesn't work, 'cause I don't really know how to use multiprocessing properly, and it's even more difficult when the function you use have multiple arguments. I read that isn't good use .pool() inside a for loop, but anyway, as I was saying, I'm really lost with this library.

How do you solve a non-linear equation using fsolve on python?

I attempted to use the code below as a guide, in order to solve a non-linear equation, but I continue to get errors such as "object too deep for desired array" and "Result from function call is not a proper array of floats".
from scipy.optimize import fsolve
from math import exp
def equations(vars):
x, y = vars
eq1 = x+y**2-4
eq2 = exp(x) + x*y - 3
return [eq1, eq2]
x, y = fsolve(equations, (1, 1))
print(x, y)
My code will be posted below. It points out the error on the line "Q = fsolve(equations, 1)"
%reset -f
from math import *
T = 4 # N·m
ω = 1800*(pi/30) # rad/s
A1 = .00131 # m^2
A2 = .00055 # m^2
P1 = 12000 # Pa
P2 = 200000 # Pa
ρ = 1000 # kg/m^3
μ = .89e-3 # N·s/m^2
η = .57 # % efficiency
g = 9.81 # m/s^2
γ = 9810 # N/m^3
Z2 = .7 # m
P_motor = T*ω
print(P_motor)
P_pump = P_motor*η
print(P_pump)
from scipy.optimize import fsolve
def equations(vars):
Q = vars
eq1 = γ*Q*( ((P2-P1)/γ) + ( ( ( (Q**2) / (A2**2) ) - ( (Q**2) / (A1**2) ) )/2*g) + Z2) - P_pump
return [eq1]
Q = fsolve(equations, 1)
print(Q)

Since you have only one equation with one unknown variable, you don't need to put the output in a list. You can replace return [eq1] with return eq1.
The new code would be:
%reset -f
from math import *
T = 4 # N·m
ω = 1800*(pi/30) # rad/s
A1 = .00131 # m^2
A2 = .00055 # m^2
P1 = 12000 # Pa
P2 = 200000 # Pa
ρ = 1000 # kg/m^3
μ = .89e-3 # N·s/m^2
η = .57 # % efficiency
g = 9.81 # m/s^2
γ = 9810 # N/m^3
Z2 = .7 # m
P_motor = T*ω
print(P_motor)
P_pump = P_motor*η
print(P_pump)
from scipy.optimize import fsolve
def equations(vars):
Q = vars
eq1 = γ*Q*( ((P2-P1)/γ) + ( ( ( (Q**2) / (A2**2) ) - ( (Q**2) / (A1**2) ) )/2*g) + Z2) - P_pump
return eq1
Q = fsolve(equations, 1)
print(Q)
Output:
753.9822368615503
429.7698750110836
[0.0011589]

The documentation states
func : callable f(x, *args)
A function that takes at least one (possibly vector) argument, and returns a value of the same length.
if your input is a list of 2 values, it is expecting the function to return something of the same shape. So in your 1st example, you pass [x,y] and you return [eq1, eq2], so it works, but in second case, you pass a scalar and return a list
So, you can change your input to Q = fsolve(equations, (1,)) or change your returned value to return eq1:
def equations(vars):
Q = vars
eq1 = γ*Q*( ((P2-P1)/γ) + ( ( ( (Q**2) / (A2**2) ) - ( (Q**2) / (A1**2) ) )/2*g) + Z2) - P_pump
return eq1

Minimize system of nonlinear equation (integral on exponent)

General:
I am using maximum entropy to find distribution for on positive integers vectors, I can estimate the mean and variance, and have three equation I am trying to find a and b,
The equations:
integral(exp(a*x^2+bx+c) from (0 , infinity))-1
integral(xexp(ax^2+bx+c)from (0 , infinity))- mean
integral(x^2*exp(a*x^2+bx+c) from (0 , infinity))- mean^2 - var
(integrals between [0,∞))
The problem:
I am trying to use numerical solver and I used fsolve of sympy
But I guess I am missing some knowledge.
My code:
import numpy as np
import sympy as sym
from scipy.optimize import *
def myFunction(x,*data):
y = sym.symbols('y')
m,v=data
F = [0]*3
x[0] = - abs(x[0])
print(x)
F[0] = (sym.integrate(sym.exp(x[0] * y ** 2 + x[1] * y + x[2]), (y, 0,sym.oo)) -1).evalf()
F[1] = (sym.integrate(y*sym.exp(x[0] * y ** 2 + x[1] * y + x[2]), (y, 0,sym.oo))-m).evalf()
F[2] = (sym.integrate((y**2)*sym.exp(x[0] * y ** 2 + x[1] * y + x[2]), (y,0,sym.oo)) -v-m).evalf()
print(F)
return F
data = (10,3.5) # mean and var for example
xGuess = [1, 1, 1]
z = fsolve(myFunction,xGuess,args = data)
print(z)
my result are not that accurate, is there a better way to solve it?
integral(exp(a*x^2+bx+c))-1 = 5.67659292676884
integral(xexp(ax^2+bx+c))- mean = −1.32123173796713
integral(x^2*exp(a*x^2+bx+c))- mean^2 - var = −2.20825624606312
Thanks

I have rewritten the problem replacing sympy with numpy and lambdas (inline functions).
Also note that in your problem statement you subtract the third equation with $mean^2$, but in your code you only subtract $mean$.
import numpy as np
from scipy.optimize import minimize
from scipy.integrate import quad
def myFunction(x,data):
m,v=data
F = np.zeros(3) # use numpy array
# use scipy.integrade.quad for integration of lambda functions
# quad output is (result, error), so we just select the result value at the end
F[0] = quad(lambda y: np.exp(x[0] * y ** 2 + x[1] * y + x[2]), 0, np.inf)[0] -1
F[1] = quad(lambda y: y*np.exp(x[0] * y ** 2 + x[1] * y + x[2]), 0, np.inf)[0] -m
F[2] = quad(lambda y: (y**2)*np.exp(x[0] * y ** 2 + x[1] * y + x[2]), 0, np.inf)[0] -v-m**2
# minimize the squared error
return np.sum(F**2)
data = (10,3.5) # mean and var for example
xGuess = [-1, 1, 1]
z = minimize(lambda x: myFunction(x, data), x0=xGuess,
bounds=((None, 0), (None, None), (None, None))) # use bounds for negative first coefficient
print(z)
# x: array([-0.99899311, 2.18819689, 1.85313181])
Does this seem more reasonable?

Solving ODE with solve_ivp gets incredibly slow or freezes completely

I use solve_ivp to solve an ODE:
def test_ode(t, y):
dydt = C - y + (y ** 8 / (1 + y ** 8))
return dydt
steady_state = []
for C in np.linspace(0, 1, 1001):
sol = solve_ivp(test_ode, [0, 1e06], [0], method='BDF')
steady_state.append(sol.y[0][-1])
This gives me a RuntimeWarning:
ETA: --:--:--/anaconda3/lib/python3.6/site-packages/scipy/integrate/_ivp/bdf.py:418:
RuntimeWarning: divide by zero encountered in power
factors = error_norms ** (-1 / np.arange(order, order + 3))
But even worse, the run basically freezes (or at least gets incredibly slow). Replacing the initial value [0] by [1e-08] does not solve the problem. How can I fix this?

You can use NumbaLSODA: https://github.com/Nicholaswogan/NumbaLSODA . Its like solve_ivp, but all the code can be compiled. So it is very speedy:
from NumbaLSODA import lsoda_sig, lsoda
import numpy as np
import numba as nb
import time
#nb.cfunc(lsoda_sig)
def test_ode(t, y_, dydt, p):
y = y_[0]
C = p[0]
dydt[0] = C - y + (y ** 8 / (1 + y ** 8))
funcptr = test_ode.address
#nb.njit()
def main():
steady_state = np.empty((1001,),np.float64)
CC = np.linspace(0, 1, 1001)
y0 = np.array([0.0])
for i in range(len(CC)):
data = np.array([CC[i]],np.float64)
t_eval = np.array([0.0, 1.0e6])
sol, success = lsoda(funcptr, y0, t_eval, data = data)
steady_state[i] = sol[-1,0]
return steady_state
main()
start = time.time()
steady_state = main()
end = time.time()
print(end-start)
result is 0.013 seconds