Develop Reference

Problem while looping variables for Scipy Optimization (SLSQP, COBYLA)

I have a constrained optimization problem where I am trying to minimize an objective function of 100+ variables which is of the form
Min F(x) = f(x1) + f(x2) + ... + f(xn)
Subject to functional constraint
(g(x1) + g(x2) + ... + g(xn))/(f(x1) + f(x2) + ... + f(xn)) - constant >= 0
I also have individual bounds for each variable x1, x2, x3...xn
a <= x1 <= b
c <= x2 <= d
...
For this, I wrote a python script, using the scipy.optimize.minimize implementation with constraints and bounds, but I am unable to fulfill my bounds and constraints in the solutions. These are all cases where optimization could converge to a solution (message: success)
Here is a sample of my code:
df is my pandas dataset
B(x) is LogNorm transform based on x and other constants
Values U, c, lb, ub are pre-calculated constant dictionaries for each index in df
import scipy
df = pd.DataFrame(..)
k = set(df.index.values) ## list of indexes to iterate on
val = 0.25 ## Arbitrary
def obj(x):
fn = 0
for n,i in enumerate(k):
x0 = x[n]
fn1 = (U[i]) * B(x0) * (x0)
fn += fn1
return fn
def cons(x):
cn = 1
c1 = 0
c2 = 0
for n,i in enumerate(k):
x0 = x[n]
c1 += (U[i]) * (B(x0) * (x0 - c[i])
c2 += (U[i]) * (B(x0) * (x0)
cn = c1/(c2)
return cn - val
const = [{'type':'ineq', 'fun':cons}]
bnds = tuple((lb[i], ub[i]) for i in k) ## Lower, Upper for each element ((lb1, ub1), (lb2, ub2)...)
x_init = [lb[i] for i in k] ## for eg. starting from lower bound
## Solution
sol = scipy.optimize.minimize(obj, x_init, method = 'COBYLA', bounds = bnds, constraints = const)
I have more pointed questions if that helps:
Is there a way to construct the same equation concisely/ without the use of loops (given the number of variables could depend on input data and I have no control over it)?
Is there any noticeable issue in my application of bounds? I can't seem to get the final values of all variables follow individual bounds.
Similarly, is there a visible flaw in the construction on constraint equation? My results often DO NOT follow the constraints is repeated runs with different inputs.
Any help with either of the questions can help me progress further at work.
I have also looked into a Lagrangian solution of the same but so far I am unable to solve it for undefined number of (n) variables.
Thanks!

Why does Python RK23 Solver blow up and give unrealistic results?

I am experimenting with RK45/RK23 solver from python scipy module. Using it to solve simple ODEs and it is not giving me the correct results. When I manually code for Runge Kutta 4th order it works perfectly so does the odeint solver in the module but RK23/RK45 does not. If someone could aid me in figuring out the issue it would be help full. I have so far implemented only simple ODEs
dydt = -K*(y^2)
Code:
import numpy as np
from scipy.integrate import solve_ivp,RK45,odeint
import matplotlib.pyplot as plt
# function model fun(y,t)
def model(y,t):
k = 0.3
dydt = -k*(y**2)
return dydt
# intial condition
y0 = np.array([0.5])
y = np.zeros(100)
t = np.linspace(0,20)
t_span = [0,20]
#RK45 implementation
yr = solve_ivp(fun=model,t_span=t_span,y0=y,t_eval=t,method=RK45)
##odeint solver
yy = odeint(func=model,y0=y0,t=t)
##manual implementation
t1 = 0
h = 0.05
y = np.zeros(21)
y[0]=y0;
i=0
k=0.3
##Runge Kutta 4th order implementation
while (t1<1.):
m1 = -k*(y[i]**2)
y1 = y[i]+ m1*h/2
m2 = -k*(y1**2)
y2 = y1 + m2*h/2
m3 = -k*(y2**2)
y3 = y2 + m3*h/2
m4 = -k*(y3**2)
i=i+1
y[i] = y[i-1] + (m1 + 2*m2 + 2*m3 + m4)/6
t1 = t1 + h
#plotting
t2 = np.linspace(0,20,num=21)
plt.plot(t2,y,'r-',label='RK4')
plt.plot(t,yy,'b--',label='odeint')
#plt.plot(t2,yr.y[0],'g:',label='RK45')
plt.xlabel('time')
plt.ylabel('y(t)')
plt.legend()
plt.show()
Output: (without showing RK45 result)
enter image description here
Output: (with just RK45 plot displayed)
enter image description here
I cant find out where am I making a mistake

Okay I have figured out the solution. RK45 requires function definition to be like fun(t,y) and odeint requires it to be func(y,t) due to which giving them same function will result in different results.

How to add skew to StepModel

I have a model made up of a sum of several lmfit StepModel(form='erf') steps, which gives a pretty good fit to my data. However, the residual shows some skew at each step. I can use skew(output.result.residual) to give me the skew over the entire dataset, but what I would really like is to have a value of skew for each component of the model, that is, a skew1 associated with step1, skew2 with step2, etc.
I've changed the code in examples/doc_builtinmodels_stepmodel.py to give an example of what I mean:
import matplotlib.pyplot as plt
import numpy as np
from lmfit.models import LinearModel, StepModel
x = np.linspace(0, 10, 201)
a = np.ones_like(x)
b = np.ones_like(x)
a[:48] = 0.0
a[48:77] = np.arange(77-48)/(77.0-48)
b[:10] = 0.0
b[10:39] = np.arange(39-10)/(39.0-10)
np.random.seed(0)
a = 110.2 * (a + 9e-3*np.random.randn(x.size)) + 12.0 + 2.22*x
b = 110.2 * (b + 9e-3*np.random.randn(x.size)) + 12.0 + 2.22*x
y = a + b
step_mod = StepModel(form='erf', prefix='s1_')
step2_mod = StepModel(form='erf', prefix='s2_')
line_mod = LinearModel(prefix='line_')
pars = line_mod.make_params(intercept=y.min(), slope=0)
pars += step_mod.guess(y, x=x, center=2.5)
pars += step2_mod.guess(y, x=x, center=2.5)
mod = step_mod + step2_mod + line_mod
out = mod.fit(y, pars, x=x)
out.plot(data_kws={'markersize': 1})
print(skew(out.result.residual))
I suspect I may have to build the skew directly into the StepModel() function, but I admit I don't really know how to do that.

Solve Non Linear Equations Numerically - Python

I am looking to reproduce results from a research article.
I am at a point, where I have to find the maximum value of the following equation (w), and corresponding independent variable value (k). k is my only variable.
from sympy import *
import numpy as np
import math
rho_l = 1352;
rho_g= 1.225;
sigma = 0.029;
nu = 0.02;
Q = rho_g/ rho_l;
u = 99.67;
h = 1.6e-5; # Half sheet thickness
k = Symbol('k', real=True)
t = tanh(k*h);
w1 = -2 * nu * k**2 * t ;
w2 = 4* nu**2 * k**4 * t**2;
w3 = - Q**2 * u**2 * k**2;
w4 = -(t + Q)
w5 = (- Q* u**2 * k**2 + (sigma*k**3/ rho_l));
w6 = -w4;
w = ( w1 + sqrt(w2 + w3 + w4*w5))/w6;
I was able to solve this using Sympy - diff & solve functions, only when I give t = 1 or a any constant.
Do anyone have suggestions on finding the maximum value of this function? Numerically also works - however, I am not sure about the initial guess value. Good thing is I only have one independent variable.
Edit:
As per the answers given here regarding gradient descent, and also plotting and seeing the maximum value. I literally copied the code lines, that include plotting and I got a different plot.
Any thoughts on why this is happening? I am using Python 3.7

There are a bunch of ways to do this. scipy in particular has a bunch of optimization algorithms. I'm going to use gradient descent (or, perhaps more appropriately, gradient ascent) and autograd because it might be fun.
First, let's import autograd and turn your function into a callable function.
import autograd.numpy as anp
from autograd import grad
import matplotlib.pyplot as plt
def w(k):
rho_l = 1352;
rho_g= 1.225;
sigma = 0.029;
nu = 0.02;
Q = rho_g/ rho_l;
u = 99.67;
h = 1.6e-5; # Half sheet thickness
t = anp.tanh(k*h);
w1 = -2 * nu * k**2 * t ;
w2 = 4* nu**2 * k**4 * t**2;
w3 = - Q**2 * u**2 * k**2;
w4 = -(t + Q)
w5 = (- Q* u**2 * k**2 + (sigma*k**3/ rho_l));
w6 = -w4;
w = ( w1 + anp.sqrt(w2 + w3 + w4*w5))/w6;
return w
Now, we can use autograd to compute the gradient of w with respect to k. You can add some logic to ensure that the procedure will terminate once we meet some tolerance threshold.
dwdk = grad(w)
#Do gradient descent
k = 10.0 #An initial guess
learning_rate = 1
for i in range(1000):
k+=learning_rate*dwdk(k)
And now, let's plot the result to ensure we found the maximum
K = np.arange(0,1000)
plt.plot(K,w(K))
plt.scatter(k, w(k), color = 'red')
plt.show()

My thought is that the function returning values for w (your blue line) might be a truncated estimate (with a polynomial maybe)? Is the formula for w off by a factor of 10 or so?

Here is another method. It is an implementation of the Metropolis algorithm, a so-called Markov Chain Monte Carlo method. Using the definition of w, it is possible to construct a Markov Chain of w(k) called wlist. The tail of this chain should be the maximum of w, and we can recover k that got it by storing the k values in a list called kvalues.
import math
import random
klist = [1.0]
wlist = [w(1.0)] # initialize the chain
# you can tune the value passed to `range`
for _ in range(5000):
k = random.gauss(klist[-1], 0.2*klist[-1]) # q
if k <= 0.0: # assuming max has positive `k` arg
continue
w_hat = w(k)
if w_hat > wlist[-1]:
klist.append(k)
wlist.append(w_hat)
else:
u = random.random()
try:
alpha = math.exp(-w_hat) / math.exp(-wlist[-1])
except ZeroDivisionError:
alpha = 1.0
if u >= alpha:
klist.append(k)
wlist.append(w_hat)
else:
klist.append(klist[-1])
wlist.append(wlist[-1])
wlist[-10:], klist[-10:]
Which should return approximately (my seed is not set) something like this:
([8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532,
8679.594992731532],
[416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436,
416.22335719432436])

I do not expect that there is an analytical solution to this problem. There is a theory of pfaffian functions for which it is possible to provide certificate that there are no roots for given range. See https://en.m.wikipedia.org/wiki/Pfaffian_function.
However this heavy artilery.
If you are not sure about initial guess try to compute the function for a large number of random points (e.g.million) and select the best one as starting point. This approach works really well for low dimensional, differentiable problems

Solving Vector ODE in Python

Id like to thank whoever answers this in advance, as it is probably a stupid question and a waste of time to answer.
The given code takes the electromagnetic forces on a particle and supposedly plots its trajectory, but I cant figure out how to use rk4 with vectors. Feel like my function is set up wrong.
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv, det, eig
from numpy.random import randn
def rk4(f, x0, dt):
tn = 0
xn = x0
while True:
yield tn,xn
k1 = dt*f(tn,xn)
k2 = dt*f(tn+dt/2,xn+k1/2)
k3 = dt*f(tn+dt/2,xn+k2/2)
k4 = dt*f(tn+dt,xn+k3)
xn = xn + (k1+2*k2+2*k3+k4)/6
tn = tn + dt
#--------------------------------------------------------
def f(t,X):
x,y,z,xv,yv,zv = X
v = [xv,yv,zv]
E = [k*x , k*y , -2*z]
a = (q/m)*(E+np.cross(v,B))
Xdot = np.array([v,a])
return Xdot
q , m , B , k = 1.6e-19, 40, [0,0,1], 10e+4
X0 = np.array([0.001 , 0 , 0.001 , 100 , 0 , 0])
solver = rk4(f,X0,10e-7)
ts = []
Xs = []
for t,X in solver:
ts.append(t)
Xs.append(X[0])
if t > 0.001:
break
#Xs = np.array(Xs)
#plt.figure()
#plt.plot(ts,Xs)
Id really appreciate any tips or hints
I suspect the problem stems from the way iv set up the function, as the code crashes once it tries to run it through rk4.

You can not apply vector arithmetic to simple python lists, thus convert the lists first into numpy arrays. You need to return a flat vector, not a matrix, thus use array concatenation to join the two parts.
def f(t,X):
x,y,z,xv,yv,zv = X
v = np.array([xv,yv,zv]) # or v = X[3:]
E = np.array([k*x , k*y , -2*z]) # or E=k*X[:3]; E[2]=-2*X[2]
a = (q/m)*(E+np.cross(v,B))
Xdot = np.concatenate([v,a])
return Xdot
For the last change see Concatenating two one-dimensional NumPy arrays