Does the SciPy implementation of the differential evolution algorithm have a maximum number of variables? My code works on a toy version of the problem with 8 variables, but when I try to optimize the actual problem with 4000 variables a value of infinity is consistently returned for the objective function.
code (see GitHub repo for input files)
import numpy as np
from scipy.optimize import differential_evolution as de
from scipy.optimize import NonlinearConstraint as nlc
def kf(x, w, freq):
kc = x>0
kw = ~np.any(w[~kc,:], axis=0)
return -freq[kw].sum()
def cons_fun(x):
return np.sum(x>0)
def optimize(w, freq):
cons = nlc(cons_fun, -np.inf, 1000)
bnds = [np.array([-1,1]),]*w.shape[0]
res = de(kf, args=(w, freq), maxiter=1000, bounds=bnds, popsize=2, polish=False,
constraints=cons, disp=True, workers=-1, updating='deferred')
output = res.x>0
np.save('output.npy', output)
if __name__ == '__main__':
# try optimizing toy version of problem
small_w = np.load('small_w.npy')
small_freq = np.load('small_freq.npy')
optimize(small_w, small_freq)
# try optimizing actual problem
w = np.load('w.npy')
freq = np.load('freq.npy')
optimize(w, freq)
program output for the actual problem
differential_evolution step 1: f(x)= inf
differential_evolution step 2: f(x)= inf
differential_evolution step 3: f(x)= inf
...and so on for hundreds of steps
more information about the optimization problem
I'm trying to determine a set of 1000 Chinese characters that maximizes your ability to write common words. The array w is a sparse boolean matrix with shape 4000 (number of potential characters) by 30000 (number of words). An element of w is true if the character corresponding to that row occurs in the word corresponding to that column. The array freq is a vector of length 30000 that contains word frequency values.
The objective function kf takes 4000-element array x as its argument. The array x contains values between -1 and 1. The trial set of characters is determined by the positive elements in x. A nonlinear constraint restricts the number of positive elements in x to 1000.
There is no limit to the number of variables that can be used in differential_evolution.
For a constrained minimization with differential_evolution the objective function is only evaluated if the constraints are feasible. This is so that computational time is not wasted on trial solutions.
A trial solution is accepted if:
* it satisfies all constraints and provides a lower or equal objective
function value, while both the compared solutions are feasible
- or -
* it is feasible while the original solution is infeasible,
- or -
* it is infeasible, but provides a lower or equal constraint violation
for all constraint functions.
Have you investigated your constraints function to check that it is possible to create a feasible solution within the bounds?
Related
I am looking for an efficient way to find local mins for multiple (>1 million) but independent 4th order polynomials in given/ specified ranges/ boundaries.
I have two requirements:
R1: efficient even for 1 million different polynomial equations
R2: the local min is accurate up to 0.01 (i.e. 2dp)
Here is some code I have created using scipy. It's okay but I am wondering if there's any other better packages in performing such a task before I go for parallel programming.
To illustrate my problem, let's start with one polynomial first:
Below I am trying to find the local min of 4x^4 + 6x^3 + 3x^2 + x + 5 within the range (-5, 5).
On my laptop, it takes about 2ms to find the local min (which is at ~ -0.72770502).
The time is alright for one polynomial but I would want something faster as I need to perform this operation over 1 million times regularly.
from scipy import optimize
import numpy as np
# Define a objective and gradient function for 4th order polynomial
# x is the value to be evaluated
# par is a numpy array of len 5 that specifies the polynomial coefficients.
def obj_grad_fun_custom(x,par):
obj = (np.array([x**4,x**3,x**2,x**1,1]) * par).sum()
grad = (np.array([4*x**3,3*x**2,2*x,1]) * par[:-1]).sum()
return obj, grad
# Try minimise an example polynomial of 4x^4 + 6x^3 + 3x^2 + x + 5
# with contrainted bound
res = optimize.minimize(
fun = obj_grad_fun_custom,
x0 = 0,
args=(np.array([4,6,3,1,5])), # polynomial coefficients
jac=True ,
bounds=[(-2, 10)],
tol=1e-10)
print(res.x)
# Timing (this takes about 2 ms for me)
%timeit optimize.minimize(fun = obj_grad_fun_custom, x0 = 0, args=(np.array([4,6,3,1,5])), jac=True, bounds=[(-5, 5)], tol=1e-10)
Below is what I am planning to do regular with 1 million different order 4 polynomials I would want to minimise locally. Hopefully, someone could point me to a more suitable package other than scipy. Or any alternative methods? Thanks!
# Multiple polynomials
result = [] # saving the local minima
poly_sim_no = 1000000 #ideally 1 million or even more
np.random.seed(0)
par_set = np.random.choice(np.arange(10), size=(poly_sim_no, 5), replace=True) #generate some order 4 polynomial coefficients
for a in par_set:
res = optimize.minimize(obj_grad_fun_custom, 0,args=(a), jac=True ,bounds=[(-5, 5)], tol=1e-10)
result.append(res.x)
print(result)
Since you're finding the minimum of a polynomial, you can take advantage of the fact that it's easy to take the derivative of a polynomial, and that there are many good algorithms for finding the roots of a polynomial.
Here's how it works:
First, take the derivative. All of the points which are minimums will have a derivative of zero.
Look for those zeros, aka find the roots of the derivative.
Once we have the list of candidates, check that the solutions are real.
Check that the solutions are within the bounds you set. (I don't know if you added bounds because you actually want the bounds, or to make it go faster. If it's the latter, feel free to remove this step.)
Actually evaluate the candidates with the polynomial and find the smallest one.
Here's the code:
import numpy as np
from numpy.polynomial import Polynomial
def find_extrema(poly, bounds):
deriv = poly.deriv()
extrema = deriv.roots()
# Filter out complex roots
extrema = extrema[np.isreal(extrema)]
# Get real part of root
extrema = np.real(extrema)
# Apply bounds check
lb, ub = bounds
extrema = extrema[(lb <= extrema) & (extrema <= ub)]
return extrema
def find_minimum(poly, bounds):
extrema = find_extrema(poly, bounds)
# Note: initially I tried taking the 2nd derivative to filter out local maxima.
# This ended up being slower than just evaluating the function.
# Either bound could end up being the minimum. Check those too.
extrema = np.concatenate((extrema, bounds))
# Check every candidate by evaluating the polynomial at each possible minimum,
# and picking the minimum.
value_at_extrema = poly(extrema)
minimum_index = np.argmin(value_at_extrema)
return extrema[minimum_index]
# Warning: polynomial expects coeffients in the opposite order that you use.
poly = Polynomial([5,1,3,6,4])
print(find_minimum(poly, (-5, 5)))
This takes 162 microseconds on my computer, making it about 6x faster than the scipy.optimize solution. (The solution shown in the question takes 1.12 ms on my computer.)
Edit: A faster alternative
Here's a faster approach. However, it abandons bounds checking, uses a deprecated API, and is generally harder to read.
p = np.poly1d([4,6,3,1,5]) # Note: polynomials are opposite order of before
def find_minimum2(poly):
roots = np.real(np.roots(poly.deriv()))
return roots[np.argmin(poly(roots))]
print(find_minimum2(p))
This clocks in at 110 microseconds, making it roughly 10x faster than the original.
I'm facing a problem while trying to implement the coupled differential equation below (also known as single-mode coupling equation) in Python 3.8.3. As for the solver, I am using Scipy's function scipy.integrate.solve_bvp, whose documentation can be read here. I want to solve the equations in the complex domain, for different values of the propagation axis (z) and different values of beta (beta_analysis).
The problem is that it is extremely slow (not manageable) compared with an equivalent implementation in Matlab using the functions bvp4c, bvpinit and bvpset. Evaluating the first few iterations of both executions, they return the same result, except for the resulting mesh which is a lot greater in the case of Scipy. The mesh sometimes even saturates to the maximum value.
The equation to be solved is shown here below, along with the boundary conditions function.
import h5py
import numpy as np
from scipy import integrate
def coupling_equation(z_mesh, a):
ka_z = k # Global
z_a = z # Global
a_p = np.empty_like(a).astype(complex)
for idx, z_i in enumerate(z_mesh):
beta_zf_i = np.interp(z_i, z_a, beta_zf) # Get beta at the desired point of the mesh
ka_z_i = np.interp(z_i, z_a, ka_z) # Get ka at the desired point of the mesh
coupling_matrix = np.empty((2, 2), complex)
coupling_matrix[0] = [-1j * beta_zf_i, ka_z_i]
coupling_matrix[1] = [ka_z_i, 1j * beta_zf_i]
a_p[:, idx] = np.matmul(coupling_matrix, a[:, idx]) # Solve the coupling matrix
return a_p
def boundary_conditions(a_a, a_b):
return np.hstack(((a_a[0]-1), a_b[1]))
Moreover, I couldn't find a way to pass k, z and beta_zf as arguments of the function coupling_equation, given that the fun argument of the solve_bpv function must be a callable with the parameters (x, y). My approach is to define some global variables, but I would appreciate any help on this too if there is a better solution.
The analysis function which I am trying to code is:
def analysis(k, z, beta_analysis, max_mesh):
s11_analysis = np.empty_like(beta_analysis, dtype=complex)
s21_analysis = np.empty_like(beta_analysis, dtype=complex)
initial_mesh = np.linspace(z[0], z[-1], 10) # Initial mesh of 10 samples along L
mesh = initial_mesh
# a_init must be complex in order to solve the problem in a complex domain
a_init = np.vstack((np.ones(np.size(initial_mesh)).astype(complex),
np.zeros(np.size(initial_mesh)).astype(complex)))
for idx, beta in enumerate(beta_analysis):
print(f"Iteration {idx}: beta_analysis = {beta}")
global beta_zf
beta_zf = beta * np.ones(len(z)) # Global variable so as to use it in coupling_equation(x, y)
a = integrate.solve_bvp(fun=coupling_equation,
bc=boundary_conditions,
x=mesh,
y=a_init,
max_nodes=max_mesh,
verbose=1)
# mesh = a.x # Mesh for the next iteration
# a_init = a.y # Initial guess for the next iteration, corresponding to the current solution
s11_analysis[idx] = a.y[1][0]
s21_analysis[idx] = a.y[0][-1]
return s11_analysis, s21_analysis
I suspect that the problem has something to do with the initial guess that is being passed to the different iterations (see commented lines inside the loop in the analysis function). I try to set the solution of an iteration as the initial guess for the following (which must reduce the time needed for the solver), but it is even slower, which I don't understand. Maybe I missed something, because it is my first time trying to solve differential equations.
The parameters used for the execution are the following:
f2 = h5py.File(r'path/to/file', 'r')
k = np.array(f2['k']).squeeze()
z = np.array(f2['z']).squeeze()
f2.close()
analysis_points = 501
max_mesh = 1e6
beta_0 = 3e2;
beta_low = 0; # Lower value of the frequency for the analysis
beta_up = beta_0; # Upper value of the frequency for the analysis
beta_analysis = np.linspace(beta_low, beta_up, analysis_points);
s11_analysis, s21_analysis = analysis(k, z, beta_analysis, max_mesh)
Any ideas on how to improve the performance of these functions? Thank you all in advance, and sorry if the question is not well-formulated, I accept any suggestions about this.
Edit: Added some information about performance and sizing of the problem.
In practice, I can't find a relation that determines de number of times coupling_equation is called. It must be a matter of the internal operation of the solver. I checked the number of callings in one iteration by printing a line, and it happened in 133 ocasions (this was one of the fastests). This must be multiplied by the number of iterations of beta. For the analyzed one, the solver returned this:
Solved in 11 iterations, number of nodes 529.
Maximum relative residual: 9.99e-04
Maximum boundary residual: 0.00e+00
The shapes of a and z_mesh are correlated, since z_mesh is a vector whose length corresponds with the size of the mesh, recalculated by the solver each time it calls coupling_equation. Given that a contains the amplitudes of the progressive and regressive waves at each point of z_mesh, the shape of a is (2, len(z_mesh)).
In terms of computation times, I only managed to achieve 19 iterations in about 2 hours with Python. In this case, the initial iterations were faster, but they start to take more time as their mesh grows, until the point that the mesh saturates to the maximum allowed value. I think this is because of the value of the input coupling coefficients in that point, because it also happens when no loop in beta_analysisis executed (just the solve_bvp function for the intermediate value of beta). Instead, Matlab managed to return a solution for the entire problem in just 6 minutes, aproximately. If I pass the result of the last iteration as initial_guess (commented lines in the analysis function, the mesh overflows even faster and it is impossible to get more than a couple iterations.
Based on semi-random inputs, we can see that max_mesh is sometimes reached. This means that coupling_equation can be called with a quite big z_mesh and a arrays. The problem is that coupling_equation contains a slow pure-Python loop iterating on each column of the arrays. You can speed the computation up a lot using Numpy vectorization. Here is an implementation:
def coupling_equation_fast(z_mesh, a):
ka_z = k # Global
z_a = z # Global
a_p = np.empty(a.shape, dtype=np.complex128)
beta_zf_i = np.interp(z_mesh, z_a, beta_zf) # Get beta at the desired point of the mesh
ka_z_i = np.interp(z_mesh, z_a, ka_z) # Get ka at the desired point of the mesh
# Fast manual matrix multiplication
a_p[0] = (-1j * beta_zf_i) * a[0] + ka_z_i * a[1]
a_p[1] = ka_z_i * a[0] + (1j * beta_zf_i) * a[1]
return a_p
This code provides a similar output with semi-random inputs compared to the original implementation but is roughly 20 times faster on my machine.
Furthermore, I do not know if max_mesh happens to be big with your inputs too and even if this is normal/intended. It may make sense to decrease the value of max_mesh in order to reduce the execution time even more.
I am evaluating a set of ODEs with time varying coefficients
def deriv(y, t, N, coefficients):
S, I, R = y
dSdt = coefficients['beta'](t) * S * I / N * -1
dIdt = coefficients['beta'](t) * S * I / N - coefficients['gamma']* I
dRdt = coefficients['gamma'] * I
return dSdt, dIdt, dRdt
In particular, I have 'beta' values in a pre-calculated array, of size equal to int(max(t)).
coefficients = {'beta' : beta_f,'gamma':0.1}
def beta_f(t):
return mybetas.iloc[int(t)]
# Initial conditions vector
y0 = (S0, I0, R0)
# Integrate the SIR equations over the time grid, t.
ret = odeint(deriv,y0,t,args=(N,coefficients))
When I run odeint, it is evaluate also for value beyond max(t), raising an index out of bound error in beta_f.
How to limit the evaluation span for odeint?
Since len(mybetas) == int(max(t)), you can get an out-of-bounds error even for values of t which are not beyond max(t).
For example, mybetas.iloc[int(max(t))] will give you the out-of-bounds error, even though int(max(t)) <= max(t) for positive values of t.
But to your point, odeint does indeed check some values outside of the domain of integration. I had to deal with a problem similar to yours just a few weeks ago, and the following two discussions on stackoverflow were really helpful:
integrate.ode sets t0 values outside of my data range
Solve ODEs with discontinuous input/forcing data
The second link explains why it might be computationally faster to solve the ODE with odeint over each individual integer time step one after the other in a for loop, instead of letting odeint deal with the discontinuities in your derivative caused by jumps in the values of your betas.
Otherwise, if this is appropriate for your study case, you can interpolate your betas, and let the function beta_f return interpolated values of beta. Of course, you will have to extend the interpolation domain slightly beyond your integration domain, since odeint might want to evaluate the derivative for some t larger than max(t).
The problem at hand is optimization of multivariate function with nonlinear constraints.
There is a differential equation (in its oversimplified form)
dy/dx = y(x)*t(x) + g(x)
I need to minimize the solution of the DE y(x), but by varying the t(x).
Since it is physics under the hood, there are constraints on t(x). I successfully implemented all of them except one:
0 < t(x) < 1 for any x in range [a,b]
For certainty, the t(x) is a general polynomial:
t(x) = a0 + a1*x + a2*x**2 + a3*x**3 + a4*x**4 + a5*x**5
The x is fixed numpy.ndarray of floats and the optimization goes for coefficients a. I use scipy.optimize with trust-constr.
What I have tried so far:
Root finding at each step and determining the minimal/maximal value of the function using optimize.root and checking for sign changes. Return 0.5 if constraints are satisfied and numpy.inf or -1 or whatever not in [0;1] range if constraints are not satisfied. The optimizer stops soon and the function is not minimized properly.
Since x is fixed-length and known, I tried to define a constraint for each point, so I got N constraints where N = len(x). This works (at least look like) but takes forever for not-so large N. Also, since x is discrete and non-uniform, I can't be sure that there are no violated constraints for any x in [a,b].
EDIT #1: the minimal reproducible example
import scipy.optimize as optimize
from scipy.optimize import Bounds
import numpy as np
# some function y(x)
x = np.linspace(-np.pi,np.pi,100)
y = np.sin(x)
# polynomial t(z)
def t(a,z):
v = 0.0;
for ii in range(len(a)):
v += a[ii]*z**ii
return v
# let's minimize the sum
def targetFn(a):
return np.sum(y*t(a,x))
# polynomial order
polyord = 3
# simple bounds to have reliable results,
# otherwise the solution will grow toward +-infinity
bnd = 10.0
bounds = Bounds([-bnd for i in range(polyord+1)],
[bnd for i in range(polyord+1)])
res = optimize.minimize(targetFn, [1.0 for i in range(polyord+1)],
bounds = bounds)
if np.max(t(res.x,x))>200:
print('max constraint violated!')
if np.min(t(res.x,x))<-100:
print('min constraint violated!')
In the reproducible example given above, let the constraints to be that the value of the polynomial t(a,x) is in range [-100;200] for the given x.
So the question is: how does one properly define a constraint to tell the optimizer that the function's values must be constrained for the given range of arguments?
How do I set all the negative value of a variable to zero in an equation. I've tried using m.max2(0,W) so when W is negative I get zero but when it's positive I get the value but since W is defined as W.dt()==s*p the value seems to be trailing I cant' set the lower bound to zero because I need the negative value elsewhere.
The max2 function is a Mathematical Program with Complementarity Constraints (MPCC) and sometimes has a hard time converging when at W=0 because it is a saddle point for optimization. Another option is the max3 function but this requires a mixed integer solver that can require more time to compute a solution. A third option is to use a function such as w * (0.5*tanh(b*w)+0.5) to get a continuously differentiable approximation to the max function. You can set the b to be a higher value but then it makes the problem harder to solve.
Another option is to successively solve the problem with higher values of b, like a barrier function in the interior point method.
Here is an example script that has all three functions:
import numpy as np
from gekko import GEKKO
m = GEKKO()
w = m.Param(np.linspace(-10,10,101))
x = m.Intermediate(w * 0.5*(m.tanh(10*w)+1))
y = m.max2(w,0)
z = m.max3(w,0)
m.options.IMODE=2
m.solve()
import matplotlib.pyplot as plt
plt.plot(w,x,'ko',label='x=0.5 w (tanh(10w)+1)')
plt.plot(w,y,'b-',label='y=min2(w,0)')
plt.plot(w,z,'r--',label='z=min3(w,0)')
plt.legend()
plt.show()