I am working with a fairly complex objective function that I am minimizing by varying 4 parameters. A while ago I decided to use the Python framework Mystic, which seamlessly allows me to use penalties for complex inequalities (which I need).
However, Mystic has a less-than-obvious way to assign hard constraints (not inequalities and not bound constraints, linear inequalities between parameters only) and even less obvious way to handle them.
All my 4 parameters have finite lower and upper bounds. I would like to add a linear inequality as a hard constraint like this:
def constraint(x): # needs to be <= 0
return x[0] - 3.0*x[2]
But if I try to use Mystic in this way:
from mystic.solvers import fmin_powell
xopt = fmin_powell(OF, x0=x0, bounds=bounds, constraints=constraint)
Then Mystic insists in calling the objective function to resolve the constraints first and then proceed with the actual optimization; since the objective function value has no impact nor any effect on the constraint function as defined above then I am not sure why this is happening. The constraint function defined above simply tells Mystic that a region of the hyperparameters search space should be off limits.
I have scoured pretty much all the examples in the Mystic folder and I stumbled across an alternative way to define a hard constraint: use a penalty function and then call a magic method "as_constraint" to "convert it" to a constraint. Unfortunately, all those examples go pretty much this way:
from mystic.solvers import fmin_powell
from mystic.constraints import as_constraint
from mystic.penalty import quadratic_inequality
def penalty_function(x): # <= 0.0
return x[0] - 3.0*x[2]
#quadratic_inequality(penalty_function)
def penalty(x):
return 0.0
solver = as_constraint(penalty)
result = fmin_powell(OF, x0=x0, bounds=bounds, penalty=penalty)
There is this magic line:
solver = as_constraint(penalty)
That I can't see what it's doing - the solver variable is never used again.
So, for the question: is there any way to define linear inequalities in Mystic that do not involve an expensive pre-solve of the constraints but simply tell Mystic to exclude certain regions of the search space?
Thank you in advance for any suggestion.
Andrea.
What mystic does is map the space it searches, so you are optimizing over a "kernel transformed" space (to use machine learning jargon). You can think of the constraints as applying an operator, if you know that that means. So, y = f(x) under some constraints x' = c(x) becomes y = f(c(x)). This is why the optimizer evaluates the constraints before evaluating the objective.
So you can build a constraint like this:
>>> import mystic.symbolic as ms
>>> equation = 'x1 - 3*a*x2 <= 0'
>>> eqn = ms.simplify(equation, locals=dict(a=1), all=True)
>>> print(eqn)
x1 <= 3*x2
>>> c = ms.generate_constraint(ms.generate_solvers(eqn, nvars=3))
>>> c([1,2,3])
[1, 2, 3]
>>> c([0,100,-100])
[0, -300.0, -100]
Or if you have more than one:
>>> equation = '''
... x1 > x2 * x0
... x0 + x1 < 10
... x1 + x2 > 5
... '''
>>> eqn = ms.simplify(equation, all=True)
>>> print(eqn)
x1 > -x2 + 5
x0 < -x1 + 10
x1 > x0*x2
>>> import mystic.constraints as mc
>>> c = ms.generate_constraint(ms.generate_solvers(eqn), join=mc.and_)
>>> c([1,2,3])
[1, 3.000000000000004, 3]
Related
I am trying to find a solution to the following system where f and g are R^2 -> R^2 functions:
f(x1,x2) = (y1,y2)
g(y1,y2) = (x1,x2)
I tried solving it using scipy.optimize.fsolve as follows:
def eqm(vars):
x1,x2,y1,y2 = vars
eq1 = f([x1, x2])[0] - y1
eq2 = f([x1, x2])[1] - y2
eq3 = g([y1, y2])[0] - x1
eq4 = g([y1, y2])[1] - x2
return [eq1, eq2, eq3, eq4]
fsolve(eqm, x0 = [1,0.5,1,0.5])
Although it is returning an output, it does not seem to be a correct one as it does not seem to satisfy the two conditions, and seems to vary a lot with the x0 specified. Also getting a warning:
'The iteration is not making good progress, as measured by the improvement from the last ten iterations.' I do know for a fact that a unique solution exists, which I have obtained algebraically.
Not sure what is going on and if there is a simpler way of solving it, especially using just two equations instead of splitting up into 4. Something like:
def equations(vars):
X,Y = vars
eq1 = f(X)-Y
eq2 = g(Y)-X
return [eq1, eq2]
fsolve(equations, x0 =[[1,0.5],[1,0.5]])
Suggestions on other modules e.g. sympy are also welcome!
First, I recommend working with numpy arrays since manipulating these is simpler than lists.
I've slighlty rewritten your code:
import scipy.optimize as opt
def f(x):
return x
def g(x):
return x
def func(vars):
input = np.array(vars)
eq1 = f(input[:2]) - input[2:]
eq2 = g(input[2:]) - input[:2]
return np.concatenate([eq1, eq2])
root = opt.fsolve(func, [1, 1, 0., 1.2])
print(root)
print(func(root)) # should be close to zeros
What you have should work correctly, so I believe there is something wrong with the equations you're using. If you provide those, I can try to see what may be wrong.
This seems to be more of a problem of numerical mathematics than Python coding. Your functions may have "ugly" behavior around the solution, may be strongly non-linear or contain singularities. We cannot help further without seeing the functions. One thing you might try is to instead solve a system
g(f(x)) - x = 0
and simplify g(f(x)) as much as possible analytically. Then calculate y = f(x) after solving the equation.
I'm a little new to optimization in Python, particularly on Scipy. If I only have one function such as this:
def prod(x):
return -x[0] * x[1]
Then I can easily know how to do maximization through:
linear_constraint = LinearConstraint([[1,1]], [20], [20])
x0 = np.array([15,5])
result = minimize(prod, x0, constraints=linear_constraint)
print(result['x'])
This is basically maximizing x0 * x1 given the constraint that x0 + x1 = 20. The answer here is array([10., 10.]).
So now, I wish to do something more sophisticated and not just optimizing one function. Suppose I would like to optimize multiple functions. Say I have this matrix P of size n x 3:
A B C
3 2 7
6 3 4
8 1 5
...
Is there a way to optimize (minimize) P[i] * x, where * here is a dot product for all i = 1..n? In short, I wish to optimize:
3*x[0] + 2*x[1] + 7*x[2]
6*x[0] + 3*x[1] + 4*x[2]
8*x[0] + 1*x[1] + 5*x[2]
...
and under the constraint that x[0] + x[1] + x[2] = 1. Anyone knows how to implement this correctly on Python (I'm using Scipy by the way)? I'm still not sure where to begin. Thanks in advance!
For standard optimization tools, the objective function must return a scalar. This is called a "single objective". There is a branch in optimization that deals with multiple objectives (with names like vector optimization, multiple objective optimization and multiple criteria optimization). There is a wide literature about this. I suggest doing some reading to get familiar with the concepts, ideas and algorithms for this.
The problem at hand is optimization of multivariate function with nonlinear constraints.
There is a differential equation (in its oversimplified form)
dy/dx = y(x)*t(x) + g(x)
I need to minimize the solution of the DE y(x), but by varying the t(x).
Since it is physics under the hood, there are constraints on t(x). I successfully implemented all of them except one:
0 < t(x) < 1 for any x in range [a,b]
For certainty, the t(x) is a general polynomial:
t(x) = a0 + a1*x + a2*x**2 + a3*x**3 + a4*x**4 + a5*x**5
The x is fixed numpy.ndarray of floats and the optimization goes for coefficients a. I use scipy.optimize with trust-constr.
What I have tried so far:
Root finding at each step and determining the minimal/maximal value of the function using optimize.root and checking for sign changes. Return 0.5 if constraints are satisfied and numpy.inf or -1 or whatever not in [0;1] range if constraints are not satisfied. The optimizer stops soon and the function is not minimized properly.
Since x is fixed-length and known, I tried to define a constraint for each point, so I got N constraints where N = len(x). This works (at least look like) but takes forever for not-so large N. Also, since x is discrete and non-uniform, I can't be sure that there are no violated constraints for any x in [a,b].
EDIT #1: the minimal reproducible example
import scipy.optimize as optimize
from scipy.optimize import Bounds
import numpy as np
# some function y(x)
x = np.linspace(-np.pi,np.pi,100)
y = np.sin(x)
# polynomial t(z)
def t(a,z):
v = 0.0;
for ii in range(len(a)):
v += a[ii]*z**ii
return v
# let's minimize the sum
def targetFn(a):
return np.sum(y*t(a,x))
# polynomial order
polyord = 3
# simple bounds to have reliable results,
# otherwise the solution will grow toward +-infinity
bnd = 10.0
bounds = Bounds([-bnd for i in range(polyord+1)],
[bnd for i in range(polyord+1)])
res = optimize.minimize(targetFn, [1.0 for i in range(polyord+1)],
bounds = bounds)
if np.max(t(res.x,x))>200:
print('max constraint violated!')
if np.min(t(res.x,x))<-100:
print('min constraint violated!')
In the reproducible example given above, let the constraints to be that the value of the polynomial t(a,x) is in range [-100;200] for the given x.
So the question is: how does one properly define a constraint to tell the optimizer that the function's values must be constrained for the given range of arguments?
I am using pyswarm, I wonder if anyone has an idea how to change the setting to get only integer numbers for the swarm particles (x1, x2)!
from pyswarm import pso
def banana(x):
x1 = x[0] # Get only integer number
x2 = x[1] # Get only integer number
return x1**4 - 2*x2*x1**2 + x2**2 + x1**2 - 2*x1 + 5
def con(x):
x1 = x[0]
x2 = x[1]
return [-(x1 + 0.25)**2 + 0.75*x2]
lb = [-3, -1]
ub = [2, 6]
xopt, fopt = pso(banana, lb, ub, f_ieqcons=con)
Facing the same problem today, I looked a bit on the code and found that this behavior is not handled at all. Also, there is no activity on the repository since 2015...
Moreover, it's not a good idea to map the initial real domain space to an integer space: all solutions between two integers will be computed with the same parameters.
Changing the objective function arguments implies you define an "error" value returned for non integers parameters (some kind of extreme value penalized by the optimization problem). But you will probably face with local optima this way.
So finally you might want to search for other libraries that implement PSO, or even try an implementation on the actual pyswarm code to support custom domains for particles.
EDIT:
Here is a paper about PSO for integer programming: http://www.cs.uoi.gr/~kostasp/papers/C03.pdf
I have a timeseries that I want to fit to function using Scipy.optimize.leastsq.
fitfunc= lambda a, x: a[0]+a[1]*exp(-x/a[4])+a[2]*exp(-x/a[5])+a[3]*exp(-x /a[6])
errfunc lambda a,x,y: fitfunc(a,x) - y
Next I would pass errfunc to leastsq to minimze it. The fit-function I use is a sum of exponentials decaying with different timescales a(4:6) and different weights (a(0:4)). (as a sideuqestion: can I use leastsq with more than 1 parameter arrays? I didn't suceed to do so....)
The question: How can I put additional side conditions on the parameters entering the fit-function. I want for example that sum(a(0:4))=1.0
Just use
import numpy as np
def fitfunc(p, x):
a = np.zeros(7)
a[1:7] = p[:6]
a[0] = 1 - a[1:4].sum()
return a[0] + a[1]*exp(-x/a[4]) + a[2]*exp(-x/a[5]) + a[3]*exp(-x/a[6])
def errfunc(p, x, y1, y2):
return np.concatenate((
fitfunc(p[:6], x) - y1,
fitfunc(p[6:12], x) - y2
))
Generally, lambda-functions are considered bad style (and they don't add anything in your code). To have several functions in the least square fit you may just append the functions as I indicated using np.concatenate. It doesn't make much sense, if none of the parameters are correlated, though. It will only slow down convergence of the algorithm. The side condition you asked for, is implemented by just calculating one weight based on the constraint you gave (see 1 - a[1:4].sum()).
If you can't solve the equations for you constraints, and you can live with the constraint being satisfied with some tolerance, another possibility is to add a term to the chi-square with the large weight which guarantees that the constraint is almost satisfied.
E.g if you need that \sum(sin(p[i])==1 ,you can do the following:
constraint_func = lambda a: sin(a).sum()-1
def fitfunc (a,x):
np.concatenate((a[0]+a[1]*exp(-x/a[4])+a[2]*exp(-x/a[5])+a[3]*exp(-x /a[6]),
[constraint_func(a)]))
def errfunc(a,x,y):
tolerance = 1e-10
return np.concatenate((fitfunc(a,x) - y, [tolerance]))
Obviously the convergence will be slower, but will be still guaranteed.