I am running a non-linear optimization problem in OpenMDAO, which I know the optimal solution of (I just want to verify the solution). I am using SLSQP driver configuration of ScipyOptimizer from openmdao.api.
I have 3 design variables A, B and C, their respective design-spaces (Amin to Amax for A and so on) and a single objective function Z. As I said, I know the optimal values for all the three design variables (let's call them Asol, Bsol and Csol) which yield the minimum value of Z (call it Zsol).
When I run this problem, I get a value for Z which is larger than Zsol, signifying that it is not an optimal solution. When I assign Csol to C and run the problem with only A and B as the design variables, I get the value of Z which is much closer to Zsol and which is actually lesser than what I got earlier (in 3 design variable scenario).
Why am I observing this behavior? Shouldn't ScipyOptimizer give the same solution in both the cases?
EDIT: Adding some code..
from openmdao.api import IndepVarComp, Group, Problem
from openmdao.api import ScipyOptimizer
class RootGroup(Group):
def __init__(self):
super(RootGroup, self).__init__()
self.add('desvar_f', IndepVarComp('f', 0.08))
self.add('desvar_twc', IndepVarComp('tool_wear_compensation', 0.06))
self.add('desvar_V', IndepVarComp('V', 32.0))
# Some more config (adding components, connections etc.)
class TurningProblem_singlepart(Problem):
def __init__(self):
super(TurningProblem_singlepart, self).__init__()
self.root = RootGroup()
self.driver = ScipyOptimizer()
self.driver.options['optimizer'] = 'SLSQP'
self.driver.add_desvar('desvar_f.f', lower=0.08, upper=0.28)
self.driver.add_desvar('desvar_twc.tool_wear_compensation', lower=0.0, upper=0.5)
self.driver.add_desvar('desvar_V.V', lower=32.0, upper=70.0)
self.driver.add_objective('Inverse_inst.comp_output')
# Other config
This code gives me incorrect result. When I remove desvar_twc from both the classes, and assign it with its optimal value (from the solution I have), I get fairly correct result i.e. the answer for objective function which is lesser than the previous scenario.
Without seeing your actual model, we can't say anything for sure. However, it is NOT the case that a local optimizer's solution is independent of the starting condition in general. That is only case if the problem is convex. So I would guess that your problem is not convex, and you'r running into local optima.
You can try to get around this by using the COBYLA optimizer instead of SLSQP, which in my experience can manage to jump over some local optima better. But if your problem is really bumpy, then I would suggest you switch to NSGA-II or ALPSO from the pyopt-sparse library. These are heuristic based optimizers that do a good job of finding the "biggest hill", though they don't always climb all the way to the top of it (they don't converge as tightly). The heuristic algorithms are also generally more expensive than the gradient based methods.
Related
I'm having trouble specifying constraints using basinhopping with method='COBYLA'. Here is a test case where things go wrong. Essentially, the constraints are ignored and there are function trials outside the specified range. I specify a simple quadratic with minimum at [0,0], searching for -3<x[0], but as you can see from the output, there are lots of searches outside that range (I increased the stepsize to make it obvious)
import numpy as np
from scipy.optimize import basinhopping
def f(x):
if x[0]<-3 :
print('outside range ',x[0])
return x[0]**2+x[1]**2
cons = [{'type':'ineq','fun': lambda x: x[0]+3}]
kwargs = {'method':'COBYLA','constraints':cons}
ret=basinhopping(f, [5,1],T=1,stepsize=1000,niter=1,minimizer_kwargs=kwargs)
print(ret)
runfile('py/cobyla_test', wdir='/py', post_mortem=True)
outside range -446.14581341127945
outside range -445.14581341127945
outside range -445.14581341127945
outside range -444.14581341127945
[etc... lots of output deleted]
[-4.81217825e-05 -5.23242054e-05] 5.0535284302996725e-09
As written at scipy.optimize.basinhopping — SciPy v1.1.0 Reference Guide, Basin-hopping is a two-step method:
first, a random jump is done (take_step callback)
then a local minimum is found from that point using the specificed minimization method
finally, it's decided if the step is accepted (accept_test callback)
The constraints you've specified are for the minimization method, they don't affect the jump step. For the jump step, either adjust stepsize (the maximum displacement for the random jump), or define your own take_step.
"I thought the point of the constraint is that it would never try an x outside the constraint" -- constraints in mathematical problems, including a constrained optimization problem, don't work that way. They only specify what conditions the solution itself must satisfy. They don't limit what points can be used while obtaining that solution, it's completely up to the algorithm to choose these.
The approach to limit the area in which a numerical method searches is to tweak method parameters in some way specific to the nature of the function and the method, to "guide" the method into the right direction.
I have been involved in analysis using a software called depict which includes affinity propagation analysis in Python.
I am keen to implement a counterpart using R/apcluster for additional analysis. It seems both use correlation but the results are slightly different. Is that possible to get to the bottom of this? Thanks very much.
af_obj = AffinityPropagation(affinity = 'precomputed', max_iter=10000, convergence_iter=1000) # using almost only default parameters
print "Affinity Propagation parameters:"
for param, val in af_obj.get_params().items():
print "\t{}: {}".format(param, val)
print "Perfoming Affinity Propagation.."
af = af_obj.fit(matrix_corr)
as in Python: https://github.com/jinghuazhao/PW-pipeline/blob/master/files/network_plot.py
require(apcluster)
apres <- apcluster(corSimMat,tRaw,details=TRUE)
as in R:
https://github.com/jinghuazhao/PW-pipeline/blob/master/files/network.R
J
Jing hua
It would be great to have all functionality of the R package apcluster available in Python!
To answer your questions regarding different results:
First of all, check whether the correlation/similarity matrixes are the same.
Also note that the results are not 100% deterministic, since a small amount of random noise is added internally.
You would have to check all parameters of the two implementations if they are all the same. Obviously, you do not get the same results for both implementations if you use default parameters. But this is only an issue if the defaults are exactly the same. As far as I know, they are not. The default damping parameter, for instance, is not the same.
I hope that helps.
I am using a scipy.minimize function, where I'd like to have one parameter only searching for options with two decimals.
def cost(parameters,input,target):
from sklearn.metrics import mean_squared_error
output = self.model(parameters = parameters,input = input)
cost = mean_squared_error(target.flatten(), output.flatten())
return cost
parameters = [1, 1] # initial parameters
res = minimize(fun=cost, x0=parameters,args=(input,target)
model_parameters = res.x
Here self.model is a function that performs some matrix manipulation based on the parameters. Input and target are two matrices. The function works the way I want to, except I would like to have parameter[1] to have a constraint. Ideally I'd just like to give an numpy array, like np.arange(0,10,0.01). Is this possible?
In general this is very hard to do as smoothness is one of the core-assumptions of those optimizers.
Problems where some variables are discrete and some are not are hard and usually tackled either by mixed-integer optimization (working good for MI-linear-programming, quite okay for MI-convex-programming although there are less good solvers) or global-optimization (usually derivative-free).
Depending on your task-details, i recommend decomposing the problem:
outer-loop for np.arange(0,10,0.01)-like fixing of variable
inner-loop for optimizing, where this variable is fixed
return the model with the best objective (with status=success)
This will effect in N inner-optimizations, where N=state-space of your to fix-var.
Depending on your task/data, it might be a good idea to traverse the fixing-space monotonically (like using np's arange) and use the solution of iteration i as initial-point for the problem i+1 (potentially less iterations needed if guess is good). But this is probably not relevant here, see next part.
If you really got 2 parameters, like indicated, this decomposition leads to an inner-problem with only 1 variable. Then, don't use minimize, use minimize_scalar (faster and more robust; does not need an initial-point).
So far I understand the minimize function with method Trust-ncg, the "method specific" parameter "max_trust_radius" is the maximum value for a new step optimization.
However, I experience a weird behaviour.
I work in my doctorate data and I have a code that invokes minimize function (with trust ncg method)
passing parameters
{
'initial_trust_radius':0.1,
'max_trust_radius':1,
'eta':0.15,
'gtol':1e-5,
'disp': True
}
I invoke minimize function as:
res = minimize(bbox, x0, method='trust-ncg',jac=bbox_der, hess=bbox_hess,options=opt_par)
where
bbox is a function to evaluate the objective function
x0 is the initial guess
bbox_der is the gradient function
bbox_hess hessian function
opt_par is the dictionary above with the parameters.
Bbox invokes simulation code and get the data. It works: minimize go back and forth, proposing new values, bbox invokes simulation.
Everything works well until I got a weird issue.
The "x" vector contains 8 values. I realize that one of the iterations, the last value is greater than 1.
Per the max_trust_radius, I think that it should be less than 1, but it is 1.0621612802208713e+00
The issue causes problems because bbox can not receive the value greater than 1, as it invokes a simulation program and there is a constraint that it can not receive 1 or greater than 1.
I found the scipy code and tried to see if I could be able to find a bug or something wrong but I am not.
My main concerns are:
My understanding is that there is a bug in the scipy minimize code as the new value is greater than max_trust_radius .
How can I manipulate or control the values to avoid that values became greater than 1?
Do you suggest something to investigate the issue?
The max_trust_radius controls how large steps you are allowed to take:
max_trust_radius : float
Maximum value of the trust-region radius.
No steps that are longer than this value will be proposed.
Since you are very likely to take many steps during the minimization, each which can be up to 1 long, it is not strange at all that you (assuming ||x0||=0) end up with ||x|| > 1.
If your problem is strictly bounded then you need to apply an optimization algorithm that supports bounds on the parameters.
For scipy.optimize.minimize only L-BFGS-B, TNC and SLSQP methods seem to support the bounds= keyword.
Abstract problem to be solved:
we have n d-dimentional design variables, say {k_0, k_1, ..., k_n}
maximize the minimum of [f(k_0), f(k_1), ... f(k_n)], where f() a nonlinear function, i.e. maximin
constraint: mean([k_0, k_1, ...,k_n])==m, m known constant
Can someone provide an example of how this can be solved (maximin, d-dim variables) via pyOpt?
EDIT: i tried this:
import scipy as sp
from pyOpt.pyOpt_optimization import Optimization
from pyOpt.pyALPSO.pyALPSO import ALPSO
def __objfunc(x,**kwargs):
f=min([x[0]+x[1],x[2]+x[3]])
g=[0.0]
g[0]=(((x[0]+x[1])+(x[2]+x[3]))/2.0)-5
fail=0
return f,g,fail
if __name__=='__main__':
op=Optimization('test', __objfunc)
op.addVarGroup('p0',4,type='c')
op.addObj('f')
op.addCon('ineq','i')
o=ALPSO()
o(op)
print(op._solutions[0])
suppose 2-dimentional design variables
is there any better way?
I probably would reformulate this as:
The min() function you used is non-differentiable (and thus dangerous). Also the mean() function can be replaced by a linear constraint (which is easier).
I am not familiar with the ALPSO solver, but this reformulation would usually be helpful for more traditional solvers like SNOPT, NLPQL and FSQP.