Develop Reference

Can openmdao compute partial derivatives across a Matlab ExternalCodeComp without explicitly defining them? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
Is it possible to have openmdao to approximate partial the derivatives across the ExternalCodeComp using finite difference.
Just by using the method self.declare_partials('*','*', method='fd') seems not to work
Optimization converges after 1 iteration with only 1 function and gradient evaluations.
The error that pop ups :
DerivativesWarning:Constraints or objectives [('p.f_xy', inds=[0])] cannot be impacted by the design variables of the problem.
DerivativesWarning:Design variables [('p.x', inds=[0]), ('p.y', inds=[0])] have no impact on the constraints or objective.
Optimization terminated successfully .

We run a test case very similar to this as part of the OpenMDAO test suite. Your declare_partials call is not quite correct because you list the first two args as '' and '' which would not match any variable names. I suspect that's just a typo in your post though because if you actually ran OpenMDAO while using those args you would get an exception telling you that the declared partials didn't match any variables. In the example shown below I declare the partials as self.declare_partials(of='*', wrt='*', method='fd'). Assuming that your partials are actually declared correctly, my guess is that for some reason the output file that your external code is generating is either not getting updated at all or you're always writing the same values to the output file. Below is a working example of an external code that computes a paraboloid. Hopefully that will help you track down the issue. If not, you can try posting your code here and we can go from there.
Here's the OpenMDAO script:
import sys
import openmdao.api as om
class ParaboloidExternalCodeCompFD(om.ExternalCodeComp):
def setup(self):
self.add_input('x', val=0.0)
self.add_input('y', val=0.0)
self.add_output('f_xy', val=0.0)
self.input_file = 'paraboloid_input.dat'
self.output_file = 'paraboloid_output.dat'
# providing these is optional; the component will verify that any input
# files exist before execution and that the output files exist after.
self.options['external_input_files'] = [self.input_file]
self.options['external_output_files'] = [self.output_file]
self.options['command'] = [
sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file
]
def setup_partials(self):
# this external code does not provide derivatives, use finite difference
self.declare_partials(of='*', wrt='*', method='fd')
def compute(self, inputs, outputs):
x = inputs['x']
y = inputs['y']
# generate the input file for the paraboloid external code
with open(self.input_file, 'w') as input_file:
input_file.write('%.16f\n%.16f\n' % (x, y))
# the parent compute function actually runs the external code
super().compute(inputs, outputs)
# parse the output file from the external code and set the value of f_xy
with open(self.output_file, 'r') as output_file:
f_xy = float(output_file.read())
outputs['f_xy'] = f_xy
prob = om.Problem()
model = prob.model
model.add_subsystem('p', ParaboloidExternalCodeCompFD())
# find optimal solution with SciPy optimize
# solution (minimum): x = 6.6667; y = -7.3333
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'
prob.model.add_design_var('p.x', lower=-50, upper=50)
prob.model.add_design_var('p.y', lower=-50, upper=50)
prob.model.add_objective('p.f_xy')
prob.driver.options['tol'] = 1e-9
prob.driver.options['disp'] = True
prob.setup()
# Set input values
prob.set_val('p.x', 3.0)
prob.set_val('p.y', -4.0)
prob.run_driver()
print('p.x =', prob.get_val('p.x'), " expected:", [6.66666667])
print('p.x =', prob.get_val('p.y'), " expected:", [-7.3333333])
And here's the external code script, which is named extcode_paraboloid.py:
#!/usr/bin/env python
#
# usage: extcode_paraboloid.py input_filename output_filename
#
# Evaluates the equation f(x,y) = (x-3)^2 + xy + (y+4)^2 - 3.
#
# Read the values of `x` and `y` from input file
# and write the value of `f_xy` to output file.
if __name__ == '__main__':
import sys
input_filename = sys.argv[1]
output_filename = sys.argv[2]
with open(input_filename, 'r') as input_file:
file_contents = input_file.readlines()
x, y = [float(f) for f in file_contents]
f_xy = (x-3.0)**2 + x*y + (y+4.0)**2 - 3.0
with open(output_filename, 'w') as output_file:
output_file.write('%.16f\n' % f_xy)
If you place them both in the same directory and run the OpenMDAO script, you should get something like:
Optimization terminated successfully. (Exit mode 0)
Current function value: -27.333333333333
Iterations: 5
Function evaluations: 6
Gradient evaluations: 5
Optimization Complete
-----------------------------------
p.x = [6.66666633] expected: [6.66666667]
p.x = [-7.33333367] expected: [-7.3333333]

How to get a transformation of a function as a return?

My objective is to implement the following system (Taken from signal processing schematic-ish):
Now, X[n] could be any signal, any function, for simplicity and for testing the program we could see it as the exponential function. And as the signal goes along the subsystems it undergoes some transformations, for S1 the transformation is:
and after S2 is:
and so far I have done this:
import numpy as np
import timeit
n=np.arange(-20,20)
y=np.zeros(n.shape)
x=np.exp(n)
def S1(inputSign, x_values, finalSys=True):
if finalSys==True:
output=np.zeros(x_values.shape)
for i in range(len(x_values)):
if x_values[i]%2==0:
output[i]=inputSign(x_values[i]/2)
else:
output[i] = 0
return output
else:
return inputSign
def S2(inputSign, x_values, finalSys=True):
if finalSys==True:
output=np.zeros(x_values.shape)
for i in range(len(x_values)):
output[i]=inputSign(x_values)+0.5*inputSign(x_values)+0.25*inputSign(x_values)
return output
else:
return inputSign+0.5*?????
Now the functions are defined with a condition finalSys that is supposed to be that if it is the last system it gives me an array of values. The objective is to "cascade"(S2(S1,n,finalSys=True), maybe not the correct term) the functions to as to get a final answer dependent only of the initial input signal but transformed along the way by the subsystems. I could do it manually and get an expression to the resulting system but I am doing this as a learning exercise so please bear with me. My trouble has been how to alter the arguments of the inputSign. As an example, S1(np.exp,n,finalSys=False)(2) should return the same as np.exp(1).

Error evaluating a derivative on Python (with .subs, .evalf and .lambdify)

I am trying to separately compute the elements of a Taylor expansion and did not obtain the results I was supposed to. The function to approximate is x**321, and the first three elements of that Taylor expansion around x=1 should be:
1 + 321(x-1) + 51360(x-1)**2
For some reason, the code associated with the second term is not working.
See my code below.
import sympy as sy
import numpy as np
import math
import matplotlib.pyplot as plt
x = sy.Symbol('x')
f = x**321
x0 = 1
func0 = f.diff(x,0).subs(x,x0)*((x-x0)**0/factorial(0))
print(func0)
func1 = f.diff(x,1).subs(x,x0)*((x-x0)**1/factorial(1))
print(func1)
func2 = f.diff(x,2).subs(x,x0)*((x-x0)**2/factorial(2))
print(func2)
The prints I obtain running this code are
1
321x - 321
51360*(x - 1)**2
I also used .evalf and .lambdify but the results were the same. I can't understand where the error is coming from.
f = x**321
x = sy.Symbol('x')
def fprime(x):
return sy.diff(f,x)
DerivativeOfF = sy.lambdify((x),fprime(x),"numpy")
print(DerivativeOfF(1)*((x-x0)**1/factorial(1)))
321*x - 321
I'm obviously just starting with the language, so thank you for your help.

I found a beginners guide how to Taylor expand in python. Check it out perhaps all your questions are answered there:
http://firsttimeprogrammer.blogspot.com/2015/03/taylor-series-with-python-and-sympy.html
I tested your code and it works fine. like Bazingaa pointed out in the comments it is just an issue how python saves functions internally. One could argument that for a computer it takes less RAM to save 321*x - 321 instead of 321*(x - 1)**1.
In your first output line it also gives you 1 instead of (x - 1)**0

Python: How to solve an ordinary differential equation with integral term in it

I dont know if this question has been asked before in SO, I will go ahead and post it here, I am attempting to solve a simple system with a PID controller, my system of differential equations are given below. I am basically attempting to code very basic PID algorithm. The structure of my control u depends on both derivative and integral of error term. I dont have any problem with the derivative term, it is the integral term that is creating problem in my code. The problem crops up when I assign s=0 in the beginning
and use it in my function as described in my code below. Is there a way to bypass it? I tried assigning s and told as global variables, but it didnt solve my problem. In a nutshell what I am doing is- I am adding state x1 every time and multiplying by dt(which is denoted by t-told).
Kindly help me iron out this issue, PFA my code attached below.
import numpy as np
from scipy.integrate import ode
import matplotlib.pyplot as plt
plt.style.use('bmh')
t0=0
y0=[0.1,0.2]
kp,kd,ki=2,0.5,0.8
s,told=0,0
def pid(t,Y):
x1,x2=Y[0],Y[1]
e=x1-1
de=x2
s=(x1+s)
integral=s*(t-told)
told=t
#ie=
u=kp*e+kd*de+ki*integral
x1dot=x2
x2dot=u-5*x1-2*x2
return[x1dot,x2dot]
solver=ode(pid).set_integrator('dopri5',rtol=1e-6,method='bdf',nsteps=1e5,max_step=1e-3)
solver.set_initial_value(y0,t0)
t1=10
dt=5e-3
sol = [ [yy] for yy in y0 ]
t=[t0]
while solver.successful() and solver.t<t1:
solver.integrate(solver.t+dt)
for k in range(2): sol[k].append(solver.y[k]);
t.append(solver.t)
print(len(sol[0]))
print(len(t))
x1=np.array(sol[0])
x2=np.array(sol[1])
e=x1-1
de=x2
u=kp*e+kd*de
for k in range(2):
if k==0:
plt.subplot(2,1,k+1)
plt.plot(t,sol[k],label='x1')
plt.plot(t,sol[k+1],label='x2')
plt.legend(loc='lower right')
else:
plt.subplot(2,1,k+1)
plt.plot(t,u)
plt.show()

You are making assumptions on the solver and the time steps that it visits that are not justified. With your hacking of the integral, even if it were mathematically sound (it should look like integral = integral + e*(t-told), which gives an order 1 integration method), you reduce the order of any integration method, probably down to 1, if you are lucky only to order 2.
A mathematically correct method to implement this system is to introduce a third variable x3 for the integral of e, that is, the derivative of x3 is e. That the correct order 1 system has to be of dimension 3 can be read of the fact that (eliminating e) your system has 3 differentiation/integration operations. With that your system becomes
def pid(t,Y):
x1, x2, x3 =Y
e=x1-1
x1dot = x2
edot = x1dot
x3dot = e
u=kp*e+kd*edot+ki*x3
x2dot=u-5*x1-2*x2
return[x1dot, x2dot, x3dot]
Note that there are no global dynamic variables necessary, only the constants (which could also be passed as parameters, whatever seems more efficient or readable).
Now you will also need an initial value for x3, it was not visible from the system what the integration variable would have to be, your code seems to suggest 0.

First of all you need to include "s" Variable into the pid function.
'
def pid(s, t, Y): ...
'

Easiest solution I can see right now is to create a class with s and told as properties of this class:
class PIDSolver:
def __init__(self)
self.t0=0
self.y0=[0.1,0.2]
self.kp,self.kd,self.ki=2,0.5,0.8
self.s,self.told=0,0
def pid(t,Y):
x1,x2=Y[0],Y[1]
e=x1-1
de=x2
self.s=(x1+self.s)
integral=self.s*(t-self.told)
self.told=t
#ie=
u=self.kp*e+self.kd*de+self.ki*integral
x1dot=x2
x2dot=u-5*x1-2*x2
return[x1dot,x2dot]
For the first part of your problem. Use pidsolver = PIDSolver() in the next part of your solution.

I solved this problem myself by using set_f_params() method and passing a list in itz argument. Also I passed a 3rd argument in pid() i.e pid(t,Y,arg). And lastly I assigned s,told=arg[0],arg[1].

The MSS-maximum likelihood method for Fluctuation Test

I am trying to implement an algorithm in the following paper (method 5) http://dx.doi.org/10.1016%2FS0076-6879(05)09012-9 in python2.7 to improve my programming skill. Implementations can be found at these locations: Apparently I cannot post so many links. If my reputation goes up, I will post the links here.
Essentially, the algorithm is used for biological research, and finds the mutation rate of cells under some condition. Here is my attempt, which has errors (NOTE: I updated this code to remove an error however I am still not getting the right answer):
import numpy as np
import sympy as sp
from scipy.optimize import minimize
def leeCoulson(nparray):
median=np.median(nparray)
x=sp.Symbol('x')
M_est=sp.solve(sp.Eq(-x*sp.log(x) - 1.24*x + median,0),x)
return M_est
def ctArray(nparray,max):
list=[0] * int(max+1)
for i in range(int(max)+1):
list[i]=nparray.count(i)
return list
values='filename1.csv'
data=np.genfromtxt(values,delimiter=',')
mVal=int(max(data))
ctArray_=ctArray(np.ndarray.tolist(data),mVal)
ef mssCalc(estM,max=mVal,count=ctArray_):
def rec(pi,r):
pr=(estM/r)+sum([(pi[i]/(r-i+1)) for i in range(0,r-1)])
return pr
prod=1
pi=[0]*max
pi[0]=np.exp(-1*estM)
for r in range(1,max):
pi[r]=rec(pi,r)
prod=prod*(pi[r]**count[r])
return -1*prod
finalM=minimize (mssCalc,leeCoulson(data),method='nelder-mead',options={'xtol':1e-3,'disp':True})
print finalM
This code gives the following errors:
mss-mle_calc.py:37: RuntimeWarning: overflow encountered in multiply
prod=prod*(pi[r]**count[r])
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/optimize.py:462: RuntimeWarning: invalid value encountered in subtract
numpy.max(numpy.abs(fsim[0] - fsim[1:])) <= ftol):
Warning: Maximum number of function evaluations has been exceeded.
Please help me make this code better if you have some time.

Thanks for looking, there were a couple stupid mistakes in my code. Here is the working code (as far as I can tell):
import numpy as np
import sympy as sp
from scipy.optimize import minimize
def leeCoulson(nparray):
median=np.median(nparray)
x=sp.Symbol('x')
M_est=sp.solve(sp.Eq(-x*sp.log(x) - 1.24*x + median,0),x)
return float(M_est[0])
def ctArray(nparray,max):
list=[0] * int(max+1)
for i in range(int(max)+1):
list[i]=nparray.count(i)
return list
values='filename1.csv'
data=np.genfromtxt(values,delimiter=',')
mVal=int(max(data))
ctArray_=ctArray(np.ndarray.tolist(data),mVal)
def mssCalc(estM,max=mVal,count=ctArray_):
def rec(pi,r):
pr=(estM/r)*sum([(pi[i]/(r-i+1)) for i in range(0,r)])
return pr
prod=1
pi=[0]*(max+1)
pi[0]=np.exp(-1.0*estM)
for r in range(1,max+1):
pi[r]=rec(pi,r)
prod=prod*(pi[r]**count[r])
return -1*prod
finalM=minimize (mssCalc,leeCoulson(data),method='nelder-mead',options={'xtol':1e-3,'disp':True})
print finalM