Is there any way to access the cost function on a per-iteration basis with scipy.minimize without using the callback and re-executing the cost function?
options.disp seems to be intended to do this, but only causes the optimizer to print the termination message.
It would be fine to print it to stdout and use contextlib.redirect_stdout with io.StringIO to gather it and parse through the data after, but I can't find a way to efficiently access the cost function on each iteration.
The method least_squares does that with parameter verbose=2. However, it is not a general-purpose minimizer, its purpose to to minimize the sum of squares of the given functions. Example:
least_squares(lambda x: [x[0]*x[1]-6, x[0]+x[1]-5], [0, 0], verbose=2)
For other methods, like minimize, there is no such option. Instead of using callback and re-evaluating the cost function, you may want to add some logging to the function itself. For example, here fun appends the computed values to global variable cost_values:
def fun(x):
c = x[0]**2 - 2*x[0] + x[1]**4
cost_values.append(c)
return c
cost_values = []
minimize(fun, [3, 2])
print(cost_values)
In this example there are 4 similar function values for each iteration step, as the minimization algorithm looks around, computing the approximate Jacobian and/or Hessian. So, print(cost_values[::4]) would be the way to get one value of the cost function per step.
But it's not always 4 values per step (depends on dimension and the method used). So it's better to use a callback function to log the costs after each step. The current cost should be stored in a global variable, so it does not have to be recomputed.
def fun(x):
global current_cost
current_cost = x[0]**2 - 2*x[0] + x[1]**4
return current_cost
def log_cost(x):
cost_values.append(current_cost)
cost_values = []
minimize(fun, [3, 2], callback=log_cost)
print(cost_values)
This prints
[3.5058199763814986, -0.2358850818406083, -0.56104822688320077, -0.88774448831043995, -0.96018358963745964, -0.98750765702936738, -0.99588975368993771, -0.99867208501468863, -0.99956795994852465, -0.99985981414137615, -0.99995446605426996, -0.99998521591611178, -0.99999519917089297, -0.99999844105574265, -0.99999949379700426, -0.99999983560485239, -0.99999994662329761, -0.99999998266175671]
I figured out a sort of hack using stdlib features, it uses a "deep" redirect of sys.stdout. Note that this does not work with jupyter since IPython hijacks sys.stdout, which removes the .fileno attribute.
It may be possible to patch Jupyter using a tempfile.SpooledTemporaryFile in this way, removing this issue. I don't know.
I believe because this uses OS-level file descriptors, it is also not threadsafe.
import os
import sys
import tempfile
class forcefully_redirect_stdout(object):
''' Forces stdout to be redirected, for both python code and C/C++/Fortran
or other linked libraries. Useful for scraping values from e.g. the
disp option for scipy.optimize.minimize.
'''
def __init__(self, to=None):
''' Creates a new forcefully_redirect_stdout context manager.
Args:
to (`None` or `str`): what to redirect to. If type(to) is None,
internally uses a tempfile.SpooledTemporaryFile and returns a UTF-8
string containing the captured output. If type(to) is str, opens a
file at that path and pipes output into it, erasing prior contents.
Returns:
`str` if type(to) is None, else returns `None`.
'''
# initialize where we will redirect to and a file descriptor for python
# stdout -- sys.stdout is used by python, while os.fd(1) is used by
# C/C++/Fortran/etc
self.to = to
self.fd = sys.stdout.fileno()
if self.to is None:
self.to = tempfile.SpooledTemporaryFile(mode='w+b')
else:
self.to = open(to, 'w+b')
self.old_stdout = os.fdopen(os.dup(self.fd), 'w')
self.captured = ''
def __enter__(self):
self._redirect_stdout(to=self.to)
return self
def __exit__(self, *args):
self._redirect_stdout(to=self.old_stdout)
self.to.seek(0)
self.captured = self.to.read().decode('utf-8')
self.to.close()
def _redirect_stdout(self, to):
sys.stdout.close() # implicit flush()
os.dup2(to.fileno(), self.fd) # fd writes to 'to' file
sys.stdout = os.fdopen(self.fd, 'w') # Python writes to fd
if __name__ == '__main__':
import re
from scipy.optimize import minimize
def foo(x):
return 1/(x+0.001)**2 + x
with forcefully_redirect_stdout() as txt:
result = minimize(foo, [100], method='L-BFGS-B', options={'disp': True})
print('this appears before `disp` output')
print('here''s the output from disp:')
print(txt.captured)
lines_with_cost_function_values = \
re.findall(r'At iterate\s*\d\s*f=\s*-*?\d*.\d*D[+-]\d*', txt.captured)
fortran_values = [s.split()[-1] for s in lines_with_cost_function_values]
# fortran uses "D" to denote double and "raw" exp notation,
# fortran value 3.0000000D+02 is equivalent to
# python value 3.0000000E+02 with double precision
python_vals = [float(s.replace('D', 'E')) for s in fortran_values]
print(python_vals)
Related
I have wrote the following code in derivation.py:
def Interpolation(ableitungWinkel,x_values):
z = medfilt(derivation,3)
diff = abs(derivation-z)
new_smootheddata = np.where(diff>3,z,derivation)
x=np.arange(0,len(x_values[:-2]))
f = interp1d(x,new_smootheddata,kind="linear")
xnew = np.arange(0, len(x_values[:-3]),0.01)
ynew = f(xnew)
s=plt.plot(x, z,"o",xnew, ynew, "-")
return s
In my project there is also integration.py. In this Python file I need the values which z calculates in the function def interpolation for this calculation:
def horizontalAcceleration(strideData):
resultsHorizontal = list()
for i in range (len(strideData)):
yAngle = z
xAcceleration = strideData.to_numpy()[i, 4]
yAcceleration = strideData.to_numpy()[i, 5]
a = ((m.cos(m.radians(yAngle)))*yAcceleration)-((m.sin(m.radians(yAngle)))*xAcceleration)
resultsHorizontal.append(a)
resultsHorizontal.insert(0, 0)
return resultsHorizontal
As you can see I have already added z to the function def horizontalAcceleration at the place where it should go.
To use z there, I tried the following: from derivation import z
But that doesn't work. Because then I get the error: ImportError: cannot import name 'z' from 'derivation'
Have anybody an idea how I can solve this problem? Thanks for helping me.
I think that your misunderstanding is because you think a function is like a script that has been run and modified a.global state. That's not what a function is. A function is a series of actions performed on its inputs (ignoring closures for a minute) which returns some results. You can call it many times, but without calling it, it never executes. Once it stops executing all its variables go out of scope.
You can import and call a function though. So you can change the return type of Interpolation to return everything you need somewhere else. E.g.
def Interpolation(...):
...
return {'z': z, 's': s}
Then somewhere you import that function, call it, get back all the data you need, then pass that to your other function.
import Interpolation from derivation
# get z and s in a dict result
result = Interpolation(...)
# pass s as well as the other argument to your other function
horizontalAcceleration(strideData, result['s'])
I'm trying to make a python proram to find derivatives and integrals as well as showing how. I have so far found that there is an integral_steps function which returns the steps used, but I have not found an equivalent for differentiation.
Does anyone know if there is an equivalent?
If there isn't, do you have any ideas on how to find the steps needed to find a derivative?
Method 1 (manual)
Looking at the code, the Derivative class is where the top-level logic lives. That's only the top-level part. From there on, the computation requires computing derivatives of different nodes inside the expression tree.
The logic for each specific node of the expression tree lives in the _eval_derivative method corresponding to each particular node type.
This would allow you to add code to those _eval_derivative methods in order to trace the entire process and find all the steps.
Method 2 (using a tracer)
Python has multiple tracing packages. python-hunter written by #ionelmc is quite good actually and fits this use-case well.
Among many other features, it allows installing certain callbacks when a function starts executing, and another one when the function returns its value. In fact that's exactly what we need.
Here's an example that shows how to use this (I ran and tested this on Python 3.7.3, SymPy 1.7 and hunter 3.3.1) :
import hunter
import sys
from hunter import Q, When, Stop
hunter.trace(
Q(module_contains="sympy",function='_eval_derivative',kind_in=["call","return"],action=hunter.CallPrinter(repr_func=str))
)
from sympy import *
x = symbols('x')
f = 1/(x * sin(x)**2)
f.diff(x)
So this allows to pick which data structures we want to inspect, how we want to print them, and it allows us to see the intermediary steps of the differentiation process:
[...]7/site-packages/sympy/core/power.py:1267 call => _eval_derivative(self=sin(x)**(-2), s=x)
[...]7/site-packages/sympy/core/power.py:1267 call => _eval_derivative(self=<sympy.core.power.Pow object at 0x7f5925337150>, s=<sympy.core.symbol.Symbol object at 0x7f5925b6a2b0>)
[...]ite-packages/sympy/core/function.py:598 call => _eval_derivative(self=sin(x), s=x)
[...]ite-packages/sympy/core/function.py:598 call => _eval_derivative(self=<sympy.functions.elementary.trigonometric.sin object at 0x7f592589ee08>, s=<sympy.core.symbol.Symbol object at 0x7f5925b6a2b0>)
[...]ite-packages/sympy/core/function.py:612 return <= _eval_derivative: cos(x)
[...]ite-packages/sympy/core/function.py:612 return <= _eval_derivative: <sympy.functions.elementary.trigonometric.cos object at 0x7f592525fef8>
[...]7/site-packages/sympy/core/power.py:1271 return <= _eval_derivative: -2*cos(x)/sin(x)**3
[...]7/site-packages/sympy/core/power.py:1271 return <= _eval_derivative: <sympy.core.mul.Mul object at 0x7f5925259b48>
[...]7/site-packages/sympy/core/power.py:1267 call => _eval_derivative(self=1/x, s=x)
[...]7/site-packages/sympy/core/power.py:1267 call => _eval_derivative(self=<sympy.core.power.Pow object at 0x7f5925337200>, s=<sympy.core.symbol.Symbol object at 0x7f5925b6a2b0>)
[...]7/site-packages/sympy/core/power.py:1271 return <= _eval_derivative: -1/x**2
[...]7/site-packages/sympy/core/power.py:1271 return <= _eval_derivative: <sympy.core.mul.Mul object at 0x7f5925259f10>
If you want to also cover the diff function you can alter the code above and have function_in=['_eval_derivative','diff'] . In this way, you can look at not only the partial results, but also the call of the diff function and its return value.
Method 3 (using a tracer, building a call graph and visualizing it)
Using graphviz, latex and a tracer (again, python-hunter) you can actually see the call graph more clearly. It does take a bit of time to render all the formulas for each intermediary step, because pdflatex is being used (I'm sure there's faster renderers for latex though).
Each node's value is in the following format:
function_name
argument => return_value
There seem to be a few diff nodes that have the argument equal to the return value which I'm not sure how to explain at the moment.
The diagram could probably be more useful if it mentioned somehow where each rule was applied (I can't think of an easy way to do that).
Here's the code for this too:
import hunter
import sys
from hunter import Q, When, Stop, Action
from hunter.actions import ColorStreamAction
formula_ltx = r'''
\documentclass[border=2pt,varwidth]{letter}
\usepackage{amsmath}
\pagenumbering{gobble}
\begin{document}
\[ \texttt{TITLE} \]
\[ FORMULA \]
\end{document}
'''
# ==============
# == Tracing ===
# ==============
from sympy.printing.latex import LatexPrinter, print_latex, latex
global call_tree_root
# a node object to hold an observed function call
# with its argument, its return value and its function name
class Node(object):
def __init__(self, arg=None, retval=None, func_name=None):
self.arg = arg
self.retval = retval
self.arg_ascii = ""
self.retval_ascii = ""
self.func_name = func_name
self.uid = 0
self.children = []
# this is a hunter action where we build a call graph and populate it
# so we can later render it
#
# CGBAction (Call Graph Builder Action)
class CGBAction(ColorStreamAction):
def __init__(self, *args, **kwargs):
super(ColorStreamAction, self).__init__(*args, **kwargs)
# a custom call stack
self.tstack = []
global call_tree_root
call_tree_root = Node(arg="",func_name="root")
self.node_idx = 1
self.tstack.append(call_tree_root)
def __call__(self, event):
if event.kind in ['return','call']:
if event.kind == 'return':
print(str(event.arg))
if len(self.tstack) > 0:
top = self.tstack.pop()
top.retval = latex(event.arg)
top.retval_ascii = str(event.arg)
elif event.kind == 'call':
print(str(event.locals.get('self')))
new = Node()
new.uid = self.node_idx
new.arg = latex(event.locals.get('self'))
new.arg_ascii = str(event.locals.get('self'))
top = self.tstack[-1]
self.tstack.append(new)
top.children.append(new)
new.func_name = event.module + ":" + event.function
self.node_idx += 1
hunter.trace(
Q(module_contains="sympy",function_in=['_eval_derivative','diff'],kind_in=["call","return"],action=CGBAction)
)
from sympy import *
x = symbols('x')
f = 1 / (x * sin(x)**2)
#f = 1 / (x * 3)
#f = sin(exp(cos(x)*asin(x)))
f.diff(x)
# ============================
# == Call graph rendering ====
# ============================
import os
import re
OUT_DIR="formulas"
if not os.path.exists(OUT_DIR):
os.mkdir(OUT_DIR)
def write_formula(prefix,uid,formula,title):
TEX = uid + prefix + ".tex"
PDF = uid + prefix + ".pdf"
PNG = uid + prefix + ".png"
TEX_PATH = OUT_DIR + "/" + TEX
with open(TEX_PATH,"w") as f:
ll = formula_ltx
ll = ll.replace("FORMULA",formula)
ll = ll.replace("TITLE",title)
f.write(ll)
# compile formula
CMD = """
cd formulas ;
pdflatex {TEX} ;
convert -trim -density 300 {PDF} -quality 90 -colorspace RGB {PNG} ;
""".format(TEX=TEX,PDF=PDF,PNG=PNG)
os.system(CMD)
buf_nodes = ""
buf_edges = ""
def dfs_tree(x):
global buf_nodes, buf_edges
arg = ("" if x.arg is None else x.arg)
rv = ("" if x.retval is None else x.retval)
arg = arg.replace("\r","")
rv = rv.replace("\r","")
formula = arg + "\\Rightarrow " + rv
print(x.func_name + " -> " + x.arg_ascii + " -> " + x.retval_ascii)
x.func_name = x.func_name.replace("_","\\_")
write_formula("",str(x.uid),formula,x.func_name)
buf_nodes += """
{0} [image="{0}.png" label=""];
""".format(x.uid);
for y in x.children:
buf_edges += "{0} -> {1};\n".format(x.uid,y.uid);
dfs_tree(y)
dfs_tree(call_tree_root)
g = open(OUT_DIR + "/graph.dot", "w")
g.write("digraph g{")
g.write(buf_nodes)
g.write(buf_edges)
g.write("}\n")
g.close()
os.system("""cd formulas ; dot -Tpng graph.dot > graph.png ;""")
Mapping SymPy logic to differentiation rules
I think one remaining step is to map intermediary nodes from SymPy to differentiation rules. Here's some of the ones I was able to map:
Product rule maps to sympy.core.mul.Mul._eval_derivative
Chain rule maps to sympy.core.function.Function._eval_derivative
Sum rule maps to sympy.core.add.Add._eval_derivative
Derivative of a summation maps to sympy.concrete.summations.Sum._eval_derivative
n-th Derivative of a product via the General Leibniz rule maps to sympy.core.mul.Mul._eval_derivative_n_times
Generalized power rule maps to sympy.core.power.Pow._eval_derivative
I haven't seen a Fraction class in sympy.core so maybe the quotient rule is handled indirectly through a product rule, and a generalized power rule with exponent -1.
Running
In order to get this to run you'll need:
sudo apt-get install graphviz imagemagick texlive texlive-latex-base
And the file /etc/ImageMagick-6/policy.xml will have to be updated with the following line to allow conversion fron PDF->PNG:
<policy domain="coder" rights="read|write" pattern="PDF" />
There's another call graph library called jonga but it's a bit generic and doesn't allow to completely filter out unwanted calls.
I am trying to understand why I must store the output of a Python function (regardless of the name of the variable I use, and regardless of whether I subsequently use that variable). I think this is more general to Python and not specifically to the software NEURON, thus I put it here on Stackoverflow.
The line of interest is here:
clamp_output = attach_current_clamp(cell)
If I just write attach_current_clamp(cell), without storing the output of the function into a variable, the code does not work (plot is empty), and yet I don't use clamp_output at all. Why cannot I not just call the function? Why must I use a variable to store the output even without using the output?
import sys
import numpy
sys.path.append('/Applications/NEURON-7.4/nrn/lib/python')
from neuron import h, gui
from matplotlib import pyplot
#SET UP CELL
class SingleCell(object):
def __init__(self):
self.soma = h.Section(name='soma', cell=self)
self.soma.L = self.soma.diam = 12.6517
self.all = h.SectionList()
self.all.wholetree(sec=self.soma)
self.soma.insert('pas')
self.soma.e_pas = -65
for sec in self.all:
sec.cm = 20
#CURRENT CLAMP
def attach_current_clamp(cell):
stim = h.IClamp(cell.soma(1))
stim.delay = 100
stim.dur = 300
stim.amp = 0.2
return stim
cell = SingleCell()
#IF I CALL THIS FUNCTION WITHOUT STORING THE OUTPUT, THEN IT DOES NOT WORK
clamp_output = attach_current_clamp(cell)
#RECORD AND PLOT
soma_v_vec = h.Vector()
t_vec = h.Vector()
soma_v_vec.record(cell.soma(0.5)._ref_v)
t_vec.record(h._ref_t)
h.tstop = 800
h.run()
pyplot.figure(figsize=(8,4))
soma_plot = pyplot.plot(t_vec,soma_v_vec)
pyplot.show()
This is a NEURON+Python specific bug/feature. It has to do with Python garbage collection and the way NEURON implements the Python-HOC interface.
When there are no more references to a NEURON object (e.g. the IClamp) from within Python or HOC, the object is removed from NEURON.
Saving the IClamp as a property of the cell averts the problem in the same way as saving the result, so that could be an option for you:
# In __init__:
self.IClamps = []
# In attach_current_clamp:
stim.amp = 0.2
cell.IClamps.append(stim)
#return stim
I'm trying to implement a GNURadio source block in Python, which has to produce a vector of a fixed size at each call of the [general_] work function.
As a first toy-example I tried to output just a vector of constant values which should change at each call of the [general_] work function.
import numpy
import sys
from gnuradio import gr
class my_source_vf(gr.sync_block):
"""
docstring for block
"""
def __init__(self, v_size):
self.v_size = v_size
self.mult = 1
self.buff = numpy.ones(v_size)
gr.sync_block.__init__(self,
name="my_source_vf",
in_sig=None,
#out_sig=[numpy.float32])
out_sig=[(numpy.float32, self.v_size)])
def work(self, input_items, output_items):
# <+signal processing here+>
print len(output_items)
out = output_items[0]
out[0][:] = self.buff*self.mult
self.mult = self.mult+1
return self.v_size
However, when I connect it to QT GUI Vector sink block I just see oscillations between 0 and 1, which let me think [general_] work function is called just once.
You mustn't return v_size – that's the length of one item, but you should return the number of items you've produced this call.
Ok, full disclosure, I'm not doing a grid search but the minimum example I could find for what I'm about to ask could (with a grain of salt) be reduced to a grid search (in case you're wondering why I'm not mentioning numpy and friends).
I'm doing a grid search in Python where one axis is discrete and the other one continuous, but for simplicity, let's say I have the following:
x_axis = ['linear', 'quadratic', 'cubic']
y_axis = range(1, 100) # simplification
The evaluation of my function depends on both axes and an intermediate log structure, so in an effort to have no global data I'm defining my function as a closure:
def get_function(xval, *args):
""" creates the closure that encapsulates thread local data
"""
log = { } # initialization depends on args
def fun(yval):
""" evaluation dedicated to single x_axis value
"""
if yval in log:
# in a proper grid search I wouldn't check twice
# but this is just to show that log is used and
# ammended inside fun()
else:
log[yval] = 0
return very_time_consuming_fun(xval, yval, log)
So the script uses this set up to run a grid search:
def func_eval(fun):
for yval in y_axis:
fun(yval)
# the loop I want to parallelize
for xval in x_axis:
fun = get_function(xval, args) # args are computed based on xval
func_eval(fun) # can I do result = func_eval(fun) ?
The things I want to ask are:
Am I correct in assuming that log works with a different instance for each x_axis value?
What is the best way to parallelize the last for loop? (If synchronization is needed for the log instances please elaborate). Again, I only want the evaluation of each x_axis value to its thread / core / you name it (best practices are welcomed)
Is there a way to get results out for each func_eval i.e. could it still be parallelized, if I had the following:
out = func_eval(fun)
I hate the silence ...
This is what I'm doing for now (so at least I'll get downvoted if it ain't right)
import multiprocessing
pool = multiprocessing.Pool()
funcs = [get_function(xval, args) for x in x_axis]
outputs = pool.map(func_eval, funcs)
this should be good enough according but I get the following error:
PicklingError: Can't pickle : attribute lookup builtin.function failed