I'd like to cache intermediate results. I think that a decorator might help.
I not only want to cache based on function arguments, but also based on the implementation.
Background: I want to cache results between program runs because I am experimenting with the individual implementations, and I want to re-use results of the last run if possible.
It seems to be possible to get a function hash with pickle:
from hashlib import sha256
import pickle
def foo(bar):
print(bar)
h = sha256()
h.update(pickle.dumps(foo))
The next step would now be to use this in a decorator:
from hashlib import sha256
import pickle
def decorated(f):
def inner(*args, **kwargs):
h = sha256()
h.update(pickle.dumps((args, kwargs, f))
print(h.hexdigest())
return f(*args, **kwargs)
return inner
#decorated
def foo(bar):
print(bar)
foo("bar")
Now pickle complains:
_pickle.PicklingError: Can't pickle <function foo at 0x7f3d177fdee0>: it's not the same object as __main__.foo
I can understand that my original function got re-assigned during the decoration and that this causes trouble (foo=decorared(foo)).
Is pickling the correct choice here at all?
edit
This question here shows how to get the source code of a method:
How can I get the source code of a Python function? – maybe that's the better approach?
Is pickling the correct choice here at all?
If you want to serialize python function and might use external modules, then you might use dill for that purpose.
I have to write a dummy function to get my code running on different systems, from which some don't have the needed packages. The function is wrapped and then called like a class-function. I am struggling with this problem, any ideas how to do that?
Here I got a short snippet, I import a python script ray.py which should contain this remote() function. The remote function has to take two arguments, without any usage.
Edit: The#ray.remote() wraps the run() function to be parallel executable. It doesn’t change the return of run(). On some systems ray is not supported and I want the same script to execute sequentially without changing anything. Therefore I import a ray-dummy instead of the real one. Now I want to write the ray.remote() to wrap the run() function in a way so that it’s callable with run.remote().
That may be a very inconvenient method to just sequentially execute a function, but necessary to achieve an easy integration for different systems.
# here the wrapped function
#ray.remote(arg1, arg2)
def run(x):
return x**2
# call it
squared = run.remote(2)
I got a working script, located in the ray.py file:
def remote(*args, **kwargs):
def new_func(func):
class Wrapper:
def __init__(self, f):
self.func = f
def remote(self, *arg):
out = self.func(*arg)
return out
ret = Wrapper(func)
return ret
return new_func
I'm trying to implement an "import hook" in Python 3. The hook is supposed to add an attribute to every class that is imported. (Not really every class, but for the sake of simplifying the question, let's assume so.)
I have a loader defined as follows:
import sys
class ConfigurableImports(object):
def find_module(self, fullname, path):
return self
def create_module(self, spec):
# ???
def exec_module(self, module):
# ???
sys.meta_path = [ConfigurableImports()]
The documentation states that as of 3.6, loaders will have to implement both create_module and exec_module. However, the documentation also has little indication what one should do to implement these, and no examples. My use case is very simple because I'm only loading Python modules and the behavior of the loader is supposed to be almost exactly the same as the default behavior.
If I could, I'd just use importlib.import_module and then modify the module contents accordingly; however, since importlib leverages the import hook, I get an infinite recursion.
EDIT: I've also tried using the imp module's load_module, but this is deprecated.
Is there any easy way to implement this functionality with import hooks, or am I going about this the wrong way?
Imho, if you only need to alter the module, that is, play with it after it has been found and loaded, there's no need to actually create a full hook that finds, loads and returns the module; just patch __import__.
This can easily be done in a few lines:
import builtins
from inspect import getmembers, isclass
old_imp = builtins.__import__
def add_attr(mod):
for name, val in getmembers(mod):
if isclass(val):
setattr(val, 'a', 10)
def custom_import(*args, **kwargs):
m = old_imp(*args, **kwargs)
add_attr(m)
return m
builtins.__import__ = custom_import
Here, __import__ is replaced by your custom import that calls the original __import__ to get the loaded module and then calls a function add_attr that does the actual modification of the classes in a module (with getmembers and isclass from inspect) before returning the module.
Of course this is created in a way that when you import the script, the changes are made.
You can and probably should create auxiliary functions that restore and change it again if needed i.e things like:
def revert(): builtins.__import__ = old_imp
def apply(): builtins.__import__ = custom_import
A context-manager would also make this implementation cleaner.
I have some code from which I want to get an email whenever it runs into an exception like this.
try:
f(**kwargs)
except Exception as e:
# email me the environment
I know that Python decorators can work for this, like:
#check_error
def f()
#check_error
def g()
What if I want every code in my module be wrapped by default? Like
def f() without the #check_error, but I still want to be able to achieve this.
Ideally, the solution should apply for both functions and class methods.
You can introspect modules, so assuming module is the module for which you want to decorate all functions you could do something like this:
import inspect
for name, f in inspect.getmembers(module, inspect.isfunction):
setattr(module, name, check_error(f))
You could even do this in the current module by using sys.modules[__name__] for module
How can I get a variable that contains the currently executing function in Python? I don't want the function's name. I know I can use inspect.stack to get the current function name. I want the actual callable object. Can this be done without using inspect.stack to retrieve the function's name and then evaling the name to get the callable object?
Edit: I have a reason to do this, but it's not even a remotely good one. I'm using plac to parse command-line arguments. You use it by doing plac.call(main), which generates an ArgumentParser object from the function signature of "main". Inside "main", if there is a problem with the arguments, I want to exit with an error message that includes the help text from the ArgumentParser object, which means that I need to directly access this object by calling plac.parser_from(main).print_help(). It would be nice to be able to say instead: plac.parser_from(get_current_function()).print_help(), so that I am not relying on the function being named "main". Right now, my implementation of "get_current_function" would be:
import inspect
def get_current_function():
return eval(inspect.stack()[1][3])
But this implementation relies on the function having a name, which I suppose is not too onerous. I'm never going to do plac.call(lambda ...).
In the long run, it might be more useful to ask the author of plac to implement a print_help method to print the help text of the function that was most-recently called using plac, or something similar.
The stack frame tells us what code object we're in. If we can find a function object that refers to that code object in its __code__ attribute, we have found the function.
Fortunately, we can ask the garbage collector which objects hold a reference to our code object, and sift through those, rather than having to traverse every active object in the Python world. There are typically only a handful of references to a code object.
Now, functions can share code objects, and do in the case where you return a function from a function, i.e. a closure. When there's more than one function using a given code object, we can't tell which function it is, so we return None.
import inspect, gc
def giveupthefunc():
frame = inspect.currentframe(1)
code = frame.f_code
globs = frame.f_globals
functype = type(lambda: 0)
funcs = []
for func in gc.get_referrers(code):
if type(func) is functype:
if getattr(func, "__code__", None) is code:
if getattr(func, "__globals__", None) is globs:
funcs.append(func)
if len(funcs) > 1:
return None
return funcs[0] if funcs else None
Some test cases:
def foo():
return giveupthefunc()
zed = lambda: giveupthefunc()
bar, foo = foo, None
print bar()
print zed()
I'm not sure about the performance characteristics of this, but i think it should be fine for your use case.
I recently spent a lot of time trying to do something like this and ended up walking away from it. There's a lot of corner cases.
If you just want the lowest level of the call stack, you can just reference the name that is used in the def statement. This will be bound to the function that you want through lexical closure.
For example:
def recursive(*args, **kwargs):
me = recursive
me will now refer to the function in question regardless of the scope that the function is called from so long as it is not redefined in the scope where the definition occurs. Is there some reason why this won't work?
To get a function that is executing higher up the call stack, I couldn't think of anything that can be reliably done.
This is what you asked for, as close as I can come. Tested in python versions 2.4, 2.6, 3.0.
#!/usr/bin/python
def getfunc():
from inspect import currentframe, getframeinfo
caller = currentframe().f_back
func_name = getframeinfo(caller)[2]
caller = caller.f_back
from pprint import pprint
func = caller.f_locals.get(
func_name, caller.f_globals.get(
func_name
)
)
return func
def main():
def inner1():
def inner2():
print("Current function is %s" % getfunc())
print("Current function is %s" % getfunc())
inner2()
print("Current function is %s" % getfunc())
inner1()
#entry point: parse arguments and call main()
if __name__ == "__main__":
main()
Output:
Current function is <function main at 0x2aec09fe2ed8>
Current function is <function inner1 at 0x2aec09fe2f50>
Current function is <function inner2 at 0x2aec0a0635f0>
Here's another possibility: a decorator that implicitly passes a reference to the called function as the first argument (similar to self in bound instance methods). You have to decorate each function that you want to receive such a reference, but "explicit is better than implicit" as they say.
Of course, it has all the disadvantage of decorators: another function call slightly degrades performance, and the signature of the wrapped function is no longer visible.
import functools
def gottahavethatfunc(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return func(func, *args, **kwargs)
return wrapper
The test case illustrates that the decorated function still gets the reference to itself even if you change the name to which the function is bound. This is because you're only changing the binding of the wrapper function. It also illustrates its use with a lambda.
#gottahavethatfunc
def quux(me):
return me
zoom = gottahavethatfunc(lambda me: me)
baz, quux = quux, None
print baz()
print zoom()
When using this decorator with an instance or class method, the method should accept the function reference as the first argument and the traditional self as the second.
class Demo(object):
#gottahavethatfunc
def method(me, self):
return me
print Demo().method()
The decorator relies on a closure to hold the reference to the wrapped function in the wrapper. Creating the closure directly might actually be cleaner, and won't have the overhead of the extra function call:
def my_func():
def my_func():
return my_func
return my_func
my_func = my_func()
Within the inner function, the name my_func always refers to that function; its value does not rely on a global name that may be changed. Then we just "lift" that function to the global namespace, replacing the reference to the outer function. Works in a class too:
class K(object):
def my_method():
def my_method(self):
return my_method
return my_method
my_method = my_method()
I just define in the beginning of each function a "keyword" which is just a reference to the actual name of the function. I just do this for any function, if it needs it or not:
def test():
this=test
if not hasattr(this,'cnt'):
this.cnt=0
else:
this.cnt+=1
print this.cnt
The call stack does not keep a reference to the function itself -
although the running frame as a reference to the code object that is the code associated to a given function.
(Functions are objects with code, and some information about their environment, such as closures, name, globals dictionary, doc string, default parameters and so on).
Therefore if you are running a regular function, you are better of using its own name on the globals dictionary to call itself, as has been pointed out.
If you are running some dynamic, or lambda code, in which you can't use the function name, the only solution is to rebuild another function object which re-uses thre currently running code object and call that new function instead.
You will loose a couple of things, like default arguments, and it may be hard to get it working with closures (although it can be done).
I have written a blog post on doing exactly that - calling anonymous functions from within themselves - I hope the code in there can help you:
http://metapython.blogspot.com/2010/11/recursive-lambda-functions.html
On a side note: avoid the use o inspect.stack -- it is too slow, as it rebuilds a lot of information each time it is called. prefer to use inspect.currentframe to deal with code frames instead.
This may sounds complicated, but the code itself is very short - I am pasting it bellow. The post above contains more information on how this works.
from inspect import currentframe
from types import FunctionType
lambda_cache = {}
def myself (*args, **kw):
caller_frame = currentframe(1)
code = caller_frame.f_code
if not code in lambda_cache:
lambda_cache[code] = FunctionType(code, caller_frame.f_globals)
return lambda_cache[code](*args, **kw)
if __name__ == "__main__":
print "Factorial of 5", (lambda n: n * myself(n - 1) if n > 1 else 1)(5)
If you really need the original function itself, the "myself" function above could be made to search on some scopes (like the calling function global dictionary) for a function object which code object would match with the one retrieved from the frame, instead of creating a new function.
sys._getframe(0).f_code returns exactly what you need: the codeobject being executed. Having a code object, you can retrieve a name with codeobject.co_name
OK after reading the question and comments again, I think this is a decent test case:
def foo(n):
""" print numbers from 0 to n """
if n: foo(n-1)
print n
g = foo # assign name 'g' to function object
foo = None # clobber name 'foo' which refers to function object
g(10) # dies with TypeError because function object tries to call NoneType
I tried solving it by using a decorator to temporarily clobber the global namespace and reassigning the function object to the original name of the function:
def selfbind(f):
""" Ensures that f's original function name is always defined as f when f is executed """
oname = f.__name__
def g(*args, **kwargs):
# Clobber global namespace
had_key = None
if globals().has_key(oname):
had_key = True
key = globals()[oname]
globals()[oname] = g
# Run function in modified environment
result = f(*args, **kwargs)
# Restore global namespace
if had_key:
globals()[oname] = key
else:
del globals()[oname]
return result
return g
#selfbind
def foo(n):
if n: foo(n-1)
print n
g = foo # assign name 'g' to function object
foo = 2 # calling 'foo' now fails since foo is an int
g(10) # print from 0..10, even though foo is now an int
print foo # prints 2 (the new value of Foo)
I'm sure I haven't thought through all the use cases. The biggest problem I see is the function object intentionally changing what its own name points to (an operation which would be overwritten by the decorator), but that should be ok as long as the recursive function doesn't redefine its own name in the middle of recursing.
Still not sure I'd ever need to do this, but thinking about was interesting.
Here a variation (Python 3.5.1) of the get_referrers() answer, which tries to distinguish between closures that are using the same code object:
import functools
import gc
import inspect
def get_func():
frame = inspect.currentframe().f_back
code = frame.f_code
return [
referer
for referer in gc.get_referrers(code)
if getattr(referer, "__code__", None) is code and
set(inspect.getclosurevars(referer).nonlocals.items()) <=
set(frame.f_locals.items())][0]
def f1(x):
def f2(y):
print(get_func())
return x + y
return f2
f_var1 = f1(1)
f_var1(3)
# <function f1.<locals>.f2 at 0x0000017235CB2C80>
# 4
f_var2 = f1(2)
f_var2(3)
# <function f1.<locals>.f2 at 0x0000017235CB2BF8>
# 5
def f3():
print(get_func())
f3()
# <function f3 at 0x0000017235CB2B70>
def wrapper(func):
functools.wraps(func)
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
return wrapped
#wrapper
def f4():
print(get_func())
f4()
# <function f4 at 0x0000017235CB2A60>
f5 = lambda: get_func()
print(f5())
# <function <lambda> at 0x0000017235CB2950>
Correction of my previous answer, because the subdict check already works with "<=" called on dict_items and the additional set() calls result in problems, if there are dict-values which are dicts themself:
import gc
import inspect
def get_func():
frame = inspect.currentframe().f_back
code = frame.f_code
return [
referer
for referer in gc.get_referrers(code)
if getattr(referer, "__code__", None) is code and
inspect.getclosurevars(referer).nonlocals.items() <=
frame.f_locals.items()][0]