Related
I need to write a function that satisfies this test in pytest:
def test_return_logger(capsys):
#return_logger
def myfunc(a, b, c):
return a + b + c
val = myfunc(4, 5, 6)
out, err = capsys.readouterr()
assert val == 15
assert "Function returned: 15" in out
Here is what I currently have and it is not working:
def return_logger(f):
def newfunc(s):
original_return_value = f(s)
return f"Function returned: {original_return_value}"
return newfunc
Im honestly completely lost!
Your test wants the function to return the value, and print the string that you were returning.
The newfunc(*args) also solves passing in multiple arguments - they are automatically unpacked when calling the second function.
You probably want this:
def return_logger(f):
def newfunc(*args):
original_return_value = f(*args)
print(f"Function returned: {original_return_value}", sys.stderr)
return original_return_value
return newfunc
Now, when you call the decorated function, it will print Function returned: 15 to stderr and return original_return_value.
I am writing a (python3) program and I got stuck when trying to implement a (function) decorator which updates an external variable, a kind of signal emitting decorator. The problem is the conflict with various functions scopes. I have look around to some similar problems but I haven't found yet the useful one... I need to respect, if possible, some design restrictions (see below) and I also would like do avoid to use external libraries.
Here a working example with the globalkeyword which can be useful as starting point
VAR = 'i am a global variable'
# decorator
def update_external_variable():
def f_wrapper(f):
def p_wrapper(p, q):
r = f(p, q) + ': updating the global variable ??'
global VAR
VAR = r
return r
return p_wrapper
return f_wrapper
#update_external_variable()
def a(p, q): return 'a({}, {})'.format(p, q) #target function
o = a('v', 'w')
print(VAR, id(VAR))
Ouput
a(v, w): updating the global variable ?? 140497617759280 # yes, it works!
Design restriction 1: the decorator, update_external_variable, should not depend on the external variable identifier (name), so it has to be passed as parameter. The signature of update_external_variable should contains the info of the global variable, VAR.
Attempt 1: the mokey patch way - I tried mimic the above working example but with no results
VAR = 'i am a global variable'
# decorator
def update_external_variable(ext_var_id): # ext_var_id: string with the variable identifier
def f_wrapper(f):
def p_wrapper(p, q):
r = f(p, q) + ': updating the global variable ??'
exec('global {}'.format(ext_var_id), {}) # -> global VAR
exec('{} = "{}"'.format(ext_var_id, eval(ext_var_id))) # initialize VAR??
#print(dir())
return r
return p_wrapper
return f_wrapper
#update_external_variable(ext_var_id='VAR')
def a(p, q): return 'a({}, {})'.format(p, q) #target function
o = a('v', 'w')
print(o, id(o))
Output
a(v, w): updating the global variable ?? 140686557781040
i am a global variable # failure!
Attempt 2: the parameters' way
VAR = 'i am a global variable'
# decorator
def update_external_variable(ext_var): # ext_var: reference of the global variable
def f_wrapper(f):
def p_wrapper(p, q, ext_var=ext_var):
r = f(p, q) + ': updating the global variable ??'
# global ext_var <- will raise to an error since point to the parameter..
print(ext_var)
ext_var = r
return r
return p_wrapper
return f_wrapper
#update_external_variable(ext_var=VAR)
def a(p, q): return 'a({}, {})'.format(p, q) # target function
o = a('v', 'w')
print(o, id(o))
print(VAR)
Output
i am a global variable
a(v, w): updating the global variable ?? 140406972742896
i am a global variable # failure!
Design restriction 2: if a solution using the attempt 2 exist then I need to impose the following restriction on the signature of the p_wrapper which can could give rise to further problems: def p_wrapper(*args, **kwargs): ... To give to the decorator a universal fingerprint I need that the arguments of p_wrapper to be those of the function to be decorated, r = func(*args, **kwargs).
If someone has an idea on how to solve this problem, it can be either for Attempt 1 or Attempt 2 or even combination of them or another solution as well, I will be very grateful!
Thanks in advance:)
Don't use individual global variables; use a dict that your decorated function can update.
VARS = {
'var1': ...,
'var2': ...,
}
# decorator
def update_external_variable(varkey):
def f_wrapper(f):
def p_wrapper(p, q):
r = f(p, q) + ': updating the global variable ??'
VARS[varkey] = r
return r
return p_wrapper
return f_wrapper
#update_external_variable('var1')
def a(p, q):
return 'a({}, {})'.format(p, q) #target function
o = a('v', 'w')
print(VARS['var1'])
If, for whatever reason, your decorator must work with existing global variables that you can't change, use globals() to get access to the necessary dict.
VAR1 = ...
VAR2 = ...
# decorator
def update_external_variable(varkey):
def f_wrapper(f):
def p_wrapper(p, q):
r = f(p, q) + ': updating the global variable ??'
globals()[varkey] = r
return r
return p_wrapper
return f_wrapper
# Still passing the *name* of the variable as a string
#update_external_variable('VAR1')
def a(p, q):
return 'a({}, {})'.format(p, q) #target function
o = a('v', 'w')
print(VAR1)
is there a feautre similar to C macros which lets you reuse code in an inline manner, without creating a seperate scope for that piece of code?
for example:
a=3
def foo():
a=4
foo()
print a
will print 3, however i want it to print 4.
i am aware of solutions involving objects like classes or a global dict, however i'm looking for a more primitive solution (like a function decorator for example) that would simply let me make changes inside the scope of the caller instead.
thank you very much
edit:any solution that requires declaring which variables i'm going to use OR declaring a "namespace" like mutabale objects beforehand is not a solution i'm looking for.
i had made an attempt on my own:
def pgame():
a=3
c=5
print locals()
game(a)
print locals()
class inline_func(object):
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
return self.f(*args, **kwargs)
#to be #inline_func
def game(b, a=4):
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1] [0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))\ninspect.stack()[1][0].f_locals.update(inspect.stack()[3][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[1][0]),ctypes.c_int(0))")
try:
print "your code here"
finally:
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))")
#inline_func
def strip_game(b, a=4):
print "your code here"
but i have ran into a serious problem with how to inject code into strip_game without ruining the debugability of the program, because i had only thought of creating a new code object or using exec, both suffering from some severe problems.
MAJOR EDIT:
ok, so i have something close to a working solution, however i encounter a very wierd problem:
import inspect
import ctypes
import struct
import dis
import types
def cgame():
a=3
c=5
print locals()
strip_game(a)
print locals()
def pgame():
a=3
c=5
print locals()
game(a)
print locals()
class empty_deco(object):
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
return self.f(*args, **kwargs)
debug_func = None
class inline_func(object):
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
init_exec_string = "inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))\n" + \
"inspect.stack()[1][0].f_locals.update(inspect.stack()[3][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[1][0]),ctypes.c_int(0))"
fini_exec_string = "inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))"
co_stacksize = max(6, self.f.func_code.co_stacksize) # make sure we have enough space on the stack for everything
co_consts = self.f.func_code.co_consts +(init_exec_string, fini_exec_string)
init = "d" + struct.pack("H", len(strip_game.f.func_code.co_consts)) #LOAD_CONST init_exec_string
init += "d\x00\x00\x04U" # LOAD_CONST None, DUP_TOP, EXEC_STMT
init += "z" + struct.pack("H", len(self.f.func_code.co_code) + 4) #SETUP_FINALLY
fini = "Wd\x00\x00" # POP_BLOCK, LOAD_CONST None
fini += "d" + struct.pack("H", len(strip_game.f.func_code.co_consts) + 1) #LOAD_CONST fini_exec_string
fini += "d\x00\x00\x04UXd\x00\x00S" # LOAD_CONST None, DUP_TOP, EXEC_STMT, END_FINALLY, LOAD_CONST None, RETURN
co_code = init + self.f.func_code.co_code + fini
co_lnotab = "\x00\x00\x0b" + self.f.func_code.co_lnotab[1:] # every error in init will be attributed to #inline_func, errors in the function will be treated as expected, errors in fini will be attributed to the last line probably.
new_code = types.CodeType(
self.f.func_code.co_argcount,
self.f.func_code.co_nlocals,
co_stacksize,
self.f.func_code.co_flags & ~(1), # optimized functions are problematic for us
co_code,
co_consts,
self.f.func_code.co_names,
self.f.func_code.co_varnames,
self.f.func_code.co_filename,
self.f.func_code.co_name,
self.f.func_code.co_firstlineno,
co_lnotab,
self.f.func_code.co_freevars,
self.f.func_code.co_cellvars,)
self.inline_f = types.FunctionType(new_code, self.f.func_globals, self.f.func_name, self.f.func_defaults, self.f.func_closure)
#dis.dis(self.inline_f)
global debug_func
debug_func = self.inline_f
return self.inline_f(*args, **kwargs)
#empty_deco
def game(b, a=4):
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))\ninspect.stack()[1][0].f_locals.update(inspect.stack()[3][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[1][0]),ctypes.c_int(0))")
try:
print "inner locals:"
print locals()
print c
return None
finally:
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))")
#inline_func
def strip_game(b, a=4):
print "inner locals:"
print locals()
print c
return None
def stupid():
exec("print 'hello'")
try:
a=1
b=2
c=3
d=4
finally:
exec("print 'goodbye'")
now this seems to work however, i get the following:
>>>cgame()
{'a': 3, 'c': 5}
{'a': 4, 'c': 5, 'b': 3}
your code here
Traceback (most recent call last):
File "<pyshell#43>", line 1, in <module>
cgame()
File "C:\Python27\somefile.py", line 14, in cgame
strip_game(a)
File "C:\Python27\somefile.py", line 78, in __call__
return self.inline_f(*args, **kwargs)
File "C:\Python27\somefile.py", line 94, in strip_game
z = c
NameError: global name 'c' is not defined
now when i disassemble the functions, i get the following very wierd compilation difference between game and strip_game:
in game:
86 16 LOAD_NAME 0 (locals)
19 CALL_FUNCTION 0
22 PRINT_ITEM
23 PRINT_NEWLINE
87 24 **LOAD_NAME** 1 (c)
27 PRINT_ITEM
28 PRINT_NEWLINE
in strip game:
95 16 LOAD_GLOBAL 0 (locals)
19 CALL_FUNCTION 0
22 PRINT_ITEM
23 PRINT_NEWLINE
96 24 LOAD_GLOBAL 1 (c)
27 PRINT_ITEM
28 PRINT_NEWLINE
why is does this difference occur?
In this case, just use the global keyword:
a=3
def foo():
global a
a=4
foo()
print (a)
That modifies the outer scope, if it is global.
If the outer scope is a function, that is done with the nonlocal keyword instead - which was introduced with Python 3.0.
dynamic scoping
Changing the scope of the caller function however, is not a premise of Python, and is a language characteristic.
It can be done. But just by calling private C api's (to bake 'locals' values back into the fast local variables) and is definettely not a good practice.
DOing it through a magic decorator would also be possible, but the decorator would have to rewrite the bytecode in the inner function - by replacing each access to a 'nonlocal' variable by retrieving and updating the value on the caler locals, and, at the end of the function - https://programtalk.com/python-examples/ctypes.pythonapi.PyFrame_LocalsToFast/
Example
So, that said, here is a proof of concept. It is, of course, thread, and async unsafe as hell - but if the attributes in the proxy class
are promoted to threadlocals or context-local (pep 555), it should work.
it should be easy to adapt this to search for the local-variables to change up on the call stack (so that changes made in a sub-sub-call could change the grandparents locals, just as in dynamic scoped languages)
As stated in the question, there is no need to declare the variables on the caller as anything - they just must be normal local variables. However, this requires the declaration, on the decorated function, the variables I want to change on the caller scope as 'global', so that changing then will go through an object I can customize. If you can't have even this, you will indeed have to resort to rewrite the bytecode on the decorated function, or use the hooks put in place for writing debuggers (setting "trace on" on the code).
nb the exact behavior of changes locals() was specified to the language recently - prior to 3.8, IIRC, - and "locals_to_fast" seems to be
an stable enough API - but it might change in the future.
# Tested in Python 3.8.0
import ctypes
from functools import wraps
from sys import _getframe as getframe
from types import FunctionType
class GlobalProxy(dict):
__slots__ = ("parent", "frame", "mode")
def __init__(self, parent):
self.parent = parent
self.frame = None
self.mode = None
def __getitem__(self, name):
if self.mode == "target":
if name in self.frame.f_locals:
return self.frame.f_locals[name]
if name in self.parent:
return self.parent[name]
return getattr(self.parent["__builtins__"], name)
return super().__getitem__(name)
"""
# This is not run - Python's VM STORE_GLOBAL bypasses the custom __setitem__ (although __getitem__ above runs)
def __setitem__(self, name, value):
if name in self.frame.f_locals:
self.frame.f_locals[name] = value
bake_locals(self.frame)
self.parent[name] = value
"""
def bake_locals(self):
ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(self.frame), ctypes.c_int(1))
def save_changes(self):
self.mode = "inner"
target = self.frame.f_locals
target_names = set(target.keys())
for key in self:
if key in target_names:
target[key] = self[key]
else:
self.parent[key] = self[key]
self.bake_locals()
def caller_changer(func):
"""Makes all global variable changes on the decorated function affect _local_ variables on the callee function instead.
"""
code = func.__code__
# NB: for Python 2, these dunder-attributes for functions have other names.
# this is for Python 3
proxy = GlobalProxy(func.__globals__)
new_function = FunctionType(code, proxy, func.__name__, func.__defaults__, func.__closure__)
#wraps(func)
def wrapper(*args, **kw):
proxy.frame = getframe().f_back
proxy.mode = "target"
result = new_function(*args, **kw)
proxy.save_changes()
return result
wrapper.proxy = proxy
return wrapper
### Example and testing code:
#caller_changer
def blah():
global iwillchange
iwillchange = "new value"
def bleh():
iwillchange = "original value"
print(iwillchange)
blah()
print(iwillchange)
And, pasting all that on an IPython shell:
In [121]: bleh()
original value
new value
(I might add that it felt weird testing that, since the functions that
have the local variables changed do not need any decorator,
or any special declaration to the variables at all)
ok, so after several hours of sitting on this thing i've managed to write a solution, there are some major pitfalls when approaching this and i'll note them below
import inspect
import ctypes
import struct
import dis
import types
def dump(obj):
for attr in dir(obj):
print("obj.%s = %r" % (attr, getattr(obj, attr)))
def cgame():
a=3
c=5
print locals()
strip_game(a)
print locals()
def pgame():
a=3
c=5
print locals()
game(a)
print locals()
class empty_deco(object):
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
return self.f(*args, **kwargs)
debug_func = None
class inline_func(object):
def __init__(self, f):
self.f = f
# this is the price we pay for using 2.7
# also, there is a huge glraing issue here, which is what happens if the user TRIES to access a global variable?
#staticmethod
def replace_globals_with_name_lookups(co):
res = ""
code = list(co)
n = len(code)
i = 0
while i < n:
c = code[i]
op = ord(c)
if dis.opname[op] == "STORE_GLOBAL":
code[i] = chr(dis.opmap['STORE_NAME'])
elif dis.opname[op] == "DELETE_GLOBAL":
code[i] = chr(dis.opmap['DELETE_NAME'])
elif dis.opname[op] == "LOAD_GLOBAL":
code[i] = chr(dis.opmap['LOAD_NAME'])
i = i+1
if op >= dis.HAVE_ARGUMENT:
i = i+2
return "".join(code)
def __call__(self, *args, **kwargs):
init_exec_string = "inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))\n" + \
"inspect.stack()[1][0].f_locals.update(inspect.stack()[3][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[1][0]),ctypes.c_int(0))"
fini_exec_string = "inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\n" + \
"ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))"
co_stacksize = max(6, self.f.func_code.co_stacksize) # make sure we have enough space on the stack for everything
co_consts = self.f.func_code.co_consts +(init_exec_string, fini_exec_string)
init = "d" + struct.pack("H", len(strip_game.f.func_code.co_consts)) #LOAD_CONST init_exec_string
init += "d\x00\x00\x04U" # LOAD_CONST None, DUP_TOP, EXEC_STMT
init += "z" + struct.pack("H", len(self.f.func_code.co_code) + 4) #SETUP_FINALLY
fini = "Wd\x00\x00" # POP_BLOCK, LOAD_CONST None
fini += "d" + struct.pack("H", len(strip_game.f.func_code.co_consts) + 1) #LOAD_CONST fini_exec_string
fini += "d\x00\x00\x04UXd\x00\x00S" # LOAD_CONST None, DUP_TOP, EXEC_STMT, END_FINALLY, LOAD_CONST None, RETURN
co_code = init + self.replace_globals_with_name_lookups(self.f.func_code.co_code) + fini
co_lnotab = "\x00\x00\x0b" + self.f.func_code.co_lnotab[1:] # every error in init will be attributed to #inline_func, errors in the function will be treated as expected, errors in fini will be attributed to the last line probably.
new_code = types.CodeType(
self.f.func_code.co_argcount,
self.f.func_code.co_nlocals,
co_stacksize,
self.f.func_code.co_flags & ~(1), # optimized functions are problematic for us
co_code,
co_consts,
self.f.func_code.co_names,
self.f.func_code.co_varnames,
self.f.func_code.co_filename,
self.f.func_code.co_name,
self.f.func_code.co_firstlineno,
co_lnotab,
self.f.func_code.co_freevars,
self.f.func_code.co_cellvars,)
self.inline_f = types.FunctionType(new_code, self.f.func_globals, self.f.func_name, self.f.func_defaults, self.f.func_closure)
#dis.dis(self.inline_f)
global debug_func
debug_func = self.inline_f
return self.inline_f(*args, **kwargs)
#empty_deco
def game(b, a=4):
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))\ninspect.stack()[1][0].f_locals.update(inspect.stack()[3][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[1][0]),ctypes.c_int(0))")
try:
print "inner locals:"
print locals()
print c
return None
finally:
exec("inspect.stack()[3][0].f_locals.update(inspect.stack()[1][0].f_locals)\nctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(inspect.stack()[3][0]),ctypes.c_int(0))")
#inline_func
def strip_game(b, a=4):
print "inner locals:"
print locals()
print c
return None
where the acutal code needed lies in the class inline_func and some of the imports (maybe you can make them internal to the class? i'm really not sure)
so what does this whole thing do? well, it makes it so the code for strip_game and game are (nearly) identical, namely:
it inserts a function prologue which updates the locals of the caller, then adds to locals of the caller to the callee.
insert a try finally block around the function
changes every symbol lookup from a global lookup to a normal (name) lookup, after some thought i had realized that this doens't really have any effects
upon entering the finally block, updates the caller locals.
there are some major pitfalls making things like these, i'll list a few problems i've encountered:
cpython compiler_nameop function optimizes namespace lookup based on the simplicity of the given function, that means that it will optimize name lookups to global lookups if it can
changing the bytecode means affecting the debug-ability of the program, i had addressed this in the co_lnotab variable
for large functions this solution won't work as some of the opcodes would have to use extended_args: namely, the loads of the variables and the try-finally block (this point is solvable by using extended_args anyways...)
thank #jsbueno for putting in the time and pointing me to PyFrame_LocalsToFast.
P.S. this solution works for python 2.7.6, python has some issues when it comes to stability of the API, so for newer versions this might need to be fixed.
I have the following code, in which I simply have a decorator for caching a function's results, and as a concrete implementation, I used the Fibonacci function.
After playing around with the code, I wanted to print the cache variable, that's initiated in the cache wrapper.
(It's not because I suspect the cache might be faulty, I simply want to know how to access it without going into debug mode and put a breakpoint inside the decorator)
I tried to explore the fib_w_cache function in debug mode, which is supposed to actually be the wrapped fib_w_cache, but with no success.
import timeit
def cache(f, cache = dict()):
def args_to_str(*args, **kwargs):
return str(args) + str(kwargs)
def wrapper(*args, **kwargs):
args_str = args_to_str(*args, **kwargs)
if args_str in cache:
#print("cache used for: %s" % args_str)
return cache[args_str]
else:
val = f(*args, **kwargs)
cache[args_str] = val
return val
return wrapper
#cache
def fib_w_cache(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_w_cache(n-2) + fib_w_cache(n-1)
def fib_wo_cache(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_wo_cache(n-1) + fib_wo_cache(n-2)
print(timeit.timeit('[fib_wo_cache(i) for i in range(0,30)]', globals=globals(), number=1))
print(timeit.timeit('[fib_w_cache(i) for i in range(0,30)]', globals=globals(), number=1))
I admit this is not an "elegant" solution in a sense, but keep in mind that python functions are also objects. So with some slight modification to your code, I managed to inject the cache as an attribute of a decorated function:
import timeit
def cache(f):
def args_to_str(*args, **kwargs):
return str(args) + str(kwargs)
def wrapper(*args, **kwargs):
args_str = args_to_str(*args, **kwargs)
if args_str in wrapper._cache:
#print("cache used for: %s" % args_str)
return wrapper._cache[args_str]
else:
val = f(*args, **kwargs)
wrapper._cache[args_str] = val
return val
wrapper._cache = {}
return wrapper
#cache
def fib_w_cache(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_w_cache(n-2) + fib_w_cache(n-1)
#cache
def fib_w_cache_1(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_w_cache(n-2) + fib_w_cache(n-1)
def fib_wo_cache(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_wo_cache(n-1) + fib_wo_cache(n-2)
print(timeit.timeit('[fib_wo_cache(i) for i in range(0,30)]', globals=globals(), number=1))
print(timeit.timeit('[fib_w_cache(i) for i in range(0,30)]', globals=globals(), number=1))
print(fib_w_cache._cache)
print(fib_w_cache_1._cache) # to prove that caches are different instances for different functions
cache is of course a perfectly normal local variable in scope within the cache function, and a perfectly normal nonlocal cellvar in scope within the wrapper function, so if you want to access the value from there, you just do it—as you already are.
But what if you wanted to access it from somewhere else? Then there are two options.
First, cache happens to be defined at the global level, meaning any code anywhere (that hasn't hidden it with a local variable named cache) can access the function object.
And if you're trying to access the values of a function's default parameters from outside the function, they're available in the attributes of the function object. The inspect module docs explain the inspection-oriented attributes of each builtin type:
__defaults__ is a sequence of the values for all positional-or-keyword parameters, in order.
__kwdefaults__ is a mapping from keywords to values for all keyword-only parameters.
So:
>>> def f(a, b=0, c=1, *, d=2, e=3): pass
>>> f.__defaults__
(0, 1)
>>> f.__kwdefaults__
{'e': 3, 'd': 2}
So, for a simple case where you know there's exactly one default value and know which argument it belongs to, all you need is:
>>> cache.__defaults__[0]
{}
If you need to do something more complicated or dynamic, like get the default value for c in the f function above, you need to dig into other information—the only way to know that c's default value will be the second one in __defaults__ is to look at the attributes of the function's code object, like f.__code__.co_varnames, and figure it out from there. But usually, it's better to just use the inspect module's helpers. For example:
>>> inspect.signature(f).parameters['c'].default
1
>>> inspect.signature(cache).parameters['cache'].default
{}
Alternatively, if you're trying to access the cache from inside fib_w_cache, while there's no variable in lexical scope in that function body you can look at, you do know that the function body is only called by the decorator wrapper, and it is available there.
So, you can get your stack frame
frame = inspect.currentframe()
… follow it back to your caller:
back = frame.f_back
… and grab it from that frame's locals:
back.f_locals['cache']
It's worth noting that f_locals works like the locals function: it's actually a copy of the internal locals storage, so modifying it may have no effect, and that copy flattens nonlocal cell variables to regular local variables. If you wanted to access the actual cell variable, you'd have to grub around in things like back.f_code.co_freevars to get the index and then dig it out of the function object's __closure__. But usually, you don't care about that.
Just for a sake of completeness, python has caching decorator built-in in functools.lru_cache with some inspecting mechanisms:
from functools import lru_cache
#lru_cache(maxsize=None)
def fib_w_cache(n):
if n == 0: return 0
elif n == 1: return 1
else:
return fib_w_cache(n-2) + fib_w_cache(n-1)
print('fib_w_cache(10) = ', fib_w_cache(10))
print(fib_w_cache.cache_info())
Prints:
fib_w_cache(10) = 55
CacheInfo(hits=8, misses=11, maxsize=None, currsize=11)
I managed to find a solution (in some sense by #Patrick Haugh's advice).
I simply accessed cache.__defaults__[0] which holds the cache's dict.
The insights about the shared cache and how to avoid it we're also quite useful.
Just as a note, the cache dictionary can only be accessed through the cache function object. It cannot be accessed through the decorated functions (at least as far as I understand). It logically aligns well with the fact that the cache is shared in my implementation, where on the other hand, in the alternative implementation that was proposed, it is local per decorated function.
You can make a class into a wrapper.
def args_to_str(*args, **kwargs):
return str(args) + str(kwargs)
class Cache(object):
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args, **kwargs):
args_str = args_to_str(*args, **kwargs)
if args_str in self.cache:
return self.cache[args_str]
else:
val = self.func(*args, **kwargs)
self.cache[args_str] = val
return val
Each function has its own cache. you can access it by calling function.cache. This also allows for any methods you wish to attach to your function.
If you wanted all decorated functions to share the same cache, you could use a class variable instead of an instance variable:
class SharedCache(object):
cache = {}
def __init__(self, func):
self.func = func
#rest of the the code is the same
#SharedCache
def function_1(stuff):
things
I'm actually trying doing this in Java, but I'm in the process of teaching myself python and it made me wonder if there was an easy/clever way to do this with wrappers or something.
I want to know how many times a specific method was called inside another method. For example:
def foo(z):
#do something
return result
def bar(x,y):
#complicated algorithm/logic involving foo
return foobar
So for each call to bar with various parameters, I'd like to know how many times foo was called, perhaps with output like this:
>>> print bar('xyz',3)
foo was called 15 times
[results here]
>>> print bar('stuv',6)
foo was called 23 times
[other results here]
edit: I realize I could just slap a counter inside bar and dump it when I return, but it would be cool if there was some magic you could do with wrappers to accomplish the same thing. It would also mean I could reuse the same wrappers somewhere else without having to modify any code inside the method.
Sounds like almost the textbook example for decorators!
def counted(fn):
def wrapper(*args, **kwargs):
wrapper.called += 1
return fn(*args, **kwargs)
wrapper.called = 0
wrapper.__name__ = fn.__name__
return wrapper
#counted
def foo():
return
>>> foo()
>>> foo.called
1
You could even use another decorator to automate the recording of how many times a function is called inside another function:
def counting(other):
def decorator(fn):
def wrapper(*args, **kwargs):
other.called = 0
try:
return fn(*args, **kwargs)
finally:
print '%s was called %i times' % (other.__name__, other.called)
wrapper.__name__ = fn.__name__
return wrapper
return decorator
#counting(foo)
def bar():
foo()
foo()
>>> bar()
foo was called 2 times
If foo or bar can end up calling themselves, though, you'd need a more complicated solution involving stacks to cope with the recursion. Then you're heading towards a full-on profiler...
Possibly this wrapped decorator stuff, which tends to be used for magic, isn't the ideal place to be looking if you're still ‘teaching yourself Python’!
This defines a decorator to do it:
def count_calls(fn):
def _counting(*args, **kwargs):
_counting.calls += 1
return fn(*args, **kwargs)
_counting.calls = 0
return _counting
#count_calls
def foo(x):
return x
def bar(y):
foo(y)
foo(y)
bar(1)
print foo.calls
After your response - here's a way with a decorator factory...
import inspect
def make_decorators():
# Mutable shared storage...
caller_L = []
callee_L = []
called_count = [0]
def caller_decorator(caller):
caller_L.append(caller)
def counting_caller(*args, **kwargs):
# Returning result here separate from the count report in case
# the result needs to be used...
result = caller(*args, **kwargs)
print callee_L[0].__name__, \
'was called', called_count[0], 'times'
called_count[0] = 0
return result
return counting_caller
def callee_decorator(callee):
callee_L.append(callee)
def counting_callee(*args, **kwargs):
# Next two lines are an alternative to
# sys._getframe(1).f_code.co_name mentioned by Ned...
current_frame = inspect.currentframe()
caller_name = inspect.getouterframes(current_frame)[1][3]
if caller_name == caller_L[0].__name__:
called_count[0] += 1
return callee(*args, **kwargs)
return counting_callee
return caller_decorator, callee_decorator
caller_decorator, callee_decorator = make_decorators()
#callee_decorator
def foo(z):
#do something
return ' foo result'
#caller_decorator
def bar(x,y):
# complicated algorithm/logic simulation...
for i in xrange(x+y):
foo(i)
foobar = 'some result other than the call count that you might use'
return foobar
bar(1,1)
bar(1,2)
bar(2,2)
And here's the output (tested with Python 2.5.2):
foo was called 2 times
foo was called 3 times
foo was called 4 times