Develop Reference

Python warn if method has not been called

Suppose you want to call a method foo on object bar, but somehow while typing the method invocation you intuitively treated foo as a property and you typed bar.foo instead of bar.foo() (with parenthesis). Now, both are syntactically correct, so no error is raised, but semantically very different. It happened to me several times already (my experience in Ruby makes it even worse) and caused me dearly in terms of long and confusing debugging sessions.
Is there a way to make Python interpreter print a warning in such cases - whenever you access an attribute which is callable, but you haven't actually called it?
For the record - I thought about overriding __getattribute__ but it's messy and ultimately won't achieve the goal since function invocation via () happens after __getattribute__ has returned.

This can't be done in all cases because sometimes you don't want to call the method, e.g. you might want to store it as a callable to be used later, like callback = object.method.
But you can use static analysis tools such as pylint or PyCharm (my recommendation) that warn you if you write a statement that looks pointless, e.g. object.method without any assignment.
Furthermore if you write x = obj.get_x but meant get_x(), then later when you try to use x a static analysis tool may be able to warn you (if you're lucky) that x is a method but an instance of X is expected.

It was quite challenging, but I think i get it done! My code isn't very complicated, but you need to be well aware of metaclasses.
Metaclass and wrapper (WarnIfNotCalled.py):
class Notifier:
def __init__(self, name, obj, callback):
self.callback = callback
self.name = name
self.obj = obj
self.called = False
def __call__(self, *args, **kwargs):
self.callback(self.obj, *args, **kwargs)
self.called = True
def __del__(self):
if not self.called:
print("Warning! {} function hasn't been called!".format(self.name))
class WarnIfNotCalled(type):
def __new__(cls, name, bases, dct):
dct_func = {}
for name, val in dct.copy().items():
if name.startswith('__') or not callable(val):
continue
else:
dct_func[name] = val
del dct[name]
def getattr(self, name):
if name in dct_func:
return Notifier(name, self, dct_func[name])
dct['__getattr__'] = getattr
return super(WarnIfNotCalled, cls).__new__(cls, name, bases, dct)
It's very easy to use - just specify a metaclass
from WarnIfNotCalled import WarnIfNotCalled
class A(metaclass = WarnIfNotCalled):
def foo(self):
print("foo has been called")
def bar(self, x):
print("bar has been called and x =", x)
If you didn't forget to call these functions, everything works as usual
a = A()
a.foo()
a.bar(5)
Output:
foo has been called
bar has been called and x = 5
But if you DID forget:
a = A()
a.foo
a.bar
You see the following
Warning! foo function hasn't been called!
Warning! bar function hasn't been called!
Happy debugging!

Idiomatic way to inject parameters with python decorators

So, this is a 2 part question -
Is there an idiomatic way in python to inject a parameter into the function signature when using a decorator?
For example:
def _mydecorator(func):
def wrapped(someval, *args, **kwargs):
do_something(someval)
return func(*args, **kwargs)
return wrapped
#_mydecorator
def foo(thisval, thatval=None):
do_stuff()
The reason around this is when using SaltStack's runner modules You define funcs within the module, and you can call those functions via the 'salt-run' command. If the above example was a Salt runner module call 'bar', I could then run:
salt-run bar.foo aval bval
The salt-run imports the module and calls the function with the arguments you've given on the command line. Any function in the module that begins with a _ or that is in a class is ignored and cannot be run via salt-run.
So, I wanted to define something like a timeout decorator to control how long the function can run for.
I realize I could do something like:
#timeout(30)
def foo():
...
But I wanted to make that value configurable, so that I could run something like:
salt-run bar.foo 30 aval bval
salt-run bar.foo 60 aval bval
The above decorator works, but it feels like a hack, since it's changing the signature of the function and the user has no idea, unless they look at the decorator.
I have another case where I want to make a decorator for taking care of 'prechecks' before the functions in the Salt runner execute. However, the precheck needs a piece of information from the function it's decorating. Here's an example:
def _precheck(func):
def wrapper(*args, **kwargs):
ok = False
if len(args) > 0:
ok = run_prechecks(args[0])
else:
ok = run_prechecks(kwargs['record_id'])
if ok:
func(*args, **kwargs)
return wrapper
#_precheck
def foo(record_id, this_val):
do_stuff()
This also seems hackish, since it requires that the function that's being decorated, a) has a parameter called 'record_id' and that b) it's the first argument.
Now, because I'm writing all these functions, it's not a huge deal, but seems like there's probably a better way of doing this ( like not using decorators to try and solve this )

The way to dynamically define decorator arguments is not using the syntactic sugar (#). Like this:
func = dec(dec_arguments)(func_name)(func_arguments)
import json
import sys
foo = lambda record_id, thatval=None: do_stuff(record_id, thatval)
def do_stuff(*args, **kwargs):
# python3
print(*args, json.dumps(kwargs))
def _mydecorator(timeout):
print('Timeout: %s' % timeout)
def decorator(func):
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
return wrapped
return decorator
if __name__ == '__main__':
_dec_default = 30
l_argv = len(sys.argv)
if l_argv == 1:
# no args sent
sys.exit('Arguments missing')
elif l_argv == 2:
# assuming record_id is a required argument
_dec_arg = _dec_default
_args = 1
else:
# three or more args: filename 1 2 [...]
# consider using optional keyword argument `timeout`
# otherwise in combination with another optional argument it's a recipe for disaster
# if only two arguments will be given - with current logic it will be tested for `timeoutedness`
try:
_dec_arg = int(sys.argv[1])
_args = 2
except (ValueError, IndexError):
_dec_arg = _dec_default
_args = 1
foo = _mydecorator(_dec_arg)(foo)(*sys.argv[_args:])

There is no idiomatic way to do this in python 3.7 as far as I know. Indeed #functools.wraps only works when you do not modify the signature (see what does functools.wraps do ?)
However there is a way to do the same as #functools.wraps (exposing the full signature, preserving the __dict__ and docstring): #makefun.wraps. With this drop-in replacement for #wraps, you can edit the exposed signature.
from makefun import wraps
def _mydecorator(func):
#wraps(func, prepend_args="someval")
def wrapped(someval, *args, **kwargs):
print("wrapper executes with %r" % someval)
return func(*args, **kwargs)
return wrapped
#_mydecorator
def foo(thisval, thatval=None):
"""A foo function"""
print("foo executes with thisval=%r thatval=%r" % (thisval, thatval))
# let's check the signature
help(foo)
# let's test it
foo(5, 1)
Yields:
Help on function foo in module __main__:
foo(someval, thisval, thatval=None)
A foo function
wrapper executes with 5
foo executes with thisval=1 thatval=None
You can see that the exposed signature contains the prepended argument.
The same mechanism applies for appending and removing arguments, as well as editing signatures more deeply. See makefun documentation for details (I'm the author by the way ;) )

variables as decorator arguments

I have found out that decorator arguments are passed at decorator definition rather than invocation like with functions.
Now I wonder if it is possible to make the decorater get the value of a variable at runtime like this, the decorater should print the current value of state instead of the one it head at definition:
def deco(msg):
def decorater(func):
def wrapper(*args, **kwargs):
print msg
func(*args, **kwargs)
return wrapper
return decorater
def func():
local = {
"state": None
}
#deco(local["state"])
def test():
pass
def setState(newState):
local["state"] = newState
setState("start")
test()
setState("test")
test()
func()

In your example, deco() is a decorator factory; you're creating the decorator which will then immediately be invoked. More generally, you invoke a decorator at the time that you're defining the function that you're decorating.
You can do what you're trying to do with minimal changes by just not passing in state, and accessing it as a global from within wrapper(), in which case you don't need deco(); you could just use #decorator directly. That said, I think there are better ways to do what you're trying to do.

John you should read this. In python, the variable is not the object. You question, is it "possible to make the decorator get the value of a variable at runtime", doesn't make sense because of python's scoping rules. The decorator function does not generally have access to the scope where state is defined. There are several ways you could get the behavior you want though.
Without knowing the specifics of what you're trying to do, here are two that might work. The first uses closure:
state = None
def with_closure(f):
def helper(*args, **kwargs):
# state is in scope for this function
print "Current state is: {}.".format(state)
return f(*args, **kwargs)
return helper
#with_closure
def foo():
return "something"
Or you could make an object to keep track of state:
class StateHolder:
def set_state(self, state):
self.state = state
def with_state_object(state_object):
def decorator(f):
def helper(*args, **kwargs):
print "Current state is: {}.".format(state_object.state)
return f(*args, **kwargs)
return helper
return decorator
global_state = StateHolder()
global_state.set_state("some state")
#with_state_object(global_state)
def foo():
return "something"

Preventing function (or decorator) from being nested

I've got some code in a decorator that I only want run once. Many other functions (utility and otherwise) will be called later down the line, and I want to ensure that other functions that may have this decorator aren't accidentally used way down in the nest of function calls.
I also want to be able to check, at any point, whether or not the current code has been wrapped in the decorator or not.
I've written this, but I just wanted to see if anyone else can think of a better/more elegant solution than checking for the (hopefully!) unique function name in the stack.
import inspect
def my_special_wrapper(fn):
def my_special_wrapper(*args, **kwargs):
""" Do some magic, only once! """
# Check we've not done this before
for frame in inspect.stack()[1:]: # get stack, ignoring current!
if frame[3] == 'my_special_wrapper':
raise StandardError('Special wrapper cannot be nested')
# Do magic then call fn
# ...
fn(*args, **kwargs)
return my_special_wrapper
def within_special_wrapper():
""" Helper to check that the function has been specially wrapped """
for frame in inspect.stack():
if frame[3] == 'my_special_wrapper':
return True
return False
#my_special_wrapper
def foo():
print within_special_wrapper()
bar()
print 'Success!'
#my_special_wrapper
def bar():
pass
foo()

Here is an example of using a global for this task - in what I believe is a relatively safe way:
from contextlib import contextmanager
from functools import wraps
_within_special_context = False
#contextmanager
def flag():
global _within_special_context
_within_special_context = True
try:
yield
finally:
_within_special_context = False
#I'd argue this would be best replaced by just checking the variable, but
#included for completeness.
def within_special_wrapper():
return _within_special_context
def my_special_wrapper(f):
#wraps(f)
def internal(*args, **kwargs):
if not _within_special_context:
with flag():
...
f(*args, **kwargs)
else:
raise Exception("No nested calls!")
return internal
#my_special_wrapper
def foo():
print(within_special_wrapper())
bar()
print('Success!')
#my_special_wrapper
def bar():
pass
foo()
Which results in:
True
Traceback (most recent call last):
File "/Users/gareth/Development/so/test.py", line 39, in <module>
foo()
File "/Users/gareth/Development/so/test.py", line 24, in internal
f(*args, **kwargs)
File "/Users/gareth/Development/so/test.py", line 32, in foo
bar()
File "/Users/gareth/Development/so/test.py", line 26, in internal
raise Exception("No nested calls!")
Exception: No nested calls!
Using a context manager ensures that the variable is unset. You could just use try/finally, but if you want to modify the behaviour for different situations, the context manager can be made to be flexible and reusable.

The obvious solution is to have special_wrapper set a global flag, and just skip its magic if the flag is set.
This is about the only good use of a global variable - to allow a single piece of code to store information that is only used within that code, but which needs to survive the life of execution in that code.
It doesn't need to be set in global scope. The function could set the flag on itself, for example, or on any object or class, as long as nothing else will touch it.
As noted by Lattyware in comments, you'll want to use either a try/except, or perhaps even better, a context manager to ensure the variable is unset.
Update: If you need the wrapped code to be able to check if it is wrapped, then provide a function which returns the value of the flag. You might want to wrap it all up with a class for neatness.
Update 2: I see you're doing this for transaction management. There are probably already libraries which do this. I strongly recommend that you at least look at their code.

While my solution technically works, it requires a manual reset of the decorator, but you could very well modify things such that the outermost function is instead a class (with the instances being the wrappers of the decorated functions passed to it in __init__), and have reset() being called in __exit__(), which would then allow you to use the with statement to create the decorator to be usable only once within the context. Also note that it requires Python 3 due to the nonlocal keyword, but that can easily be adapted to 2.7 with a dict in place of the flag variable.
def once_usable(decorator):
"Apply this decorator function to the decorator you want to be usable only once until it is reset."
def outer_wrapper():
flag = False
def inner_wrapper(*args, **kwargs):
nonlocal flag
if not flag:
flag = True
return decorator(*args, **kwargs)
else:
print("Decorator currently unusable.") # raising an Error also works
def decorator_reset():
nonlocal flag
flag = False
return (inner_wrapper, decorator_reset)
return outer_wrapper()
Testing:
>>> def a(aa):
return aa*2
>>> def b(bb):
def wrapper(*args, **kwargs):
print("Decorated.")
return bb(*args, **kwargs)
return wrapper
>>> dec, reset = once_usable(b)
>>> aa = dec(a)
>>> aa(22)
Decorated.
44
>>> aaa = dec(a)
Decorator currently unusable.
>>> reset()
>>> aaa = dec(a)
>>> aaa(11)
Decorated.
22

toggling decorators

What's the best way to toggle decorators on and off, without actually going to each decoration and commenting it out? Say you have a benchmarking decorator:
# deco.py
def benchmark(func):
def decorator():
# fancy benchmarking
return decorator
and in your module something like:
# mymodule.py
from deco import benchmark
class foo(object):
#benchmark
def f():
# code
#benchmark
def g():
# more code
That's fine, but sometimes you don't care about the benchmarks and don't want the overhead. I have been doing the following. Add another decorator:
# anothermodule.py
def noop(func):
# do nothing, just return the original function
return func
And then comment out the import line and add another:
# mymodule.py
#from deco import benchmark
from anothermodule import noop as benchmark
Now benchmarks are toggled on a per-file basis, having only to change the import statement in the module in question. Individual decorators can be controlled independently.
Is there a better way to do this? It would be nice to not have to edit the source file at all, and to specify which decorators to use in which files elsewhere.

You could add the conditional to the decorator itself:
def use_benchmark(modname):
return modname == "mymodule"
def benchmark(func):
if not use_benchmark(func.__module__):
return func
def decorator():
# fancy benchmarking
return decorator
If you apply this decorator in mymodule.py, it will be enabled; if you apply it in othermodule.py, it will not be enabled.

I've been using the following approach. It's almost identical to the one suggested by CaptainMurphy, but it has the advantage that you don't need to call the decorator like a function.
import functools
class SwitchedDecorator:
def __init__(self, enabled_func):
self._enabled = False
self._enabled_func = enabled_func
#property
def enabled(self):
return self._enabled
#enabled.setter
def enabled(self, new_value):
if not isinstance(new_value, bool):
raise ValueError("enabled can only be set to a boolean value")
self._enabled = new_value
def __call__(self, target):
if self._enabled:
return self._enabled_func(target)
return target
def deco_func(target):
"""This is the actual decorator function. It's written just like any other decorator."""
def g(*args,**kwargs):
print("your function has been wrapped")
return target(*args,**kwargs)
functools.update_wrapper(g, target)
return g
# This is where we wrap our decorator in the SwitchedDecorator class.
my_decorator = SwitchedDecorator(deco_func)
# Now my_decorator functions just like the deco_func decorator,
# EXCEPT that we can turn it on and off.
my_decorator.enabled=True
#my_decorator
def example1():
print("example1 function")
# we'll now disable my_decorator. Any subsequent uses will not
# actually decorate the target function.
my_decorator.enabled=False
#my_decorator
def example2():
print("example2 function")
In the above, example1 will be decorated, and example2 will NOT be decorated. When I have to enable or disable decorators by module, I just have a function that makes a new SwitchedDecorator whenever I need a different copy.

I think you should use a decorator a to decorate the decorator b, which let you switch the decorator b on or off with the help of a decision function.
This sounds complex, but the idea is rather simple.
So let's say you have a decorator logger:
from functools import wraps
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
This is a very boring decorator and I have a dozen or so of these, cachers, loggers, things which inject stuff, benchmarking etc. I could easily extend it with an if statement, but this seems to be a bad choice; because then I have to change a dozen of decorators, which is not fun at all.
So what to do? Let's step one level higher. Say we have a decorator, which can decorate a decorator? This decorator would look like this:
#point_cut_decorator(logger)
def my_oddly_behaving_function
This decorator accepts logger, which is not a very interesting fact. But it also has enough power to choose if the logger should be applied or not to my_oddly_behaving_function. I called it point_cut_decorator, because it has some aspects of aspect oriented programming. A point cut is a set of locations, where some code (advice) has to be interwoven with the execution flow. The definitions of point cuts is usually in one place. This technique seems to be very similar.
How can we implement it decision logic. Well I have chosen to make a function, which accepts the decoratee, the decorator, file and name, which can only say if a decorator should be applied or not. These are the coordinates, which are good enough to pinpoint the location very precisely.
This is the implementation of point_cut_decorator, I have chosen to implement the decision function as a simple function, you could extend it to let it decide from your settings or configuration, if you use regexes for all 4 coordinates, you will end up with something very powerful:
from functools import wraps
myselector is the decision function, on true a decorator is applied on false it is not applied. Parameters are the filename, the module name, the decorated object and finally the decorator. This allows us to switch of behaviour in a fine grained manner.
def myselector(fname, name, decoratee, decorator):
print fname
if decoratee.__name__ == "test" and fname == "decorated.py" and decorator.__name__ == "logger":
return True
return False
This decorates a function, checks myselector and if myselector says go on, it will apply the decorator to the function.
def point_cut_decorator(d):
def innerdecorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
if myselector(__file__, __name__, f, d):
ps = d(f)
return ps(*args, **kwargs)
else:
return f(*args, **kwargs)
return wrapper
return innerdecorator
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
And this is how you use it:
#point_cut_decorator(logger)
def test(a):
print "hello"
return "world"
test(1)
EDIT:
This is the regular expression approach I talked about:
from functools import wraps
import re
As you can see, I can specify somewhere a couple of rules, which decides a decorator should be applied or not:
rules = [{
"file": "decorated.py",
"module": ".*",
"decoratee": ".*test.*",
"decorator": "logger"
}]
Then I loop over all rules and return True if a rule matches or false if a rule doesn't matches. By making rules empty in production, this will not slow down your application too much:
def myselector(fname, name, decoratee, decorator):
for rule in rules:
file_rule, module_rule, decoratee_rule, decorator_rule = rule["file"], rule["module"], rule["decoratee"], rule["decorator"]
if (
re.match(file_rule, fname)
and re.match(module_rule, name)
and re.match(decoratee_rule, decoratee.__name__)
and re.match(decorator_rule, decorator.__name__)
):
return True
return False

Here is what I finally came up with for per-module toggling. It uses #nneonneo's suggestion as a starting point.
Random modules use decorators as normal, no knowledge of toggling.
foopkg.py:
from toggledeco import benchmark
#benchmark
def foo():
print("function in foopkg")
barpkg.py:
from toggledeco import benchmark
#benchmark
def bar():
print("function in barpkg")
The decorator module itself maintains a set of function references for all decorators that have been disabled, and each decorator checks for its existence in this set. If so, it just returns the raw function (no decorator). By default the set is empty (everything enabled).
toggledeco.py:
import functools
_disabled = set()
def disable(func):
_disabled.add(func)
def enable(func):
_disabled.discard(func)
def benchmark(func):
if benchmark in _disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco
The main program can toggle individual decorators on and off during imports:
from toggledeco import benchmark, disable, enable
disable(benchmark) # no benchmarks...
import foopkg
enable(benchmark) # until they are enabled again
import barpkg
foopkg.foo() # no benchmarking
barpkg.bar() # yes benchmarking
reload(foopkg)
foopkg.foo() # now with benchmarking
Output:
function in foopkg
--> benchmarking bar((),{})
function in barpkg
<-- done
--> benchmarking foo((),{})
function in foopkg
<-- done
This has the added bug/feature that enabling/disabling will trickle down to any submodules imported from modules imported in the main function.
EDIT:
Here's class suggested by #nneonneo. In order to use it, the decorator must be called as a function ( #benchmark(), not #benchmark ).
class benchmark:
disabled = False
#classmethod
def enable(cls):
cls.disabled = False
#classmethod
def disable(cls):
cls.disabled = True
def __call__(cls,func):
if cls.disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco

I would implement a check for a config file inside the decorator's body. If benchmark has to be used according to the config file, then I would go to your current decorator's body. If not, I would return the function and do nothing more. Something in this flavor:
# deco.py
def benchmark(func):
if config == 'dontUseDecorators': # no use of decorator
# do nothing
return func
def decorator(): # else call decorator
# fancy benchmarking
return decorator
What happens when calling a decorated function ? # in
#benchmark
def f():
# body comes here
is syntactic sugar for this
f = benchmark(f)
so if config wants you to overlook decorator, you are just doing f = f() which is what you expect.

I don't think anyone has suggested this yet:
benchmark_modules = set('mod1', 'mod2') # Load this from a config file
def benchmark(func):
if not func.__module__ in benchmark_modules:
return func
def decorator():
# fancy benchmarking
return decorator
Each function or method has a __module__ attribute that is the name of the module where the function is defined. Create a whitelist (or blacklist if you prefer) of modules where benchmarking is to occur, and if you don't want to benchmark that module just return the original undecorated function.

another straight way:
# mymodule.py
from deco import benchmark
class foo(object):
def f():
# code
if <config.use_benchmark>:
f = benchmark(f)
def g():
# more code
if <config.use_benchmark>:
g = benchmark(g)

Here's a workaround to automatically toggle a decorator (here: #profile used by line_profiler):
if 'profile' not in __builtins__ or type(__builtins__) is not dict: profile=lambda x: None;
More info
This conditional (only if needed) instantiation of the profile variable (as an empty lambda function) prevents raising NameError when trying to import our module with user-defined functions where the decorator #profile is applied to every profiled user function. If I ever want to use the decorator for profiling - it will still work, not being overwritten (already existing in an external script kernprof that contains this decorator).