Develop Reference

Using classes as method decorators [duplicate]

This question already has answers here:
How can I decorate an instance method with a decorator class?
(2 answers)
Closed 4 years ago.
While there are plenty of resources about using classes as decorators, I haven't been able to find any that deal with the problem of decorating methods. The goal of this question is to fix that. I will post my own solution, but of course everyone else is invited to post theirs as well.
Why the "standard" implementation doesn't work
The problem with the standard decorator class implementation is that python will not create a bound method of the decorated function:
class Deco:
def __init__(self, func):
self.func= func
def __call__(self, *args):
self.func(*args)
class Class:
#Deco
def hello(self):
print('hello world')
Class().hello() # throws TypeError: hello() missing 1 required positional argument: 'self'
A method decorator needs to overcome this hurdle.
Requirements
Taking the classes from the previous example, the following things are expected to work:
>>> i= Class()
>>> i.hello()
hello world
>>> i.hello
<__main__.Deco object at 0x7f4ae8b518d0>
>>> Class.hello is Class().hello
False
>>> Class().hello is Class().hello
False
>>> i.hello is i.hello
True
Ideally, the function's __doc__ and signature and similar attributes are preserved as well.

Usually when a method is accessed as some_instance.some_method(), python's descriptor protocol kicks in and calls some_method.__get__(), which returns a bound method. However, because the method has been replaced with an instance of the Deco class, that does not happen - because Deco is not a descriptor. In order to make Deco work as expected, it must implement a __get__ method that returns a bound copy of itself.
Implementation
Here's basic "do nothing" decorator class:
import inspect
import functools
from copy import copy
class Deco(object):
def __init__(self, func):
self.__self__ = None # "__self__" is also used by bound methods
self.__wrapped__ = func
functools.update_wrapper(self, func)
def __call__(self, *args, **kwargs):
# if bound to an object, pass it as the first argument
if self.__self__ is not None:
args = (self.__self__,) + args
#== change the following line to make the decorator do something ==
return self.__wrapped__(*args, **kwargs)
def __get__(self, instance, owner):
if instance is None:
return self
# create a bound copy
bound = copy(self)
bound.__self__ = instance
# update __doc__ and similar attributes
functools.update_wrapper(bound, self.__wrapped__)
# add the bound instance to the object's dict so that
# __get__ won't be called a 2nd time
setattr(instance, self.__wrapped__.__name__, bound)
return bound
To make the decorator do something, add your code in the __call__ method.
Here's one that takes parameters:
class DecoWithArgs(object):
#== change the constructor's parameters to fit your needs ==
def __init__(self, *args):
self.args = args
self.__wrapped__ = None
self.__self__ = None
def __call__(self, *args, **kwargs):
if self.__wrapped__ is None:
return self.__wrap(*args, **kwargs)
else:
return self.__call_wrapped_function(*args, **kwargs)
def __wrap(self, func):
# update __doc__ and similar attributes
functools.update_wrapper(self, func)
return self
def __call_wrapped_function(self, *args, **kwargs):
# if bound to an object, pass it as the first argument
if self.__self__ is not None:
args = (self.__self__,) + args
#== change the following line to make the decorator do something ==
return self.__wrapped__(*args, **kwargs)
def __get__(self, instance, owner):
if instance is None:
return self
# create a bound copy of this object
bound = copy(self)
bound.__self__ = instance
bound.__wrap(self.__wrapped__)
# add the bound decorator to the object's dict so that
# __get__ won't be called a 2nd time
setattr(instance, self.__wrapped__.__name__, bound)
return bound
An implementation like this lets us use the decorator on methods as well as functions, so I think it should be considered good practice.

Lazy-loading variables using overloaded decorators

I have a state object that represents a system. Properties within the state object are populated from [huge] text files. As not every property is accessed every time a state instance, is created, it makes sense to lazily load them.:
class State:
def import_positions(self):
self._positions = {}
# Code which populates self._positions
#property
def positions(self):
try:
return self._positions
except AttributeError:
self.import_positions()
return self._positions
def import_forces(self):
self._forces = {}
# Code which populates self._forces
#property
def forces(self):
try:
return self._forces
except AttributeError:
self.import_forces()
return self._forces
There's a lot of repetitive boilerplate code here. Moreover, sometimes an import_abc can populate a few variables (i.e. import a few variables from a small data file if its already open).
It makes sense to overload #property such that it accepts a function to "provide" that variable, viz:
class State:
def import_positions(self):
self._positions = {}
# Code which populates self._positions
#lazyproperty(import_positions)
def positions(self):
pass
def import_forces(self):
self._forces = {}
# Code which populates self._forces and self._strain
#lazyproperty(import_forces)
def forces(self):
pass
#lazyproperty(import_forces)
def strain(self):
pass
However, I cannot seem to find a way to trace exactly what method are being called in the #property decorator. As such, I don't know how to approach overloading #property into my own #lazyproperty.
Any thoughts?

Maybe you want something like this. It's a sort of simple memoization function combined with #property.
def lazyproperty(func):
values = {}
def wrapper(self):
if not self in values:
values[self] = func(self)
return values[self]
wrapper.__name__ = func.__name__
return property(wrapper)
class State:
#lazyproperty
def positions(self):
print 'loading positions'
return {1, 2, 3}
s = State()
print s.positions
print s.positions
Which prints:
loading positions
set([1, 2, 3])
set([1, 2, 3])
Caveat: entries in the values dictionary won't be garbage collected, so it's not suitable for long-running programs. If the loaded value is immutable across all classes, it can be stored on the function object itself for better speed and memory use:
try:
return func.value
except AttributeError:
func.value = func(self)
return func.value

I think you can remove even more boilerplate by writing a custom descriptor class that decorates the loader method. The idea is to have the descriptor itself encode the lazy-loading logic, meaning that the only thing you define in an actual method is the loader itself (which is the only thing that, apparently, really does have to vary for different values). Here's an example:
class LazyDesc(object):
def __init__(self, func):
self.loader = func
self.secretAttr = '_' + func.__name__
def __get__(self, obj, cls):
try:
return getattr(obj, self.secretAttr)
except AttributeError:
print("Lazily loading", self.secretAttr)
self.loader(obj)
return getattr(obj, self.secretAttr)
class State(object):
#LazyDesc
def positions(self):
self._positions = {'some': 'positions'}
#LazyDesc
def forces(self):
self._forces = {'some': 'forces'}
Then:
>>> x = State()
>>> x.forces
Lazily loading _forces
{'some': 'forces'}
>>> x.forces
{'some': 'forces'}
>>> x.positions
Lazily loading _positions
{'some': 'positions'}
>>> x.positions
{'some': 'positions'}
Notice that the "lazy loading" message was printed only on the first access for each attribute. This version also auto-creates the "secret" attribute to hold the real data by prepending an underscore to the method name (i.e., data for positions is stored in _positions. In this example, there's no setter, so you can't do x.positions = blah (although you can still mutate the positions with x.positions['key'] = val), but the approach could be extended to allow setting as well.
The nice thing about this approach is that your lazy logic is transparently encoded in the descriptor __get__, meaning that it easily generalizes to other kinds of boilerplate that you might want to abstract away in a similar manner.

However, I cannot seem to find a way to trace exactly what method are
being called in the #property decorator.
property is actually a type (whether you use it with the decorator syntax of not is orthogonal), which implements the descriptor protocol (https://docs.python.org/2/howto/descriptor.html). An overly simplified (I skipped the deleter, doc and quite a few other things...) pure-python implementation would look like this:
class property(object):
def __init__(self, fget=None, fset=None):
self.fget = fget
self.fset = fset
def setter(self, func):
self.fset = func
return func
def __get__(self, obj, type=None):
return self.fget(obj)
def __set__(self, obj, value):
if self.fset:
self.fset(obj, value)
else:
raise AttributeError("Attribute is read-only")
Now overloading property is not necessarily the simplest solution. In fact there are actually quite a couple existing implementations out there, including Django's "cached_property" (cf http://ericplumb.com/blog/understanding-djangos-cached_property-decorator.html for more about it) and pydanny's "cached-property" package (https://pypi.python.org/pypi/cached-property/0.1.5)

Python decorator for attribute and method?

Is it possible to have a decorator that makes a method work like an attribute if values are assigned to it using class.something = 2 and work like a method if it is called like class.something(2, True)?
What I currently have
To be more concrete, I currently have the following
class attributeSetter(object):
''' Makes functions appear as attributes. Takes care of autologging.'''
def __init__(self, func):
self.func = func
def __set__(self, obj, value):
return self.func(obj, value)
def __repr__(self):
return repr(self.__getattribute__)
So with this class with a decorated method:
class myClass(object):
#attributeSetter # decorated!
def myAttrib(self, value, add=False, subtract=False):
if add:
value += 2
if subtract:
value -= 2
self.__dict__['myAttrib'] = value
I can do this:
instance = myClass()
instance.myAttrib = 2 # sets the value to 2
instance.myAttrib *= 4 # now 8
print instance.myAttrib # 8
What I want
But I want, in addition, to be able to do this:
instance.myAttrib(3, add=True) # now 8 + 3 + 2 = 13
instance.myAttrib(0, subtact=True) # now 13 + 0 - 2 = 11
print instance.myAttrib # 11
My hunch is that I just have to add a __call__ to the attributeSetter decorator like this:
def __call__(self, *args, **kwargs):
self.func(*args, **kwargs)
but then it won't let me set values using the "=" syntax.
Why?
I'm co-developing PsychoPy (a python module for stimulus delivery in psychphysics) and we want do to some processing when each stimulus parameter is set, hence the decorator to avoid setter/getter methods. We log each attribute change by default but in some cases, users want to disable logging of this particular setting with an extra log=False argument, or provide more arguments. Hence it would be nice in those cases if could just use the attribute like a function/method, which it really is in the first place.

You'll need to implement a __get__ method returning a callable; your decorator already provides a descriptor object, simply make it work when accessing as an attribute.
This can be as simple as turning around and binding the wrapped function (so you get a bound method):
class attributeSetter(object):
''' Makes functions appear as attributes. Takes care of autologging.'''
def __init__(self, func):
self.func = func
def __get__(self, instance, owner):
return self.func.__get__(instance, owner)
def __set__(self, obj, value):
return self.func(obj, value)
However, this makes it incompatible with simply accessing the attribute! instance.myAttrib now returns a bound method! You cannot have it both ways here; methods are simply bound attributes (so attributes that passed through the descriptor protocol), that happen to be callable.
You could of course return a proxy object from __get__; one that implements a __call__ method and otherwise tries to act as much as possible as the underlying managed instance attribute (what your function stored in self.__dict__['myAttrib']); but this path is fraught with problems as a proxy object can never truly be the underlying attribute value.

How to replace __str__ for a function

I want to change the string representation of a python function to be just the function name.
Eg for some function
def blah(x):
...
str(blah) currently gives
<function blah at 0x10127b578>
So I replace __str__ like this:
blah.__str__=lambda: 'blah'
but it doesn't get called by str(blah).
Is it possible to change __str__ for a function?

Replacing __str__ like that never actually works on any type of object. The builtin function type isn't special:
>>> class Foo(object):
pass
>>> f = Foo()
>>> f.__str__ = lambda: 'f'
>>> str(f)
'<__main__.Foo object at 0x0000000002D13F28>'
>>> f.__str__()
'f'
The str builtin doesn't lookup __str__ via the usual lookup protocol, it skips straight to looking at the class of its argument. So you can't "override" the __str__ method on an individual object; you have to set it on a class to apply to all objects of that class.1
But then again, the builtin function type isn't special. You can make a class that behaves mostly like a function, and use that instead:
class Function(object):
def __init__(self, raw_function):
self.raw_function = raw_function
def __call__(self, *args, **kwargs):
return self.raw_function(*args, **kwargs)
def __str__(self):
return self.raw_function.__name__
The __call__ method means you can call objects of this class exactly as if they were functions. This one just passes whatever arguments it receives straight through to the underlying raw function and returns whatever it returns (or throws whatever exceptions it throws). And since this class takes a single function as an argument, it also fits the pattern for a decorator:
#Function
def blah(x):
return x + 1
With that:
>>> blah
<__main__.Function object at 0x0000000002D13EB8>
>>> str(blah)
'blah'
>>> blah(33)
34
1 The builtin function type is special in some regards (I lied), one of which being that you can't assign to its attributes (that's the attributes of the class itself, not the attributes of any particular function object). So before you think maybe it would be great if you could monkeypatch the builtin function type so that str worked the way you want on all functions, that doesn't work. It's probably for the best; modifying things as global and fundamental as the builtin function type would be a great way to invent some really bizarre bugs.

As Joran said:
class NamedFunction:
def __init__(self, name, f):
self.f = f
self.name = name
def __call__(self, *args, **kwargs):
return self.f(*args, **kwargs)
def __str__(self):
return self.name
f = NamedFunction("lambda: 'blah'", lambda: 'blah')
print(f())
print(f)

class FNMagic:
def __init__(self,fn,fn_name):
self.fn = fn
self.fn_name = fn_name
def __call__(self,*args,**kwargs):
return self.fn(*args,**kwargs)
def __str__(self):
return self.fn_name
def blah(x):
return x
blah = FNMagic(blah,"blah!")
print blah
you could make a simple decorator
class NamedFn:
def __init__(self,name):
self.fn_name = name
def __call__(self,fn):
return FNMagic(fn,self.fn_name)
#NamedFn("some_name!!!")
def blah2(x,y):
return x*y
print blah2

As noted already, the answer is no. If you just want to get the name of a function as a string, you can use blah.__name__.

Writing decorator for pytest test method

Assuming the following structure:
class SetupTestParam(object):
def setup_method(self, method):
self.foo = bar()
#pytest.fixture
def some_fixture():
self.baz = 'foobar'
I use SetupTestParam as a parent class for test classes.
class TestSomething(SetupTestParam):
def test_a_lot(self, some_fixture):
with self.baz as magic:
with magic.fooz as more_magic:
blah = more_magic.much_more_magic() # repetative bleh
... # not repetative code here
assert spam == 'something cool'
Now, writing tests gets repetitive (with statement usage) and I would like to write a decorator to reduce the number of code lines. But there is a problem with pytest and the function signature.
I found out library which should be helpful but I can't manage to get it to work.
I made a classmethod in my SetupTestParam class.
#classmethod
#decorator.decorator
def this_is_decorator(cls, f):
def wrapper(self, *args, **kw):
with self.baz as magic:
with magic.fooz as more_magic:
blah = more_magic.much_more_magic() # repetative bleh
return f(self, *args)
return wrapper
After I decorate the test_a_lot method, I receive the error TypeError: transaction_decorator() takes exactly 1 argument (2 given)
Can someone explain me please what am I doing wrong? (I assume there is a problem with self from the test method?)

Chaining decorators is not the simplest thing to do. One solution might be to separate the two decorators. Keep the classmethod but move decorator.decorator to the end:
#classmethod
def this_is_decorator(cls, f):
def wrapper(self, *args, **kw):
with self.baz as magic:
with magic.fooz as more_magic:
blah = more_magic.much_more_magic() # repetative bleh
return f(self, *args)
return decorator.decorator(wrapper, f)
Maybe this works for you.

After some tweaking and realizing that I need to pass a parameter to decorator I choosed to write it as a class:
class ThisIsDecorator(object):
def __init__(self, param):
self.param = param # Parameter may vary with the function being decorated
def __call__(self, fn):
wraps(fn) # [1]
def wrapper(fn, fn_self, *args): # [2] fn_self refers to original self param from function fn (test_a_lot) [2]
with fn_self.baz as fn_self.magic: # I pass magic to fn_self to make magic accesible in function fn (test_a_lot)
with fn_self.magic.fooz as more_magic:
blah = self.param.much_more_magic() # repetative bleh
return fn(fn_self, *args)
return decorator.decorator(wrapper, fn)
[1] I use wraps to have original fn __name__, __module__ and __doc__.
[2] Params passed to wrapper were self = <function test_a_lot at 0x24820c8> args = (<TestSomething object at 0x29c77d0>, None, None, None, None), kw = {} so I took out the args[0] as a fn_self.
Original version (without passing a parameter):
#classmethod
def this_is_decorator(cls, fn):
#wraps(fn)
def wrapper(fn, fn_self, *args):
with fn_self.baz as fn_self.magic:
with fn_self.magic.fooz as more_magic:
blah = more_magic.much_more_magic() # repetative bleh
return fn(fn_self, *args)
return decorator.decorator(wrapper,fn)
Thanks go to Mike Muller for pointing out the right direction.

Here's what happens in time order as this method is defined.
this_is_decorator is created (not called).
decorator.decorator(this_is_decorator) is called. This returns a new function which becomes this_is_decorator and has the same usage.
classmethod(this_is_decorator) is called, and the result of that is a classmethod that accepts (cls, f) and returns wrapper.
Later at runtime, a call to this_is_decorator will return wrapper.
But considering that this_is_decorator is a class method, it's not clear to me that this is what you want. I'm guessing that you may want something more like this:
from decorator import decorator
#decorator
def mydecorator(f):
def wrapper(cls, *args, **kw):
# ... logging, reporting, caching, whatever
return f(*args, **kw)
return wrapper
class MyClass(object):
#classmethod
#mydecorator
def myclsmethod(a, b, c):
# no cls or self value accepted here; this is a function not a method
# ...
Here your decorator is defined outside your class, because it's changing an ordinary function into a classmethod (and because you may want to use it in other places). The order of execution here is:
mydecorator is defined, not called.
decorator(mydecorator) is called, and the result becomes mydecorator.
Creation of MyClass starts.
myclsmethod is created. It is an ordinary function, not a method. There is a difference within the VM, so that you do not have to explicitly supply cls or self arguments to methods.
myclsmethod is passed to mydecorator (which has itself been decorated before) and the result (wrapper) is still a function not a method.
The result of mydecorator is passed to classmethod which returns an actual class method that is bound to MyClass.myclsmethod.
Definition of MyClass finishes.
Later when MyClass.myclsmethod(a, b, c) is called, wrapper executes, which then calls the original myclsmethod(a, b, c) function (which it knows as f) without supplying the cls argument.
Since you have an additional need to preserve the argument list exactly, so that even the names of the arguments are preserved in the decorated function, except with an extra initial argument cls, then you could implement mydecorator this way:
from decorator import decorator
from inspect import getargspec
#decorator
def mydecorator(func):
result = [None] # necessary so exec can "return" objects
namespace = {'f': func, 'result': result}
source = []
add = lambda indent, line: source.append(' ' * indent + line) # shorthand
arglist = ', '.join(getargspec(func).args) # this does not cover keyword or default args
add(0, 'def wrapper(cls, %s):' % (arglist,))
add(2, 'return f(%s)' % (arglist,))
add(0, 'result[0] = wrapper') # this is how to "return" something from exec
exec '\n'.join(source) in namespace
return result[0] # this is wrapper
It's kinda ugly, but this is the only way I know to dynamically set the argument list of a function based on data. If returning a lambda is OK, you could use eval instead of exec, which eliminates the need for an array that gets written into but is otherwise about the same.