Related
In my undertstanding, decorator class should contain __call__ or __new__ method. But cached_property in cpython repo doesn't follow the rules. Can anyone explain it for me?
class cached_property:
def __init__(self, func):
xxx
def __set_name__(self, owner, name):
xxx
def __get__(self, instance, owner=None):
xxx
__class_getitem__ = classmethod(GenericAlias)
Do all decorator classes need __call__?
decorator class should contain __call__ or __new__ method
Not all decorator classes need to implement __call__.
It's only required when we want to call the decorated object with ().
A decorator class that takes a callable to produce a callable has to implement __call__.
In this example, __call__ is implemented because we want to do data.calculate().
# Decorator to call and cache the function immediately
class PreCompute:
def __init__(self, func):
self.value = func()
def __call__(self, *args, **kwds):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# This actually calls PreCompute's __call__
print(data.calculate())
The definition of class Data here is roughly desugared to something like this,
so when calling data.calculate() we're actually calling the __call__ function from class PreCompute.
class Data:
def calculate():
print("Data.calculate called")
return 42
calculate = PreCompute(calculate)
A decorator class that takes a callable but does not produce a callable does not have to implement __call__.
For example, we can modify the class Precompute decorator to the following code, which allows us to access data.calculate as if it's an attribute.
For more information about what __get__ does, see Descriptor HowTo Guide from Python docs.
class PreCompute:
def __init__(self, func):
self.value = func()
def __get__(self, instance, owner):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# Access .calculate like an attribute
print(data.calculate)
What about __new__?
I'm not sure how OP got the impression that decorator classes must define either __call__ or __new__. I've seen __new__ being defined for use cases like #singleton decorator for classes, but as discussed in the previous section about __call__, this is also not strictly required. The only function we must define is an __init__ that receives the object to be decorated.
How does #functools.cached_property work, then?
Now going back to the question, notice from the documentation of #functools.cached_property that
it "transform a method of a class into a property", which is to be accessed without the parentheses ().
Therefore, class cached_property implements __get__ but not __call__, which is similar to the second example above.
I’m trying to create a decorator that is called within a class, which would pull attributes from that class, and use those class attributes to edit the function’s docstring.
My problem is that I have found examples of decorators that edit the docstring of the function (setting the function's __doc__ attribute equal to a new string), and I have also found examples of decorators that pull attributes from the parent class (by passing self into the decorator), but I haven’t been able to find an example of a decorator that is able to do both.
I have tried to combine these two examples, but it isn't working:
def my_decorator(func):
def wrapper(self, *args, **kwargs):
name = func.__name__ # pull function name
cls = self.__class__.__name__ # pull class name
func.__doc__ = "{} is new for the function {} in class {}".format(
str(func.__doc__), name, cls) # set them to docstring
return func(self, *args, **kwargs)
return wrapper
class Test():
#my_decorator
def example(self, examplearg=1):
"""Docstring"""
pass
With this, I would hope that the following would return "Docstring is now new for the function: example":
Test().example.__doc__
Instead it returns None.
Edit: Note that I am not interested in how to access the name of the class specifically, so much as how to access the class attributes in general (where here self.__class__.__name__ is used as an example).
example is replaced with wrapper; the decoration is equivalent to
def example(self, examplearg=1):
"""Docstring"""
pass
example = my_decorator(example)
so you need to set wrapper.__doc__, not func.__doc__.
def my_decorator(func):
def wrapper(self, *args, **kwargs):
return func(self, *args, **kwargs)
wrapper.__doc__ = "{} is new for the function {}".format(
str(func.__doc__),
func.__name__)
return wrapper
Note that at the time you call my_decorator, you don't have any information about what class the decorated function/method belongs to. You would have to pass its name explicitly:
def my_decorator(cls_name):
def _decorator(func):
def wrapper(self, *args, **kwargs):
return func(self, *args, **kwargs)
wrapper.__doc__ = "{} is new for function {} in class {}".format(
func.__doc__,
func.__name__,
cls_name)
return wrapper
return _decorator
class Test():
#my_decorator("Test")
def example(self, examplearg=1):
"""Docstring"""
# or
# def example(self, examplearg=1):
# """Docstring"""
#
# example = my_decorator("Test")(example)
You can simply modify the __doc__ attribute when the decorator is called instead, and use the first token of the dot-delimited __qualname__ attribute of the function to obtain the class name:
def my_decorator(func):
func.__doc__ = "{} is new for the function {} in class {}".format(
str(func.__doc__), func.__name__, func.__qualname__.split('.')[0])
return func
so that:
class Test():
#my_decorator
def example(self, examplearg=1):
"""Docstring"""
pass
print(Test().example.__doc__)
would output:
Docstring is new for the function example in class Test
Turns out that accessing class attributes from within a class is impossible, as the class has yet to be executed when the decorator is called. So the original goal - using a decorator within a class to access class attributes - does not seem to be possible.
However, thanks to jdehesa for pointing me to a workaround that allows access to the class attributes using a class decorator, here: Can a Python decorator of an instance method access the class?.
I was able to use the class decorator to alter the specific method's docstring using class attributes like so:
def class_decorator(cls):
for name, method in cls.__dict__.items():
if name == 'example':
# do something with the method
method.__doc__ = "{} is new for function {} in class {}".format(method.__doc__, name, cls.__name__)
# Note that other class attributes such as cls.__base__
# can also be accessed in this way
return cls
#class_decorator
class Test():
def example(self, examplearg=1):
"""Docstring"""
print(Test().example.__doc__)
# Returns "Docstring is new for function example in class Test"
Part 1
I have a setup where I have a set of classes that I want to mock, my idea was that in the cases where I want to do this I pass a mock keyword argument into the constructor and in __new__ intercept this and instead pass back a mocked version of that object.
It looks like this (Edited the keyword lookup after #mgilsons suggestion):
class RealObject(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
return MockRealObject()
return super(RealObect, cls).__new__(cls, *args, **kwargs)
def __init__(self, whatever = None):
'''
Constructor
'''
#stuff happens
I then call the constructor like this:
ro = RealObject(mock = bool)
The issue I have here is that I get the following error when bool is False:
TypeError: __init__() got an unexpected keyword argument 'mock'
This works if I add mock as a keyword argument to __init__ but what I am asking if this is possible to avoid. I even pop the mock from the kwargs dict.
This is also a question about the design. Is there a better way to do this? (of course!) I wanted to try doing it this way, without using a factory or a superclass or anything. But still, should I use another keyword maybe? __call__?
Part 2 based on jsbueno's answer
So I wanted to extract the metaclass and the __new__ function into a separate module. I did this:
class Mockable(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
mock_cls = eval('{0}{1}'.format('Mock',cls.__name__))
return super(mock_cls, mock_cls).__new__(mock_cls)
return super(cls, cls).__new__(cls,*args, **kwargs)
class MockableMetaclass(type):
def __call__(self, *args, **kwargs):
obj = self.__new__(self, *args, **kwargs)
if "mock" in kwargs:
del kwargs["mock"]
obj.__init__(*args, **kwargs)
return obj
And I have defined in a separate module the classes RealObject and MockRealObject.
I have two problems now:
If MockableMetaclass and Mockable are not in the same module as the RealObject class the eval will raise a NameError if I provide mock = True.
If mock = False the code will enter into an endless recursion that ends in an impressive RuntimeError: maximum recursion depth exceeded while calling a Python objec. I'm guessing this is due to RealObject's superclass no longer being object but instead Mockable.
How can I fix these problems? is my approach incorrect? Should I instead have Mockable as a decorator? I tried that but that didn't seem to work since __new__ of an instance is only read-only it seems.
This is a job for the metaclass! :-)
The code responsible to call both __new__ and __init__ when instantiating a Python new-style object lies in the __call__method for the class metaclass. (or the semantically equivalent to that).
In other words - when you do:
RealObject() - what is really called is the RealObject.__class__.__call__ method.
Since without declaring a explicit metaclass, the metaclass is type, it is type.__call__ which is called.
Most recipes around dealing with metaclasses deal with subclassing the __new__ method - automating actions when the class is created. But overriding __call__ we can take actions when the class is instantiated, instead.
In this case, all that is needed is to remove the "mock" keyword parameter, if any, before calling __init__:
class MetaMock(type):
def __call__(cls, *args, **kw):
obj = cls.__new__(cls, *args, **kw)
if "mock" in kw:
del kw["mock"]
obj.__init__(*args, **kw)
return obj
class RealObject(metaclass=MetaMock):
...
A subclass is pretty much essential, since __new__ always passes the arguments to the constructor call to the __init__ method. If you add a subclass via a class decorator as a mixin then you can intercept the mock argument in the subclass __init__:
def mock_with(mock_cls):
class MockMixin(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock'):
return mock_cls()
return super(MockMixin, cls).__new__(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
kwargs.pop('mock')
super(MockMixin, self).__init__(*args, **kwargs)
def decorator(real_cls):
return type(real_cls.__name__, (MockMixin, real_cls), {})
return decorator
class MockRealObject(object):
pass
#mock_with(MockRealObject)
class RealObject(object):
def __init__(self, whatever=None):
pass
r = RealObject(mock=False)
assert isinstance(r, RealObject)
m = RealObject(mock=True)
assert isinstance(m, MockRealObject)
The alternative is for the subclass __new__ method to return RealObject(cls, *args, **kwargs); in that case, since the returned object isn't an instance of the subclass. However in that case the isinstance check will fail.
I'd like to create a Python class decorator (*) that would be able to seamlessly wrap all method types the class might have: instance, class and static.
This is the code I have for now, with the parts that break it commented:
def wrapItUp(method):
def wrapped(*args, **kwargs):
print "This method call was wrapped!"
return method(*args, **kwargs)
return wrapped
dundersICareAbout = ["__init__", "__str__", "__repr__"]#, "__new__"]
def doICareAboutThisOne(cls, methodName):
return (callable(getattr(cls, methodName))
and (not (methodName.startswith("__") and methodName.endswith("__"))
or methodName in dundersICareAbout))
def classDeco(cls):
myCallables = ((aname, getattr(cls, aname)) for aname in dir(cls) if doICareAboutThisOne(cls, aname))
for name, call in myCallables:
print "*** Decorating: %s.%s(...)" % (cls.__name__, name)
setattr(cls, name, wrapItUp(call))
return cls
#classDeco
class SomeClass(object):
def instanceMethod(self, p):
print "instanceMethod: p =", p
#classmethod
def classMethod(cls, p):
print "classMethod: p =", p
#staticmethod
def staticMethod(p):
print "staticMethod: p =", p
instance = SomeClass()
instance.instanceMethod(1)
#SomeClass.classMethod(2)
#instance.classMethod(2)
#SomeClass.staticMethod(3)
#instance.staticMethod(3)
I'm having two issues trying to make this work:
When iterating over all callables, how do I find out if it is of an instance, class or static type?
How to I overwrite the method with a proper wrapped version of it that is invoked correctly for each of those cases?
Currently, this code generates different TypeErrors depending on what commented snippet is uncommented, like:
TypeError: unbound method wrapped() must be called with SomeClass instance as first argument (got int instance instead)
TypeError: classMethod() takes exactly 2 arguments (3 given)
(*): The same problem is much simpler if you're decorating the methods directly.
Because methods are wrappers for functions, to apply a decorator to a method on a class after the class has been constructed, you have to:
Extract the underlying function from the method using its im_func attribute.
Decorate the function.
Re-apply the wrapper.
Overwrite the attribute with the wrapped, decorated function.
It is difficult to distinguish a classmethod from a regular method once the #classmethod decorator has been applied; both kinds of methods are of type instancemethod. However, you can check the im_self attribute and see whether it is None. If so, it's a regular instance method; otherwise it's a classmethod.
Static methods are simple functions (the #staticmethod decorator merely prevents the usual method wrapper from being applied). So you don't have to do anything special for these, it looks like.
So basically your algorithm looks like this:
Get the attribute.
Is it callable? If not, proceed to the next attribute.
Is its type types.MethodType? If so, it is either a class method or an instance method.
If its im_self is None, it is an instance method. Extract the underlying function via the im_func attribute, decorate that, and re-apply the instance method: meth = types.MethodType(func, None, cls)
If its im_self is not None, it is a class method. Exctract the underlying function via im_func and decorate that. Now you have to reapply the classmethod decorator but you can't because classmethod() doesn't take a class, so there's no way to specify what class it will be attached to. Instead you have to use the instance method decorator: meth = types.MethodType(func, cls, type). Note that the type here is the actual built-in, type.
If its type is not types.MethodType then it is a static method or other non-bound callable, so just decorate it.
Set the new attribute back onto the class.
These change somewhat in Python 3 -- unbound methods are functions there, IIRC. In any case this will probably need to be completely rethought there.
There is an undocumented function, inspect.classify_class_attrs, which can tell you which attributes are classmethods or staticmethods. Under the hood, it uses isinstance(obj, staticmethod) and isinstance(obj, classmethod) to classify static and class methods. Following that pattern, this works in both Python2 and Python3:
def wrapItUp(method,kind='method'):
if kind=='static method':
#staticmethod
def wrapped(*args, **kwargs):
return _wrapped(*args,**kwargs)
elif kind=='class method':
#classmethod
def wrapped(cls,*args, **kwargs):
return _wrapped(*args,**kwargs)
else:
def wrapped(self,*args, **kwargs):
return _wrapped(self,*args,**kwargs)
def _wrapped(*args, **kwargs):
print("This method call was wrapped!")
return method(*args, **kwargs)
return wrapped
def classDeco(cls):
for name in (name
for name in dir(cls)
if (callable(getattr(cls,name))
and (not (name.startswith('__') and name.endswith('__'))
or name in '__init__ __str__ __repr__'.split()))
):
method = getattr(cls, name)
obj = cls.__dict__[name] if name in cls.__dict__ else method
if isinstance(obj, staticmethod):
kind = "static method"
elif isinstance(obj, classmethod):
kind = "class method"
else:
kind = "method"
print("*** Decorating: {t} {c}.{n}".format(
t=kind,c=cls.__name__,n=name))
setattr(cls, name, wrapItUp(method,kind))
return cls
#classDeco
class SomeClass(object):
def instanceMethod(self, p):
print("instanceMethod: p = {}".format(p))
#classmethod
def classMethod(cls, p):
print("classMethod: p = {}".format(p))
#staticmethod
def staticMethod(p):
print("staticMethod: p = {}".format(p))
instance = SomeClass()
instance.instanceMethod(1)
SomeClass.classMethod(2)
instance.classMethod(2)
SomeClass.staticMethod(3)
instance.staticMethod(3)
I am trying to write a decorator to do logging:
def logger(myFunc):
def new(*args, **keyargs):
print 'Entering %s.%s' % (myFunc.im_class.__name__, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
#logger
def f():
pass
C().f()
I would like this to print:
Entering C.f
but instead I get this error message:
AttributeError: 'function' object has no attribute 'im_class'
Presumably this is something to do with the scope of 'myFunc' inside 'logger', but I've no idea what.
Claudiu's answer is correct, but you can also cheat by getting the class name off of the self argument. This will give misleading log statements in cases of inheritance, but will tell you the class of the object whose method is being called. For example:
from functools import wraps # use this to preserve function signatures and docstrings
def logger(func):
#wraps(func)
def with_logging(*args, **kwargs):
print "Entering %s.%s" % (args[0].__class__.__name__, func.__name__)
return func(*args, **kwargs)
return with_logging
class C(object):
#logger
def f(self):
pass
C().f()
As I said, this won't work properly in cases where you've inherited a function from a parent class; in this case you might say
class B(C):
pass
b = B()
b.f()
and get the message Entering B.f where you actually want to get the message Entering C.f since that's the correct class. On the other hand, this might be acceptable, in which case I'd recommend this approach over Claudiu's suggestion.
Functions only become methods at runtime. That is, when you get C.f you get a bound function (and C.f.im_class is C). At the time your function is defined it is just a plain function, it is not bound to any class. This unbound and disassociated function is what is decorated by logger.
self.__class__.__name__ will give you the name of the class, but you can also use descriptors to accomplish this in a somewhat more general way. This pattern is described in a blog post on Decorators and Descriptors, and an implementation of your logger decorator in particular would look like:
class logger(object):
def __init__(self, func):
self.func = func
def __get__(self, obj, type=None):
return self.__class__(self.func.__get__(obj, type))
def __call__(self, *args, **kw):
print 'Entering %s' % self.func
return self.func(*args, **kw)
class C(object):
#logger
def f(self, x, y):
return x+y
C().f(1, 2)
# => Entering <bound method C.f of <__main__.C object at 0x...>>
Obviously the output can be improved (by using, for example, getattr(self.func, 'im_class', None)), but this general pattern will work for both methods and functions. However it will not work for old-style classes (but just don't use those ;)
Ideas proposed here are excellent, but have some disadvantages:
inspect.getouterframes and args[0].__class__.__name__ are not suitable for plain functions and static-methods.
__get__ must be in a class, that is rejected by #wraps.
#wraps itself should be hiding traces better.
So, I've combined some ideas from this page, links, docs and my own head,
and finally found a solution, that lacks all three disadvantages above.
As a result, method_decorator:
Knows the class the decorated method is bound to.
Hides decorator traces by answering to system attributes more correctly than functools.wraps() does.
Is covered with unit-tests for bound an unbound instance-methods, class-methods, static-methods, and plain functions.
Usage:
pip install method_decorator
from method_decorator import method_decorator
class my_decorator(method_decorator):
# ...
See full unit-tests for usage details.
And here is just the code of the method_decorator class:
class method_decorator(object):
def __init__(self, func, obj=None, cls=None, method_type='function'):
# These defaults are OK for plain functions
# and will be changed by __get__() for methods once a method is dot-referenced.
self.func, self.obj, self.cls, self.method_type = func, obj, cls, method_type
def __get__(self, obj=None, cls=None):
# It is executed when decorated func is referenced as a method: cls.func or obj.func.
if self.obj == obj and self.cls == cls:
return self # Use the same instance that is already processed by previous call to this __get__().
method_type = (
'staticmethod' if isinstance(self.func, staticmethod) else
'classmethod' if isinstance(self.func, classmethod) else
'instancemethod'
# No branch for plain function - correct method_type for it is already set in __init__() defaults.
)
return object.__getattribute__(self, '__class__')( # Use specialized method_decorator (or descendant) instance, don't change current instance attributes - it leads to conflicts.
self.func.__get__(obj, cls), obj, cls, method_type) # Use bound or unbound method with this underlying func.
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def __getattribute__(self, attr_name): # Hiding traces of decoration.
if attr_name in ('__init__', '__get__', '__call__', '__getattribute__', 'func', 'obj', 'cls', 'method_type'): # Our known names. '__class__' is not included because is used only with explicit object.__getattribute__().
return object.__getattribute__(self, attr_name) # Stopping recursion.
# All other attr_names, including auto-defined by system in self, are searched in decorated self.func, e.g.: __module__, __class__, __name__, __doc__, im_*, func_*, etc.
return getattr(self.func, attr_name) # Raises correct AttributeError if name is not found in decorated self.func.
def __repr__(self): # Special case: __repr__ ignores __getattribute__.
return self.func.__repr__()
It seems that while the class is being created, Python creates regular function objects. They only get turned into unbound method objects afterwards. Knowing that, this is the only way I could find to do what you want:
def logger(myFunc):
def new(*args, **keyargs):
print 'Entering %s.%s' % (myFunc.im_class.__name__, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
def f(self):
pass
C.f = logger(C.f)
C().f()
This outputs the desired result.
If you want to wrap all the methods in a class, then you probably want to create a wrapClass function, which you could then use like this:
C = wrapClass(C)
Class functions should always take self as their first argument, so you can use that instead of im_class.
def logger(myFunc):
def new(self, *args, **keyargs):
print 'Entering %s.%s' % (self.__class__.__name__, myFunc.__name__)
return myFunc(self, *args, **keyargs)
return new
class C(object):
#logger
def f(self):
pass
C().f()
at first I wanted to use self.__name__ but that doesn't work because the instance has no name. you must use self.__class__.__name__ to get the name of the class.
I found another solution to a very similar problem using the inspect library. When the decorator is called, even though the function is not yet bound to the class, you can inspect the stack and discover which class is calling the decorator. You can at least get the string name of the class, if that is all you need (probably can't reference it yet since it is being created). Then you do not need to call anything after the class has been created.
import inspect
def logger(myFunc):
classname = inspect.getouterframes(inspect.currentframe())[1][3]
def new(*args, **keyargs):
print 'Entering %s.%s' % (classname, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
#logger
def f(self):
pass
C().f()
While this is not necessarily better than the others, it is the only way I can figure out to discover the class name of the future method during the call to the decorator. Make note of not keeping references to frames around in the inspect library documentation.
As shown in Asa Ayers' answer, you don't need to access the class object. It may be worth to know that since Python 3.3, you can also use __qualname__, which gives you the fully qualified name:
>>> def logger(myFunc):
... def new(*args, **keyargs):
... print('Entering %s' % myFunc.__qualname__)
... return myFunc(*args, **keyargs)
...
... return new
...
>>> class C(object):
... #logger
... def f(self):
... pass
...
>>> C().f()
Entering C.f
This has the added advantage of working also in the case of nested classes, as shown in this example taken from PEP 3155:
>>> class C:
... def f(): pass
... class D:
... def g(): pass
...
>>> C.__qualname__
'C'
>>> C.f.__qualname__
'C.f'
>>> C.D.__qualname__
'C.D'
>>> C.D.g.__qualname__
'C.D.g'
Notice also that in Python 3 the im_class attribute is gone, therefore if you really wish to access the class in a decorator, you need an other method. The approach I currently use involves object.__set_name__ and is detailed in my answer to "Can a Python decorator of an instance method access the class?"
You can also use new.instancemethod() to create an instance method (either bound or unbound) from a function.
Instead of injecting decorating code at definition time, when function doesn't know it's class, delay running this code until function is accessed/called. Descriptor object facilitates injecting own code late, at access/call time:
class decorated(object):
def __init__(self, func, type_=None):
self.func = func
self.type = type_
def __get__(self, obj, type_=None):
return self.__class__(self.func.__get__(obj, type_), type_)
def __call__(self, *args, **kwargs):
name = '%s.%s' % (self.type.__name__, self.func.__name__)
print('called %s with args=%s kwargs=%s' % (name, args, kwargs))
return self.func(*args, **kwargs)
class Foo(object):
#decorated
def foo(self, a, b):
pass
Now we can inspect class both at access time (__get__) and at call time (__call__). This mechanism works for plain methods as well as static|class methods:
>>> Foo().foo(1, b=2)
called Foo.foo with args=(1,) kwargs={'b': 2}
Full example at: https://github.com/aurzenligl/study/blob/master/python-robotwrap/Example4.py