I try define class decorator. I have problem with __init__ method in decorated class. If __init__ method invokes super the RuntimeError maximum recursion depth exceeded is raised.
Code example:
def decorate(cls):
class NewClass(cls): pass
return NewClass
#decorate
class Foo(object):
def __init__(self, *args, **kwargs):
super(Foo, self).__init__(*args, **kwargs)
What am I doing wrong?
Thanks, Michał
Edit 1
Thanks to Mike Boers answer I realized that correct question is what should I do to achive that super(Foo, self) point to proper class.
I have also two limitation. I want invoke Foo.__init__ method and I can't change Foo class definition.
Edit 2
I have solved this problem. I modify decorator function body. I don't return new class. Instead of I wrap methods of orginal class.
You need to override NewClass.__init__ to prevent recursion, because NewClass.__init__ is Foo.__init__ and it keeps calling itself.
def decorate(cls):
class NewClass(cls):
def __init__(self):
pass
return NewClass
New idea:
How about not subclassing it? Maybe monkey patching is your friend?
def decorate(cls):
old_do_something = cls.do_something
def new_do_something(self):
print "decorated",
old_do_something(self)
cls.do_something = new_do_something
return cls
#decorate
class Foo(object):
def __init__(self, *args, **kwargs):
super(Foo, self).__init__(*args, **kwargs)
def do_something(self):
print "Foo"
f = Foo()
f.do_something()
Remember that a decorator is simply syntactic sugar for:
>>> Foo = decorate(Foo)
So in this case the name Foo actually refers to the NewClass class. Within the Foo.__init__ method you are in fact asking for the super __init__ of NewClass, which is Foo.__init__ (which is what is currently running).
Thus, your Foo.__init__ keeps receiving its own __init__ to call, and you end up in an infinite recursion.
Related
In my undertstanding, decorator class should contain __call__ or __new__ method. But cached_property in cpython repo doesn't follow the rules. Can anyone explain it for me?
class cached_property:
def __init__(self, func):
xxx
def __set_name__(self, owner, name):
xxx
def __get__(self, instance, owner=None):
xxx
__class_getitem__ = classmethod(GenericAlias)
Do all decorator classes need __call__?
decorator class should contain __call__ or __new__ method
Not all decorator classes need to implement __call__.
It's only required when we want to call the decorated object with ().
A decorator class that takes a callable to produce a callable has to implement __call__.
In this example, __call__ is implemented because we want to do data.calculate().
# Decorator to call and cache the function immediately
class PreCompute:
def __init__(self, func):
self.value = func()
def __call__(self, *args, **kwds):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# This actually calls PreCompute's __call__
print(data.calculate())
The definition of class Data here is roughly desugared to something like this,
so when calling data.calculate() we're actually calling the __call__ function from class PreCompute.
class Data:
def calculate():
print("Data.calculate called")
return 42
calculate = PreCompute(calculate)
A decorator class that takes a callable but does not produce a callable does not have to implement __call__.
For example, we can modify the class Precompute decorator to the following code, which allows us to access data.calculate as if it's an attribute.
For more information about what __get__ does, see Descriptor HowTo Guide from Python docs.
class PreCompute:
def __init__(self, func):
self.value = func()
def __get__(self, instance, owner):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# Access .calculate like an attribute
print(data.calculate)
What about __new__?
I'm not sure how OP got the impression that decorator classes must define either __call__ or __new__. I've seen __new__ being defined for use cases like #singleton decorator for classes, but as discussed in the previous section about __call__, this is also not strictly required. The only function we must define is an __init__ that receives the object to be decorated.
How does #functools.cached_property work, then?
Now going back to the question, notice from the documentation of #functools.cached_property that
it "transform a method of a class into a property", which is to be accessed without the parentheses ().
Therefore, class cached_property implements __get__ but not __call__, which is similar to the second example above.
I try do this:
import unittest
def decorator(cls):
class Decorator(cls):
def __init__(self, *args, **kwargs):
super(Decorator, self).__init__(*args, **kwargs)
return Decorator
#decorator
class myClass(unittest.TestCase):
def __init__(self, *args, **kwargs):
super(myClass, self).__init__(*args, **kwargs)
self.test = 'test'
def test_test(self):
pass
myClass().run()
But I get recursion in MyClass.__init__. Are there any ways to avoid this?
You cannot use super(myClass, self) within a decorated class in this way.
myClass is looked up as a global, and the global myClass is rebound to Decorator, so you are telling Python to look in the class MRO for __init__ starting from Decorator which is myClass, which calls super(myClass, self).__init__(), looking up myClass as a global, which is bound to Decorator, etc.
The easiest work-around is to not use super() here:
#decorator
class myClass(unittest.TestCase):
def __init__(self, *args, **kwargs):
unittest.TestCase.__init__(self, *args, **kwargs)
self.test = 'test'
This is one of the reasons why in Python 3 the argument-less version of super() was introduced, giving methods a __class__ cell instead.
You could jump through some (very tricky) hoops to re-compile the myClass.__init__() method to give it a myClass closure bound to the original undecorated class object instead, but for a unittest, I would not bother.
Part 1
I have a setup where I have a set of classes that I want to mock, my idea was that in the cases where I want to do this I pass a mock keyword argument into the constructor and in __new__ intercept this and instead pass back a mocked version of that object.
It looks like this (Edited the keyword lookup after #mgilsons suggestion):
class RealObject(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
return MockRealObject()
return super(RealObect, cls).__new__(cls, *args, **kwargs)
def __init__(self, whatever = None):
'''
Constructor
'''
#stuff happens
I then call the constructor like this:
ro = RealObject(mock = bool)
The issue I have here is that I get the following error when bool is False:
TypeError: __init__() got an unexpected keyword argument 'mock'
This works if I add mock as a keyword argument to __init__ but what I am asking if this is possible to avoid. I even pop the mock from the kwargs dict.
This is also a question about the design. Is there a better way to do this? (of course!) I wanted to try doing it this way, without using a factory or a superclass or anything. But still, should I use another keyword maybe? __call__?
Part 2 based on jsbueno's answer
So I wanted to extract the metaclass and the __new__ function into a separate module. I did this:
class Mockable(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
mock_cls = eval('{0}{1}'.format('Mock',cls.__name__))
return super(mock_cls, mock_cls).__new__(mock_cls)
return super(cls, cls).__new__(cls,*args, **kwargs)
class MockableMetaclass(type):
def __call__(self, *args, **kwargs):
obj = self.__new__(self, *args, **kwargs)
if "mock" in kwargs:
del kwargs["mock"]
obj.__init__(*args, **kwargs)
return obj
And I have defined in a separate module the classes RealObject and MockRealObject.
I have two problems now:
If MockableMetaclass and Mockable are not in the same module as the RealObject class the eval will raise a NameError if I provide mock = True.
If mock = False the code will enter into an endless recursion that ends in an impressive RuntimeError: maximum recursion depth exceeded while calling a Python objec. I'm guessing this is due to RealObject's superclass no longer being object but instead Mockable.
How can I fix these problems? is my approach incorrect? Should I instead have Mockable as a decorator? I tried that but that didn't seem to work since __new__ of an instance is only read-only it seems.
This is a job for the metaclass! :-)
The code responsible to call both __new__ and __init__ when instantiating a Python new-style object lies in the __call__method for the class metaclass. (or the semantically equivalent to that).
In other words - when you do:
RealObject() - what is really called is the RealObject.__class__.__call__ method.
Since without declaring a explicit metaclass, the metaclass is type, it is type.__call__ which is called.
Most recipes around dealing with metaclasses deal with subclassing the __new__ method - automating actions when the class is created. But overriding __call__ we can take actions when the class is instantiated, instead.
In this case, all that is needed is to remove the "mock" keyword parameter, if any, before calling __init__:
class MetaMock(type):
def __call__(cls, *args, **kw):
obj = cls.__new__(cls, *args, **kw)
if "mock" in kw:
del kw["mock"]
obj.__init__(*args, **kw)
return obj
class RealObject(metaclass=MetaMock):
...
A subclass is pretty much essential, since __new__ always passes the arguments to the constructor call to the __init__ method. If you add a subclass via a class decorator as a mixin then you can intercept the mock argument in the subclass __init__:
def mock_with(mock_cls):
class MockMixin(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock'):
return mock_cls()
return super(MockMixin, cls).__new__(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
kwargs.pop('mock')
super(MockMixin, self).__init__(*args, **kwargs)
def decorator(real_cls):
return type(real_cls.__name__, (MockMixin, real_cls), {})
return decorator
class MockRealObject(object):
pass
#mock_with(MockRealObject)
class RealObject(object):
def __init__(self, whatever=None):
pass
r = RealObject(mock=False)
assert isinstance(r, RealObject)
m = RealObject(mock=True)
assert isinstance(m, MockRealObject)
The alternative is for the subclass __new__ method to return RealObject(cls, *args, **kwargs); in that case, since the returned object isn't an instance of the subclass. However in that case the isinstance check will fail.
This is Python 2.5, and it's GAE too, not that it matters.
I have the following code. I'm decorating the foo() method in bar, using the dec_check class as a decorator.
class dec_check(object):
def __init__(self, f):
self.func = f
def __call__(self):
print 'In dec_check.__init__()'
self.func()
class bar(object):
#dec_check
def foo(self):
print 'In bar.foo()'
b = bar()
b.foo()
When executing this I was hoping to see:
In dec_check.__init__()
In bar.foo()
But I'm getting TypeError: foo() takes exactly 1 argument (0 given) as .foo(), being an object method, takes self as an argument. I'm guessing problem is that the instance of bar doesn't actually exist when I'm executing the decorator code.
So how do I pass an instance of bar to the decorator class?
You need to make the decorator into a descriptor -- either by ensuring its (meta)class has a __get__ method, or, way simpler, by using a decorator function instead of a decorator class (since functions are already descriptors). E.g.:
def dec_check(f):
def deco(self):
print 'In deco'
f(self)
return deco
class bar(object):
#dec_check
def foo(self):
print 'in bar.foo'
b = bar()
b.foo()
this prints
In deco
in bar.foo
as desired.
Alex's answer suffices when a function is sufficient. However, when you need a class you can make it work by adding the following method to the decorator class.
def __get__(self, obj, objtype):
"""Support instance methods."""
import functools
return functools.partial(self.__call__, obj)
To understand this you need to understand the descriptor protocol. The descriptor protocol is the mechanism for binding a thing to an instance. It consists of __get__, __set__ and __delete__, which are called when the thing is got, set or deleted from the instances dictionary.
In this case when the thing is got from the instance we are binding the first argument of its __call__ method to the instance, using partial. This is done automatically for member functions when the class is constructed, but for a synthetic member function like this we need to do it explicitly.
If you want to write the decorator as a class you can do:
from functools import update_wrapper, partial
class MyDecorator(object):
def __init__(self, func):
update_wrapper(self, func)
self.func = func
def __get__(self, obj, objtype):
"""Support instance methods."""
return partial(self.__call__, obj)
def __call__(self, obj, *args, **kwargs):
print('Logic here')
return self.func(obj, *args, **kwargs)
my_decorator = MyDecorator
class MyClass(object):
#my_decorator
def my_method(self):
pass
I am trying to write a decorator to do logging:
def logger(myFunc):
def new(*args, **keyargs):
print 'Entering %s.%s' % (myFunc.im_class.__name__, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
#logger
def f():
pass
C().f()
I would like this to print:
Entering C.f
but instead I get this error message:
AttributeError: 'function' object has no attribute 'im_class'
Presumably this is something to do with the scope of 'myFunc' inside 'logger', but I've no idea what.
Claudiu's answer is correct, but you can also cheat by getting the class name off of the self argument. This will give misleading log statements in cases of inheritance, but will tell you the class of the object whose method is being called. For example:
from functools import wraps # use this to preserve function signatures and docstrings
def logger(func):
#wraps(func)
def with_logging(*args, **kwargs):
print "Entering %s.%s" % (args[0].__class__.__name__, func.__name__)
return func(*args, **kwargs)
return with_logging
class C(object):
#logger
def f(self):
pass
C().f()
As I said, this won't work properly in cases where you've inherited a function from a parent class; in this case you might say
class B(C):
pass
b = B()
b.f()
and get the message Entering B.f where you actually want to get the message Entering C.f since that's the correct class. On the other hand, this might be acceptable, in which case I'd recommend this approach over Claudiu's suggestion.
Functions only become methods at runtime. That is, when you get C.f you get a bound function (and C.f.im_class is C). At the time your function is defined it is just a plain function, it is not bound to any class. This unbound and disassociated function is what is decorated by logger.
self.__class__.__name__ will give you the name of the class, but you can also use descriptors to accomplish this in a somewhat more general way. This pattern is described in a blog post on Decorators and Descriptors, and an implementation of your logger decorator in particular would look like:
class logger(object):
def __init__(self, func):
self.func = func
def __get__(self, obj, type=None):
return self.__class__(self.func.__get__(obj, type))
def __call__(self, *args, **kw):
print 'Entering %s' % self.func
return self.func(*args, **kw)
class C(object):
#logger
def f(self, x, y):
return x+y
C().f(1, 2)
# => Entering <bound method C.f of <__main__.C object at 0x...>>
Obviously the output can be improved (by using, for example, getattr(self.func, 'im_class', None)), but this general pattern will work for both methods and functions. However it will not work for old-style classes (but just don't use those ;)
Ideas proposed here are excellent, but have some disadvantages:
inspect.getouterframes and args[0].__class__.__name__ are not suitable for plain functions and static-methods.
__get__ must be in a class, that is rejected by #wraps.
#wraps itself should be hiding traces better.
So, I've combined some ideas from this page, links, docs and my own head,
and finally found a solution, that lacks all three disadvantages above.
As a result, method_decorator:
Knows the class the decorated method is bound to.
Hides decorator traces by answering to system attributes more correctly than functools.wraps() does.
Is covered with unit-tests for bound an unbound instance-methods, class-methods, static-methods, and plain functions.
Usage:
pip install method_decorator
from method_decorator import method_decorator
class my_decorator(method_decorator):
# ...
See full unit-tests for usage details.
And here is just the code of the method_decorator class:
class method_decorator(object):
def __init__(self, func, obj=None, cls=None, method_type='function'):
# These defaults are OK for plain functions
# and will be changed by __get__() for methods once a method is dot-referenced.
self.func, self.obj, self.cls, self.method_type = func, obj, cls, method_type
def __get__(self, obj=None, cls=None):
# It is executed when decorated func is referenced as a method: cls.func or obj.func.
if self.obj == obj and self.cls == cls:
return self # Use the same instance that is already processed by previous call to this __get__().
method_type = (
'staticmethod' if isinstance(self.func, staticmethod) else
'classmethod' if isinstance(self.func, classmethod) else
'instancemethod'
# No branch for plain function - correct method_type for it is already set in __init__() defaults.
)
return object.__getattribute__(self, '__class__')( # Use specialized method_decorator (or descendant) instance, don't change current instance attributes - it leads to conflicts.
self.func.__get__(obj, cls), obj, cls, method_type) # Use bound or unbound method with this underlying func.
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def __getattribute__(self, attr_name): # Hiding traces of decoration.
if attr_name in ('__init__', '__get__', '__call__', '__getattribute__', 'func', 'obj', 'cls', 'method_type'): # Our known names. '__class__' is not included because is used only with explicit object.__getattribute__().
return object.__getattribute__(self, attr_name) # Stopping recursion.
# All other attr_names, including auto-defined by system in self, are searched in decorated self.func, e.g.: __module__, __class__, __name__, __doc__, im_*, func_*, etc.
return getattr(self.func, attr_name) # Raises correct AttributeError if name is not found in decorated self.func.
def __repr__(self): # Special case: __repr__ ignores __getattribute__.
return self.func.__repr__()
It seems that while the class is being created, Python creates regular function objects. They only get turned into unbound method objects afterwards. Knowing that, this is the only way I could find to do what you want:
def logger(myFunc):
def new(*args, **keyargs):
print 'Entering %s.%s' % (myFunc.im_class.__name__, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
def f(self):
pass
C.f = logger(C.f)
C().f()
This outputs the desired result.
If you want to wrap all the methods in a class, then you probably want to create a wrapClass function, which you could then use like this:
C = wrapClass(C)
Class functions should always take self as their first argument, so you can use that instead of im_class.
def logger(myFunc):
def new(self, *args, **keyargs):
print 'Entering %s.%s' % (self.__class__.__name__, myFunc.__name__)
return myFunc(self, *args, **keyargs)
return new
class C(object):
#logger
def f(self):
pass
C().f()
at first I wanted to use self.__name__ but that doesn't work because the instance has no name. you must use self.__class__.__name__ to get the name of the class.
I found another solution to a very similar problem using the inspect library. When the decorator is called, even though the function is not yet bound to the class, you can inspect the stack and discover which class is calling the decorator. You can at least get the string name of the class, if that is all you need (probably can't reference it yet since it is being created). Then you do not need to call anything after the class has been created.
import inspect
def logger(myFunc):
classname = inspect.getouterframes(inspect.currentframe())[1][3]
def new(*args, **keyargs):
print 'Entering %s.%s' % (classname, myFunc.__name__)
return myFunc(*args, **keyargs)
return new
class C(object):
#logger
def f(self):
pass
C().f()
While this is not necessarily better than the others, it is the only way I can figure out to discover the class name of the future method during the call to the decorator. Make note of not keeping references to frames around in the inspect library documentation.
As shown in Asa Ayers' answer, you don't need to access the class object. It may be worth to know that since Python 3.3, you can also use __qualname__, which gives you the fully qualified name:
>>> def logger(myFunc):
... def new(*args, **keyargs):
... print('Entering %s' % myFunc.__qualname__)
... return myFunc(*args, **keyargs)
...
... return new
...
>>> class C(object):
... #logger
... def f(self):
... pass
...
>>> C().f()
Entering C.f
This has the added advantage of working also in the case of nested classes, as shown in this example taken from PEP 3155:
>>> class C:
... def f(): pass
... class D:
... def g(): pass
...
>>> C.__qualname__
'C'
>>> C.f.__qualname__
'C.f'
>>> C.D.__qualname__
'C.D'
>>> C.D.g.__qualname__
'C.D.g'
Notice also that in Python 3 the im_class attribute is gone, therefore if you really wish to access the class in a decorator, you need an other method. The approach I currently use involves object.__set_name__ and is detailed in my answer to "Can a Python decorator of an instance method access the class?"
You can also use new.instancemethod() to create an instance method (either bound or unbound) from a function.
Instead of injecting decorating code at definition time, when function doesn't know it's class, delay running this code until function is accessed/called. Descriptor object facilitates injecting own code late, at access/call time:
class decorated(object):
def __init__(self, func, type_=None):
self.func = func
self.type = type_
def __get__(self, obj, type_=None):
return self.__class__(self.func.__get__(obj, type_), type_)
def __call__(self, *args, **kwargs):
name = '%s.%s' % (self.type.__name__, self.func.__name__)
print('called %s with args=%s kwargs=%s' % (name, args, kwargs))
return self.func(*args, **kwargs)
class Foo(object):
#decorated
def foo(self, a, b):
pass
Now we can inspect class both at access time (__get__) and at call time (__call__). This mechanism works for plain methods as well as static|class methods:
>>> Foo().foo(1, b=2)
called Foo.foo with args=(1,) kwargs={'b': 2}
Full example at: https://github.com/aurzenligl/study/blob/master/python-robotwrap/Example4.py