I'm having a hard time understanding what happens when I try to nest descriptors/decorators. I'm using python 2.7.
For example, let's take the following simplified versions of property and classmethod:
class MyProperty(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, obj, objtype=None):
print 'IN MyProperty.__get__'
return self.fget(obj)
class MyClassMethod(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, objtype=None):
print 'IN MyClassMethod.__get__'
def f(*args, **kwargs):
return self.f(objtype, *args, **kwargs)
return f
Trying to nest them:
class A(object):
# doesn't work:
#MyProperty
#MyClassMethod
def klsproperty(cls):
return 555
# works:
#MyProperty
def prop(self):
return 111
# works:
#MyClassMethod
def klsmethod(cls, x):
return x**2
% print A.klsproperty
IN MyProperty.__get__
...
TypeError: 'MyClassMethod' object is not callable
The __get__ method of the inner descriptor MyClassMethod is not getting called.
Failing to figure out why, I tried throwing in (what I think is) a no-op descriptor:
class NoopDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, objtype=None):
print 'IN NoopDescriptor.__get__'
return self.f.__get__(obj, objtype=objtype)
Trying to use the no-op descriptor/decorator in nesting:
class B(object):
# works:
#NoopDescriptor
#MyProperty
def prop1(self):
return 888
# doesn't work:
#MyProperty
#NoopDescriptor
def prop2(self):
return 999
% print B().prop1
IN NoopDescriptor.__get__
IN MyProperty.__get__
888
% print B().prop2
IN MyProperty.__get__
...
TypeError: 'NoopDescriptor' object is not callable
I don't understand why B().prop1 works and B().prop2 does not.
Questions:
What am I doing wrong? Why am I getting a object is not callable error?
What's the right way? e.g. what is the best way to define MyClassProperty while re-using MyClassMethod and MyProperty (or classmethod and property)
In this case, when the decorators are used without parameters, a decorator is called with the function it decorates as its parameter. The decorator's return value is used instead of the decorated function. So:
#MyProperty
def prop(self):
...
is equivalent to:
def prop(self):
...
prop = MyProperty(prop)
Since MyProperty implements the descriptor protocol, accessing A.prop will actually call A.prop.__get__(), and you've defined __get__ to call the object which was decorated (in this case, the original function/method), so everything works fine.
Now, in the nested case:
#MyProperty
#MyClassMethod
def prop(self):
...
The equivalent is:
def prop(self):
...
prop = MyClassMethod(prop) # prop is now instance of MyClassMethod
prop = MyProperty(prop) # prop is now instance of MyProperty
# (with fget == MyClassMethod instance)
Now, as before, accessing A.prop will actually call A.prop.__get__() (in MyProperty) which then tries to call the instance of MyClassMethod (the object which was decorated and stored in the fget attribute).
But the MyClassMethod does not have a __call__ method defined, so you get the error MyClassMethod is not callable.
And to address your second question: A property is already a class attribute - in your example, accessing A.prop will return the value of the property in the class object and A().prop will return the value of the property in an instance object (which can be the same as the class object if the instance did not override it).
You can make your code work if you make MyProperty apply the descriptor protocol to its wrapped object:
class MyProperty(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, obj, objtype=None):
print('IN MyProperty.__get__')
try:
return self.fget.__get__(obj, objtype)()
except AttributeError: # self.fget has no __get__ method
return self.fget(obj)
Now your example code works:
class A(object):
#MyProperty
#MyClassMethod
def klsproperty(cls):
return 555
print(A.klsproperty)
The output is:
IN MyProperty.__get__
IN MyClassMethod.__get__
555
I found the definitive answer to my own old question in Graham Dumpleton's fascinating blog.
In short, the decorators I wrote do not honour the descriptors protocol, by trying to call the wrapped function/object directly, instead of first giving them a chance to perform their "descriptor magic" (by calling their __get__() first).
Related
This question already has answers here:
How can I decorate an instance method with a decorator class?
(2 answers)
Closed 2 years ago.
I thought the following would work as a decorator
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
class A:
#D
def f(self, x):
pass
a=A()
a.f(1)
but I get TypeError: f() missing 1 required positional argument: 'x'
What's going on and is there a way a can use a class as a decorator like this?
The thing is that besides the decorator mechanism, there is the mechanism that Python uses so that functions inside class bodies behave as instance methods: it is the "descriptor protocol". That is actually simple: all function objects have a __get__ method (but not __set__ or __del__) method, which make of them "non data descriptors". When Python retrieves the attribute from an instance, __get__ is called with the instance as a parameter - the __get__ method them have to return a callable that will work as the method, and has to know which was the instance called:
# example only - DO NOT DO THIS but for learning purposes,
# due to concurrency problems:
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(self.instance, *args, **kwargs)
def __get__(self, instance, owner):
self.instance = instance
return self
class A:
#D
def f(self, x):
print(self, x)
a=A()
a.f(1)
This will print "<main.A object at 0x...> 1"
However, as it is easily perceivable this only allows the decorated method to be called in a single instance at once - even non parallel code that owns more than an instance of "A" could have the method called with the wrong instance in mind. That is, this sequence:
In [127]: a1 = A()
In [128]: a2 = A()
In [129]: f1 = a1.f
In [130]: f2 = a2.f
In [131]: f1()
will end up calling "a2.f()" not "a1.f()"
To avoid this, what you have to return is a callable from __get__ that won't need to retrieve the instance as a class attribute. One way to do that is to create a partial callable and include that - however, note that since this is a necessary step, there is no need for the decorator class itself to have the "run wrapper + original code" function in the __call__ method - it could have any name:
from functools import partial
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, _instance=None, **kwargs):
if _instance:
return self.func(_instance, *args, **kwargs)
else:
return self.func(*args, **kwargs)
def __get__(self, instance, owner):
return partial(self.__call__, _instance=instance)
class A:
#D
def f(self, x):
print(self, x)
a=A()
a.f(1)
I've read somewhere that the use of ‘self’ in Python converts myobject.method (arg1, arg2)
into MyClass.method(myobject, arg1, arg2).
Does anyone know how I can prove this?
Is it only possible if I look at the bytecode by using dis.dis?
self doesn't do anything. self is merely the conventional name given to the first argument of a method in Python class definitions. This argument will be passed an instance of the class.
Essentially, to understand what is actually going on, you have to understand Python descriptors. The best places are the official docs
To boil it down, descriptor objects are objects that implement __get__, __set__ or __delete__. These methods intercept object attribute access, obj.x, object attribute assignment: obj.x = 42, and object attribute deletion, del obj.x.
Also, check out the HOWTO, where they show how Python functions and methods are simply descriptors, and show an example Python implementation (of course, in CPython, this is implemented in C):
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
We can "cheat" and create our own object that merely wraps a function object, and see that this works.
import types
class Function:
def __init__(self, func):
self._func = func
def __call__(self, *args, **kwargs):
return self._func(*args, **kwargs)
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c https://docs.python.org/3/howto/descriptor.html#functions-and-methods"
if obj is None:
return self
else:
return types.MethodType(self, obj)
class Foo:
def __init__(self):
self.foo = 42
bar = Function(lambda self:
self.foo ** 2
And or, in a REPL:
>>> import types
>>>
>>> class Function:
... def __init__(self, func):
... self._func = func
... def __call__(self, *args, **kwargs):
... return self._func(*args, **kwargs)
... def __get__(self, obj, objtype=None):
... "Simulate func_descr_get() in Objects/funcobject.c https://docs.python.org/3/howto/descriptor.html#functions-and-methods"
... if obj is None:
... return self
... else:
... return types.MethodType(self, obj)
...
>>> class Foo:
... def __init__(self):
... self.foo = 42
... bar = Function(lambda self:
... self.foo ** 2
... )
...
>>> Foo().bar()
1764
This shows you that the magic behind "self" is merely that function objects are descriptors, they implement a __get__ method which either returns the function itself if called without an instance, or returns a method-object that binds the first argument.
Trying it out for yourself in IDLE could help you out with this:
class MyObject(object):
def method(self, arg1, arg2):
print(self)
#staticmethod
def static_method(arg1, arg2):
print(arg1)
my_object = MyObject()
my_object.method(1, 2)
>>> <MyObject at 0x1234>
my_object.static_method(1, 2)
>>> 1
Python doesn't convert anything, it just silently passes the class instance as the first argument to a class method. Above you can see if you made the method static (via #staticmethod decorator), you avoid that extra parameter.
I'm trying to add extra decorator for magic method (__get__) in descriptor class.
I'm able to do it when I use #property but not when I use descriptor class.
I check range because my object set registers on the bus and some registers can take only specific range of values:
import functools
def check_range(min, max):
def decorator(f):
#functools.wraps(f)
def wrap(self, value):
if value not in range(min, max+1):
return
return f(self, value)
return wrap
return decorator
This works:
class Foo:
def __init__(self):
self.device.init_smth('my_object')
#property
def my_object(self):
return self.device.get_value('my_object')
#my_object.setter
#check_range(0,1)
def my_object(self, value):
self.device.set_value('my_object', value)
a = Foo()
print(a.my_object)
a.my_object = 1
print(a.my_object)
a.myobject = -1
And in this example everything works the same but check_range is not invoked:
class Register:
def __init__(self, name, device):
self.name = name
device.init_smth(name)
def __get__(self, instance, owner):
return instance.device.get_value(self.name)
#check_range(0,1)
def __set__(self, instance, value):
instance.device.set_value(self.name, value)
class Foo:
def __init__(self):
self.my_object = Register('my_object', self.device)
a = Foo()
print(a.my_object)
a.my_object = 1
print(a.my_object)
a.myobject = -1
I may be wrong, but most probably your descriptor not invoked at all, decorator is not the problem. Descriptors meant to be used like
class Foo2:
my_object = Register('my_object', 'init_value')
— you're defining it like class attribute. And python will execute all machinery with __get__/__set__/__del__ if your class attribute supports it (i.e. it is descriptor).
This is why there is an "instance" argument in descriptor methods — you're defining descriptor as class variable, but i.e. __set__ method will receive actual instance of your class, so you can manage per-instance data, like your device
I have an issue with using a class to decorate another class' method. Code is as follows:
class decorator(object):
def __init__(self, func):
self.func = func
def __call__(self, *args):
return self.func(*args)
class test(object):
#decorator
def func(self, x, y):
print x, y
t = test()
t.func(1, 2)
It shows this error
TypeError: func() takes exactly 3 arguments (2 given).
If called using:
t.func(t, 1, 2)
then it passes. But then if the decorator is taken away, then this line will have issue again.
Why this is happening and how to solve it?
Edit: second version of the code to show the self in decorator.__call__ should be different than the self in test.func:
class decorator(object):
def __init__(self, func):
self.func = func
def __call__(self, *args):
return self.func(*args)
class test(object):
def __init__(self):
self.x = 1
self.y = 2
#decorator
def func(self):
print self
print self.x, self.y
t = test()
t.func()
This shows the same error. But
t.func(t)
works but not ideal.
To work as a method, an object in a class needs to implement part of the descriptor protocol. That is, it should have a __get__ method that returns a callable object which has been "bound" to the instance the method was looked up on.
Here's one way that you could make that work, using a wrapper function:
class decorator(object):
def __init__(self, func):
self.func = func
def __get__(self, instance, owner):
def wrapper(*args):
return self.func(instance, *args) # note, self here is the descriptor object
return wrapper
You could instead return an instance of some other class from __get__, rather than a function, and use the __call__ method of that other class to implement the wrapper. If you're not using a closure though, you'd need to pass the instance to the wrapper class explicitly (as well as the function, since self.func won't work outside the descriptor class).
There is an answered question about classmethod and property combined together: Using property() on classmethods
I still don't understand the cause of the problem, please help.
My understanding of classmethod was that it simply replaces self with cls. With this in mind I wrote several classmethods during the past few years and now I see I was wrong all that time.
So what is the difference between #classmethod and #cm from the code below?
def cm(func):
def decorated(self, *args, **kwargs):
return func(self.__class__, *args, **kwargs)
return decorated
class C:
V = 0
#property
#classmethod
def inc1(cls):
cls.V += 1
print("V1 =", cls.V)
#property
#cm
def inc3(cls):
cls.V += 3
print("V3 =", cls.V)
c = C()
#c.inc1 # fails with: TypeError: 'classmethod' object is not callable
c.inc3 # works
inc3 with cm works, but inc1 with classmethod does not.
what is the difference between #classmethod and #cm from the code below?
decorator is calling during class creation time before an instance is created.
In your case, since #cm returns func(self.__class__, *args, **kwargs), which is relied on self, it should be used as a instance method.
On the other hand, #classmethod is able to use before an instance is created.
def cm(func):
def decorated(self, *args, **kwargs):
return func(self.__class__, *args, **kwargs)
return decorated
class C:
#classmethod
def inc1(cls):
(blablabla)
#cm
def inc3(cls):
(blablabla)
C().inc1() # works as a instance method
C.inc1() # works as a classmethod
C().inc3() # works as a instance method
C.inc3() # TypeError: unbound method decorated() must be called with C instance as first argument (got nothing instead)
For a combination of classmethod and property, it could be done by return an customized object. Reference
class ClassPropertyDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
return self.f.__get__(obj, klass)()
def classproperty(func):
if not isinstance(func, (classmethod, staticmethod)):
func = classmethod(func)
return ClassPropertyDescriptor(func)
class C:
#classproperty
def inc1(cls):
(blablabla)
C.inc1 # works as a classmethod property
[Edit]
Q. What does the classmethod() call do with the method it decorates to achieve that?
The implementation can be done by using descriptor
class ClassMethodDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
def myclassmethod(func):
return ClassMethodDescriptor(func)
class C:
#myclassmethod
def inc1(cls):
(blablabla)
C.inc1() # works as a classmethod
Q. Why is the result not callable?
Because the implementation of ClassMethodDescriptor does not define __call__ function. Once using #property, it will return ClassMethodDescriptor which is not callable.
The difference is that classmethod is not callable, and cm method is callable. This means that when the property(class) makes a call to the inputed func(which it is supposed to do), it works as you'll except for cm, but will not work for classmethod since classmethod does not have a call implemented.
class method does not know anything about instance and does not require it.
instance method knows about it's instance and it's class.
class Foo:
some = 'some'
class Bar(Foo):
def __init__(self):
self.some = 'not some'
#classmethod
def cls_some(cls):
print(cls.some)
def instance_some(self):
print(self.some)
Bar.cls_some()
>>>some
Bar().instance_some()
>>>not some
Also as you can see you don't need an instance to call classmethod.