I've read somewhere that the use of ‘self’ in Python converts myobject.method (arg1, arg2)
into MyClass.method(myobject, arg1, arg2).
Does anyone know how I can prove this?
Is it only possible if I look at the bytecode by using dis.dis?
self doesn't do anything. self is merely the conventional name given to the first argument of a method in Python class definitions. This argument will be passed an instance of the class.
Essentially, to understand what is actually going on, you have to understand Python descriptors. The best places are the official docs
To boil it down, descriptor objects are objects that implement __get__, __set__ or __delete__. These methods intercept object attribute access, obj.x, object attribute assignment: obj.x = 42, and object attribute deletion, del obj.x.
Also, check out the HOWTO, where they show how Python functions and methods are simply descriptors, and show an example Python implementation (of course, in CPython, this is implemented in C):
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
We can "cheat" and create our own object that merely wraps a function object, and see that this works.
import types
class Function:
def __init__(self, func):
self._func = func
def __call__(self, *args, **kwargs):
return self._func(*args, **kwargs)
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c https://docs.python.org/3/howto/descriptor.html#functions-and-methods"
if obj is None:
return self
else:
return types.MethodType(self, obj)
class Foo:
def __init__(self):
self.foo = 42
bar = Function(lambda self:
self.foo ** 2
And or, in a REPL:
>>> import types
>>>
>>> class Function:
... def __init__(self, func):
... self._func = func
... def __call__(self, *args, **kwargs):
... return self._func(*args, **kwargs)
... def __get__(self, obj, objtype=None):
... "Simulate func_descr_get() in Objects/funcobject.c https://docs.python.org/3/howto/descriptor.html#functions-and-methods"
... if obj is None:
... return self
... else:
... return types.MethodType(self, obj)
...
>>> class Foo:
... def __init__(self):
... self.foo = 42
... bar = Function(lambda self:
... self.foo ** 2
... )
...
>>> Foo().bar()
1764
This shows you that the magic behind "self" is merely that function objects are descriptors, they implement a __get__ method which either returns the function itself if called without an instance, or returns a method-object that binds the first argument.
Trying it out for yourself in IDLE could help you out with this:
class MyObject(object):
def method(self, arg1, arg2):
print(self)
#staticmethod
def static_method(arg1, arg2):
print(arg1)
my_object = MyObject()
my_object.method(1, 2)
>>> <MyObject at 0x1234>
my_object.static_method(1, 2)
>>> 1
Python doesn't convert anything, it just silently passes the class instance as the first argument to a class method. Above you can see if you made the method static (via #staticmethod decorator), you avoid that extra parameter.
Related
i am trying to do my own property but i can't understand how to implement method set.
this command x.value = 20 doesnt work. how can i do setter for decorator #myownproperties?
class myownproperties:
def __init__(self, dget, dset=None):
self.dget = dget
self.dset = dset
def __get__(self, obj, klass = None):
return self.dget.__get__(obj, klass)()
def __set__(self, obj, value):
type_ = type(obj)
return self.dset.__get__(obj, type_)(value)
def setter(self, func):
self.dset = classmethod(func)
return self
class Foo:
def __init__(self, foo):
self.foo = foo
#myownproperties
def value(self):
return self.foo
#value.setter
def value_setter(self, new_val):
self.foo = new_val
x = Foo(10)
print(x.value)
x.value = 20
print(x.value)
Here's a simple version:
class myownproperties:
def __init__(self, dget, dset=None):
self.dget = dget
self.dset = dset
def __get__(self, obj, _cls):
return self.dget(obj)
def __set__(self, obj, value):
self.dset(obj, value)
def setter(self, func):
self.dset = func
return func
class Foo:
def __init__(self, foo):
self.foo = foo
#myownproperties
def value(self):
return self.foo
#value.setter
def value_setter(self, new_val):
self.foo = new_val
x = Foo(10)
print(x.value) # 10
x.value = 20
print(x.value) # 20
I couldn't figure out the purpose behind wrapping dset in classmethod or in calling __get__ on the resulting object, so I just removed all that and it seems to work fine.
At the time the decorator is applied, and within that scope, def value_setter(self, new_val): is just an ordinary function. (We aren't looking it up on an instance of the class, so there is no opportunity for method binding to occur.) So in the decorator's setter method, we should just have self.dset = func, just like how we have just self.dget = dget in the __init__.
When the __set__ of the descriptor is called, we should directly invoke this function, because it's the function that contains the logic for what should happen upon the attribute access. That is, just self.dset(obj, value) - here, self is the descriptor; its dset attribute (a function which is stored in the object, not a bound method) is looked up and called with obj as the Foo instance and value as the value to set. The function takes in self, obj where self is bound to the Foo instance - not by method-call binding, but by normal argument passing. The setter does self.foo = new_val, thus assigning the foo attribute of the Foo instance.
Similarly for __get__ - just use self.dget directly, and return the result.
Putting these pieces together gives Samwise's version of the code.
(Incidentally, in case you weren't aware: method binding, itself, uses this mechanism! Functions are descriptors; their class implements __get__ to do the binding and create an object that represents the bound method, which can then be called.)
This question already has answers here:
How can I decorate an instance method with a decorator class?
(2 answers)
Closed 2 years ago.
I thought the following would work as a decorator
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
class A:
#D
def f(self, x):
pass
a=A()
a.f(1)
but I get TypeError: f() missing 1 required positional argument: 'x'
What's going on and is there a way a can use a class as a decorator like this?
The thing is that besides the decorator mechanism, there is the mechanism that Python uses so that functions inside class bodies behave as instance methods: it is the "descriptor protocol". That is actually simple: all function objects have a __get__ method (but not __set__ or __del__) method, which make of them "non data descriptors". When Python retrieves the attribute from an instance, __get__ is called with the instance as a parameter - the __get__ method them have to return a callable that will work as the method, and has to know which was the instance called:
# example only - DO NOT DO THIS but for learning purposes,
# due to concurrency problems:
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(self.instance, *args, **kwargs)
def __get__(self, instance, owner):
self.instance = instance
return self
class A:
#D
def f(self, x):
print(self, x)
a=A()
a.f(1)
This will print "<main.A object at 0x...> 1"
However, as it is easily perceivable this only allows the decorated method to be called in a single instance at once - even non parallel code that owns more than an instance of "A" could have the method called with the wrong instance in mind. That is, this sequence:
In [127]: a1 = A()
In [128]: a2 = A()
In [129]: f1 = a1.f
In [130]: f2 = a2.f
In [131]: f1()
will end up calling "a2.f()" not "a1.f()"
To avoid this, what you have to return is a callable from __get__ that won't need to retrieve the instance as a class attribute. One way to do that is to create a partial callable and include that - however, note that since this is a necessary step, there is no need for the decorator class itself to have the "run wrapper + original code" function in the __call__ method - it could have any name:
from functools import partial
class D:
def __init__(self, func):
self.func = func
def __call__(self, *args, _instance=None, **kwargs):
if _instance:
return self.func(_instance, *args, **kwargs)
else:
return self.func(*args, **kwargs)
def __get__(self, instance, owner):
return partial(self.__call__, _instance=instance)
class A:
#D
def f(self, x):
print(self, x)
a=A()
a.f(1)
This question already has answers here:
Decorating class methods - how to pass the instance to the decorator?
(3 answers)
Closed 2 years ago.
NOTE:
I've got a related question here:
How to access variables from a Class Decorator from within the method it's applied on?
I'm planning to write a fairly complicated decorator. Therefore, the decorator itself should be a class of its own. I know this is possible in Python (Python 3.8):
import functools
class MyDecoratorClass:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
def __call__(self, *args, **kwargs):
# do stuff before
retval = self.func(*args, **kwargs)
# do stuff after
return retval
#MyDecoratorClass
def foo():
print("foo")
Now my problem starts when I try to apply the decorator on a method instead of just a function - especially if it's a method from another class. Let me show you what I've tried:
1. Trial one: identity loss
The decorator MyDecoratorClass below doesn't (or shouldn't) do anything. It's just boilerplate code, ready to be put to use later on. The method foo() from class Foobar prints the object it is called on:
import functools
class MyDecoratorClass:
def __init__(self, method):
functools.update_wrapper(self, method)
self.method = method
def __call__(self, *args, **kwargs):
# do stuff before
retval = self.method(self, *args, **kwargs)
# do stuff after
return retval
class Foobar:
def __init__(self):
# initialize stuff
pass
#MyDecoratorClass
def foo(self):
print(f"foo() called on object {self}")
return
Now what you observe here is that the self in the foo() method gets swapped. It's no longer a Foobar() instance, but a MyDecoratorClass() instance instead:
>>> foobar = Foobar()
>>> foobar.foo()
foo() called from object <__main__.MyDecoratorClass object at 0x000002DAE0B77A60>
In other words, the method foo() loses its original identity. That brings us to the next trial.
2. Trial two: keep identity, but crash
I attempt to preserve the original identity of the foo() method:
import functools
class MyDecoratorClass:
def __init__(self, method):
functools.update_wrapper(self, method)
self.method = method
def __call__(self, *args, **kwargs):
# do stuff before
retval = self.method(self.method.__self__, *args, **kwargs)
# do stuff after
return retval
class Foobar:
def __init__(self):
# initialize stuff
pass
#MyDecoratorClass
def foo(self):
print(f"foo() called on object {self}")
return
Now let's test:
>>> foobar = Foobar()
>>> foobar.foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __call__
AttributeError: 'function' object has no attribute '__self__'
Yikes!
EDIT
Thank you #AlexHall and #juanpa.arrivillaga for your solutions. They both work. However, there is a subtle difference between them.
Let's first take a look at this one:
def __get__(self, obj, objtype) -> object:
temp = type(self)(self.method.__get__(obj, objtype))
print(temp)
return temp
I've introduced a temporary variable, just to print what __get__() returns. Each time you access the method foo(), this __get__() function returns a new MyDecoratorClass() instance:
>>> f = Foobar()
>>> func1 = f.foo
>>> func2 = f.foo
>>> print(func1 == func2)
>>> print(func1 is func2)
<__main__.MyDecoratorClass object at 0x000001B7E974D3A0>
<__main__.MyDecoratorClass object at 0x000001B7E96C5520>
False
False
The second approach (from #juanpa.arrivillaga) is different:
def __get__(self, obj, objtype) -> object:
temp = types.MethodType(self, obj)
print(temp)
return temp
The output:
>>> f = Foobar()
>>> func1 = f.foo
>>> func2 = f.foo
>>> print(func1 == func2)
>>> print(func1 is func2)
<bound method Foobar.foo of <__main__.Foobar object at 0x000002824BBEF4C0>>
<bound method Foobar.foo of <__main__.Foobar object at 0x000002824BBEF4C0>>
True
False
There is a subtle difference, but I'm not sure why.
Functions are descriptors and that's what allows them to auto-bind self. The easiest way to deal with this is to implement decorators using functions so that this is handled for you. Otherwise you need to explicitly invoke the descriptor. Here's one way:
import functools
class MyDecoratorClass:
def __init__(self, method):
functools.update_wrapper(self, method)
self.method = method
def __get__(self, instance, owner):
return type(self)(self.method.__get__(instance, owner))
def __call__(self, *args, **kwargs):
# do stuff before
retval = self.method(*args, **kwargs)
# do stuff after
return retval
class Foobar:
def __init__(self):
# initialize stuff
pass
#MyDecoratorClass
def foo(self, x, y):
print(f"{[self, x, y]=}")
#MyDecoratorClass
def bar(spam):
print(f"{[spam]=}")
Foobar().foo(1, 2)
bar(3)
Here the __get__ method creates a new instance of MyDecoratorClass with the bound method (previously self.method was just a function since no instance existed yet). Also note that __call__ just calls self.method(*args, **kwargs) - if self.method is now a bound method, the self of FooBar is already implied.
You can implement the descriptor protocol, an example of how functions do it (but in pure python) is available in the Descriptor HOWTO, translated to your case:
import functools
import types
class MyDecoratorClass:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
def __call__(self, *args, **kwargs):
# do stuff before
retval = self.func(*args, **kwargs)
# do stuff after
return retval
def __get__(self, obj, objtype=None):
if obj is None:
return self
return types.MethodType(self, obj)
Note, return types.MethodType(self, obj) is essentially equivalent to
return lambda *args, **kwargs : self.func(obj, *args, **kwargs)
Note from Kristof
Could it be that you meant this:
return types.MethodType(self, obj) is essentially equivalent to
return lambda *args, **kwargs : self(obj, *args, **kwargs)
Note that I replaced self.func(..) with self(..). I tried, and only this way I can ensure that the statements at # do stuff before and # do stuff after actually run.
There is an answered question about classmethod and property combined together: Using property() on classmethods
I still don't understand the cause of the problem, please help.
My understanding of classmethod was that it simply replaces self with cls. With this in mind I wrote several classmethods during the past few years and now I see I was wrong all that time.
So what is the difference between #classmethod and #cm from the code below?
def cm(func):
def decorated(self, *args, **kwargs):
return func(self.__class__, *args, **kwargs)
return decorated
class C:
V = 0
#property
#classmethod
def inc1(cls):
cls.V += 1
print("V1 =", cls.V)
#property
#cm
def inc3(cls):
cls.V += 3
print("V3 =", cls.V)
c = C()
#c.inc1 # fails with: TypeError: 'classmethod' object is not callable
c.inc3 # works
inc3 with cm works, but inc1 with classmethod does not.
what is the difference between #classmethod and #cm from the code below?
decorator is calling during class creation time before an instance is created.
In your case, since #cm returns func(self.__class__, *args, **kwargs), which is relied on self, it should be used as a instance method.
On the other hand, #classmethod is able to use before an instance is created.
def cm(func):
def decorated(self, *args, **kwargs):
return func(self.__class__, *args, **kwargs)
return decorated
class C:
#classmethod
def inc1(cls):
(blablabla)
#cm
def inc3(cls):
(blablabla)
C().inc1() # works as a instance method
C.inc1() # works as a classmethod
C().inc3() # works as a instance method
C.inc3() # TypeError: unbound method decorated() must be called with C instance as first argument (got nothing instead)
For a combination of classmethod and property, it could be done by return an customized object. Reference
class ClassPropertyDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
return self.f.__get__(obj, klass)()
def classproperty(func):
if not isinstance(func, (classmethod, staticmethod)):
func = classmethod(func)
return ClassPropertyDescriptor(func)
class C:
#classproperty
def inc1(cls):
(blablabla)
C.inc1 # works as a classmethod property
[Edit]
Q. What does the classmethod() call do with the method it decorates to achieve that?
The implementation can be done by using descriptor
class ClassMethodDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
def myclassmethod(func):
return ClassMethodDescriptor(func)
class C:
#myclassmethod
def inc1(cls):
(blablabla)
C.inc1() # works as a classmethod
Q. Why is the result not callable?
Because the implementation of ClassMethodDescriptor does not define __call__ function. Once using #property, it will return ClassMethodDescriptor which is not callable.
The difference is that classmethod is not callable, and cm method is callable. This means that when the property(class) makes a call to the inputed func(which it is supposed to do), it works as you'll except for cm, but will not work for classmethod since classmethod does not have a call implemented.
class method does not know anything about instance and does not require it.
instance method knows about it's instance and it's class.
class Foo:
some = 'some'
class Bar(Foo):
def __init__(self):
self.some = 'not some'
#classmethod
def cls_some(cls):
print(cls.some)
def instance_some(self):
print(self.some)
Bar.cls_some()
>>>some
Bar().instance_some()
>>>not some
Also as you can see you don't need an instance to call classmethod.
I'm having a hard time understanding what happens when I try to nest descriptors/decorators. I'm using python 2.7.
For example, let's take the following simplified versions of property and classmethod:
class MyProperty(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, obj, objtype=None):
print 'IN MyProperty.__get__'
return self.fget(obj)
class MyClassMethod(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, objtype=None):
print 'IN MyClassMethod.__get__'
def f(*args, **kwargs):
return self.f(objtype, *args, **kwargs)
return f
Trying to nest them:
class A(object):
# doesn't work:
#MyProperty
#MyClassMethod
def klsproperty(cls):
return 555
# works:
#MyProperty
def prop(self):
return 111
# works:
#MyClassMethod
def klsmethod(cls, x):
return x**2
% print A.klsproperty
IN MyProperty.__get__
...
TypeError: 'MyClassMethod' object is not callable
The __get__ method of the inner descriptor MyClassMethod is not getting called.
Failing to figure out why, I tried throwing in (what I think is) a no-op descriptor:
class NoopDescriptor(object):
def __init__(self, f):
self.f = f
def __get__(self, obj, objtype=None):
print 'IN NoopDescriptor.__get__'
return self.f.__get__(obj, objtype=objtype)
Trying to use the no-op descriptor/decorator in nesting:
class B(object):
# works:
#NoopDescriptor
#MyProperty
def prop1(self):
return 888
# doesn't work:
#MyProperty
#NoopDescriptor
def prop2(self):
return 999
% print B().prop1
IN NoopDescriptor.__get__
IN MyProperty.__get__
888
% print B().prop2
IN MyProperty.__get__
...
TypeError: 'NoopDescriptor' object is not callable
I don't understand why B().prop1 works and B().prop2 does not.
Questions:
What am I doing wrong? Why am I getting a object is not callable error?
What's the right way? e.g. what is the best way to define MyClassProperty while re-using MyClassMethod and MyProperty (or classmethod and property)
In this case, when the decorators are used without parameters, a decorator is called with the function it decorates as its parameter. The decorator's return value is used instead of the decorated function. So:
#MyProperty
def prop(self):
...
is equivalent to:
def prop(self):
...
prop = MyProperty(prop)
Since MyProperty implements the descriptor protocol, accessing A.prop will actually call A.prop.__get__(), and you've defined __get__ to call the object which was decorated (in this case, the original function/method), so everything works fine.
Now, in the nested case:
#MyProperty
#MyClassMethod
def prop(self):
...
The equivalent is:
def prop(self):
...
prop = MyClassMethod(prop) # prop is now instance of MyClassMethod
prop = MyProperty(prop) # prop is now instance of MyProperty
# (with fget == MyClassMethod instance)
Now, as before, accessing A.prop will actually call A.prop.__get__() (in MyProperty) which then tries to call the instance of MyClassMethod (the object which was decorated and stored in the fget attribute).
But the MyClassMethod does not have a __call__ method defined, so you get the error MyClassMethod is not callable.
And to address your second question: A property is already a class attribute - in your example, accessing A.prop will return the value of the property in the class object and A().prop will return the value of the property in an instance object (which can be the same as the class object if the instance did not override it).
You can make your code work if you make MyProperty apply the descriptor protocol to its wrapped object:
class MyProperty(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, obj, objtype=None):
print('IN MyProperty.__get__')
try:
return self.fget.__get__(obj, objtype)()
except AttributeError: # self.fget has no __get__ method
return self.fget(obj)
Now your example code works:
class A(object):
#MyProperty
#MyClassMethod
def klsproperty(cls):
return 555
print(A.klsproperty)
The output is:
IN MyProperty.__get__
IN MyClassMethod.__get__
555
I found the definitive answer to my own old question in Graham Dumpleton's fascinating blog.
In short, the decorators I wrote do not honour the descriptors protocol, by trying to call the wrapped function/object directly, instead of first giving them a chance to perform their "descriptor magic" (by calling their __get__() first).