Instance method is not replaced in python [duplicate] - python

I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>

Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().

As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.

The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).

For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091

TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.

Related

An object that implements "iterator protocol" by means of individual attributes is not recognised as iterable [duplicate]

I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.

Differences between methods of a class, which are "function" and which are "bound method"?

I have experimented a little. By checking __dict__ of a class or an instance, I can see some method has type function and some bound method. The experiment is messy, and I can't figure the following questions out.
In Python 3, what are the differences between methods of a class or instance, which are "function" and which are "bound method"?
How are they created respectively?
Can they both be called on a class and on an instance? Will they both be implicitly given an instance as their first argument?
Is "bound method" an attribute of a class or an instance of a class?
Thanks.
This answer will be really technical, I hope it's still understandable though. The problem is that it requires knowledge of the descriptor protocol to understand how methods in Python work.
All functions in Python 3 are descriptors, to be precise they are non-data descriptors. That means they implements a __get__ method - but no __set__ method.
That's interesting because descriptors can do (almost) anything if they are looked up on a class or an instance.
By the definition of the __get__ method in Pythons data model:
object.__get__(self, instance, owner)
Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). owner is always the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner. This method should return the (computed) attribute value or raise an AttributeError exception.
So what does this have to do with the difference between function and bound_method?
It's easy, a function accessed through __get__ with an instance=None will return itself:
>>> def func(x): return x
>>> func.__get__(None, object)
<function __main__.func>
>>> func.__get__(None, object) is func
True
While it will be a bound_method if accessed with an not-None instance:
>>> func.__get__(object())
<bound method func of <object object at 0x00000155614A0610>>
It's basically just a wrapper around func with the instance stored:
>>> m = func.__get__(object())
>>> m.__self__ # stored instance
<object at 0x155614a0650>
>>> m.__func__ # stored function
<function __main__.func>
However, when called, it will pass the instance as first argument to the wrapped function:
>>> m()
<object at 0x155614a0650>
So, bound methods will pass the instance as first argument, while functions do not (they requires all attributes).
So when you look at a class all normal methods will display as functions while all normal methods on an instance will be bound methods.
Why did I mention normal methods? Because you can define arbitrary descriptors. For example the Python built-ins already contain several exceptions:
classmethod
staticmethod
property (this is in fact a data-descriptor so I'll neglect it in the following discussion)
classmethods will display as bound method even when looked up on the class. That's because they are meant to be callable on the class and pass the class to the function, no matter if they are called on the class or the instance:
class Test(object):
#classmethod
def func(x):
return x
>>> Test.func
<bound method Test.func of <class '__main__.Test'>>
>>> Test().func
<bound method Test.func of <class '__main__.Test'>>
And staticmethods always display as functions because they never pass anything additional to the function:
class Test(object):
#staticmethod
def func(x):
return x
>>> Test().func
<function __main__.Test.func>
>>> Test.func
<function __main__.Test.func>
So it's easily possible to see also bound methods on the class (e.g. classmethods) and likewise one could also find normal functions on instances (e.g. staticmethods).

Why does rebinding the __setitem__ method of a UserDict instance have no effect on updates to the object? [duplicate]

I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.

defining __call__ of object in __init__ in Python 3.4 [duplicate]

I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.

Replacing __iter__ on a Python class instance doesn't result in the new function being called [duplicate]

I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.

Categories

Resources