Develop Reference

How to get unbounded classmethod

I'm trying to "wrap" an existing classmethod, i.e.,
def Foo:
#classmethod
def bar(cls, x): return x + 2
old_bar = Foo.bar
def wrapped_bar(cls, x):
result = old_bar(cls, x) # Results in an error
return result
Foo.bar = wrapped_bar
It seems that Foo.bar is already bound with cls = Foo, how do I get the unbound version of the function bar?
[I'm not allowed to modify Foo, it exists in another codebase that I'm patching]

Suppose, you have:
>>> class Foo:
... #classmethod
... def bar(cls, x):
... return x*42
...
>>> Foo.bar(2)
84
Then one way is to access the name-space of your class directly. Then you should be able to access the classmethod object and obtain the decorated function available at the __func__ attribute:
>>> vars(Foo)['bar']
<classmethod object at 0x103eec520>
>>> vars(Foo)['bar'].__func__
<function Foo.bar at 0x1043e49d0>
Alternatively, it is accessible on the bound-method object itself:
>>> bound = Foo.bar
>>> bound
<bound method Foo.bar of <class '__main__.Foo'>>
>>> bound.__func__
<function Foo.bar at 0x1043e49d0>

The __func__ attribute holds the wrapped method. Note that your wrapper will also need the #classmethod decorator in order to attach properly (otherwise it'll act as an instance method).
#classmethod
def wrapped_bar(cls, x):
result = Foo.bar.__func__(cls, x)
return result
Foo.bar = wrapped_bar

How to access a caller method's __class__ attribute from a callee function in Python?

Here __class__ should not be confused with self.__class__ which I am already able to access with the inspect module:
import inspect
class A:
def __init__(self):
print(__class__.__name__) # I want to move this statement inside f
f()
class B(A):
pass
def f():
prev_frame = inspect.currentframe().f_back
self = prev_frame.f_locals["self"]
print(self.__class__.__name__)
B() # prints A B

The implicit __class__ reference is created at compile-time only if you actually reference it within the method (or use super). For example this code:
class Foo:
def bar(self):
print('bar', locals())
def baz(self):
print('baz', locals())
if False:
__class__
if __name__ == '__main__':
foo = Foo()
foo.bar()
foo.baz()
Produces this output:
bar {'self': <__main__.Foo object at 0x10f45f978>}
baz {'self': <__main__.Foo object at 0x10f45f978>, '__class__': <class '__main__.Foo'>}
To find the calling function's class (in most cases) you could chain together a few CPython-specific inspect incantations:
Find the calling function: How to get current function into a variable?
Find that function's class: Get defining class of unbound method object in Python 3
I wouldn't recommend it.

__getattr__ returning lambda function requiring one argument does not work

I am in the process of learning Python 3 and just ran into the getattr function. From what I can tell, it is invoked when the attribute call is not found in the class definition as a function or a variable.
In order to understand the behaviour, I wrote the following test class (based on what I've read):
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm is 'test':
return lambda x: "%s%s" % (x.foo, x.bar)
raise AttributeError(itm)
And I then initate my object and call the non-existent function test which, expectedly, returns the reference to the function:
t = Test("Foo", "Bar")
print(t.test)
<function Test.__getattr__.<locals>.<lambda> at 0x01A138E8>
However, if I call the function, the result is not the expected "FooBar", but an error:
print(t.test())
TypeError: <lambda>() missing 1 required positional argument: 'x'
In order to get my expected results, I need to call the function with the same object as the first parameter, like this:
print(t.test(t))
FooBar
I find this behaviour rather strange, as when calling p.some_function(), is said to add p as the first argument.
I would be grateful if someone could shine some light over this headache of mine. I am using PyDev in Eclipse.

__getattr__ return values are "raw", they don't behave like class attributes, invoking the descriptor protocol that plain methods involve that causes the creation of bound methods (where self is passed implicitly). To bind the function as a method, you need to perform the binding manually:
import types
...
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# Explicitly bind function to self
return types.MethodType(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
types.MethodType is poorly documented (the interactive help is more helpful), but basically, you pass it a user-defined function and an instance of a class and it returns a bound method that, when called, implicitly passes that instance as the first positional argument (the self argument).
Note that in your specific case, you could just rely on closure scope to make a zero-argument function continue to work:
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# No binding, but referring to self captures it in closure scope
return lambda: "%s%s" % (self.foo, self.bar)
raise AttributeError(itm)
Now it's not a bound method at all, just a function that happens to have captured self from the scope in which it was defined (the __getattr__ call). Which solution is best depends on your needs; creating a bound method is slightly slower, but gets a true bound method, while relying on closure scope is (trivially, ~10ns out of >400ns) faster, but returns a plain function (which may be a problem if, for example, it's passed as a callback to code that assumes it's a bound method and can have __self__ and __func__ extracted separately for instance).

To get what you want, you need a lambda that doesn't take arguments:
return lambda: "%s%s" % (self.foo, self.bar)
But you should really use a property for this, instead.
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test)
# FooBar
Note the lack of parentheses.
If you're absolutely determined that it must be a function, do this:
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return lambda: "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test())
# FooBar

You need to create something that behaves like a bound method, you could simply use functools.partial to bind the instance to the function:
from functools import partial
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm == 'test': # you shouldn't use "is" for comparisons!
return partial(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
The test:
t = Test("Foo", "Bar")
print(t.test)
# functools.partial(<function Test.__getattr__.<locals>.<lambda> at 0x0000020C70CA6510>, <__main__.Test object at 0x0000020C7217F8D0>)
print(t.test())
# FooBar

"I find this behaviour rather strange, as when calling
p.some_function(), is said to add p as the first argument."
some_function is actually a method, which is why it gets passed an instance implicitly when the method is "bound to an object." But plain functions don't work that way, only functions defined in the class body have this magic applied to them automagically. And actually, unbound methods (accessed via the class directly) function the same as normal functions! The terminology "bound and unbound" methods no longer applies, because in Python 3 we only have methods and functions (getting rid of the distinction between unbound methods and plain functions). When an instance is instantiated, accessing the attribute returns a method which implicitly calls the instance on invocation.
>>> class A:
... def method(self, x):
... return x
...
>>> a.method
<bound method A.method of <__main__.A object at 0x101a5b3c8>>
>>> type(a.method)
<class 'method'>
However, if you access the attribute of the class you'll see it's just a function:
>>> A.method
<function A.method at 0x101a64950>
>>> type(A.method)
<class 'function'>
>>> a = A()
Now, observe:
>>> bound = a.method
>>> bound(42)
42
>>> unbound = A.method
>>> unbound(42)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() missing 1 required positional argument: 'x'
But this is the magic of classes. Note, you can even add functions to classes dynamically, and they get magically turned into methods when you invoke them on an instance:
>>> A.method2 = lambda self, x: x*2
>>> a2 = A()
>>> a2.method2(4)
8
And, as one would hope, the behavior still applies to objects already created!
>>> a.method2(2)
4
Note, this doesn't work if you dynamically add to an instance:
>>> a.method3 = lambda self, x: x*3
>>> a.method3(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: <lambda>() missing 1 required positional argument: 'x'
You have to do the magic yourself:
>>> from types import MethodType
>>> a.method4 = MethodType((lambda self, x: x*4), a)
>>> a.method4(4)
16
>>>

Notice that if you do print(t.__getattr__) you get something like <bound method Test.__getattr__ of <__main__.Test object at 0x00000123FBAE4DA0>>. The key point is that methods defined on an object are said to be 'bound' and so always take the object as the first parameter. Your lambda function is just an anonymous function not 'bound' to anything, so for it to access the object it needs to be explicitly passed in.
I presume you are only doing this to experiment with using `__getattr__', as what you are doing could be much more easily achieved by making your lambda a method on the object.

Why set a bound method to python object create a circular reference?

I'm working in Python 2.7 and I fond that issue that puzzling me.
That is the simplest example:
>>> class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
>>> a = A()
>>> del a
DEL
That is OK like expected... now I'm trying to change the a() method of object a and what happen is that after change it I can't delete a any more:
>>> a = A()
>>> a.a = a.a
>>> del a
Just to do some checks I've print the a.a reference before and after the assignment
>>> a = A()
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>
>>> a.a = a.a
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>
Finally I used objgraph module to try to understand why the object is not released:
>>> b = A()
>>> import objgraph
>>> objgraph.show_backrefs([b], filename='pre-backref-graph.png')
>>> b.a = b.a
>>> objgraph.show_backrefs([b], filename='post-backref-graph.png')
As you can see in the post-backref-graph.png image there is a __self__ references in b that have no sense for me because the self references of instance method should be ignored (as was before the assignment).
Somebody can explain why that behaviour and how can I work around it?

When you write a.a, it effectively runs:
A.a.__get__(a, A)
because you are not accessing a pre-bound method but the class' method that is being
bound at runtime.
When you do
a.a = a.a
you effectively "cache" the act of binding the method. As the bound method has a reference to the object (obviously, as it has to pass self to the function) this creates a circular reference.
So I'm modelling your problem like:
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls(function):
def inner(*args, **kwargs):
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls(a.a)
a.a()
You can use weak references to bind on demand inside log_all_calls like:
import weakref
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls_weakmethod(method):
cls = method.im_class
func = method.im_func
instance_ref = weakref.ref(method.im_self)
del method
def inner(*args, **kwargs):
instance = instance_ref()
if instance is None:
raise ValueError("Cannot call weak decorator with dead instance")
function = func.__get__(instance, cls)
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls_weakmethod(a.a)
a.a()
This is really ugly, so I would rather extract it out to make a weakmethod decorator:
import weakref
def weakmethod(method):
cls = method.im_class
func = method.im_func
instance_ref = weakref.ref(method.im_self)
del method
def inner(*args, **kwargs):
instance = instance_ref()
if instance is None:
raise ValueError("Cannot call weak method with dead instance")
return func.__get__(instance, cls)(*args, **kwargs)
return inner
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls(function):
def inner(*args, **kwargs):
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls(weakmethod(a.a))
a.a()
Done!
FWIW, not only does Python 3.4 not have these issues, it also has WeakMethod pre-built for you.

Veedrac's answer about the bound method keeping a reference to the instance is only part of the answer. CPython's garbage collector knows how to detect and handle cyclic references - except when some object that's part of the cycle has a __del__ method, as mentioned here https://docs.python.org/2/library/gc.html#gc.garbage :
Objects that have __del__() methods and are part of a reference cycle
cause the entire reference cycle to be uncollectable, including
objects not necessarily in the cycle but reachable only from it.
Python doesn’t collect such cycles automatically because, in general,
it isn’t possible for Python to guess a safe order in which to run the
__del__() methods. (...) It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods, and
garbage can be examined in that case to verify that no such cycles are
being created.
IOW : remove your __del__ method and you should be fine.
EDIT: wrt/ your comment :
I use it on the object as function a.a = functor(a.a). When the test
is done I would like replace the functor by the original method.
Then the solution is plain and simple:
a = A()
a.a = functor(a.a)
test(a)
del a.a
Until you explicitely bind it, a has no 'a' instance atribute, so it's looked up on the class and a new method instance is returned (cf https://wiki.python.org/moin/FromFunctionToMethod for more on this). This method instance is then called, and (usually) discarded.

As to why Python does this. Technically all objects contain circular references if they have methods. However, garbage collection would take much longer if the garbage collector had to do explicit checks on an objects methods to make sure freeing the object wouldn't cause a problem. As such Python stores the methods separately from an object's __dict__. So when you write a.a = a.a, you are shadowing the method with itself in the a field on the object. And thus, there is an explicit reference to the method which prevents the object from being freed properly.
The solution to your problem is not bother to keep a "cache" of the original method and just delete the shadowed variable when you're done with it. This will unshadow the method and make it available again.
>>> class A(object):
... def __del__(self):
... print("del")
... def method(self):
... print("method")
>>> a = A()
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method = a.method
>>> vars(a)
{'method': <bound method A.method of <__main__.A object at 0x0000000001F07940>>}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a.method
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a
del
Here vars shows what's in the __dict__ attribute of an object. Note how __dict__ doesn't contain a reference to itself even though a.__dict__ is valid. dir produces a list of all the attributes reachable from the given object. Here we can see all the attributes and methods on an object and all the methods and attributes of its classes and their bases. This shows that the bound method of a is stored in place separate to where a's attributes are stored.

Dynamic method generation in python

I currently have this code:
class Generator(object):
def __getattr__(self, name):
def f(self):
return ("Result of"+name, self)
f.__name__ = name
return f
def foo(self):
pass
g = Generator()
print g.foo
print Generator.foo
print g.bar
print Generator.bar
Which gives:
<bound method Generator.foo of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.foo>
<function bar at 0x00A9DE70>
AttributeError: type object 'Generator' has no attribute 'bar'
What do I have to do to make it give:
<bound method Generator.foo of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.foo>
<bound method Generator.bar of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.bar>

Here's a metaclass that adds the __getattr__ function from the class definition back to the metaclass itself. This avoids having to define the function in multiple places, or as a separate global function defined beforehand and added individually to the metaclass and class.
class Meta(type):
def __new__(mcls, name, bases, dikt):
fgetattr = dikt.get('__getattr__')
if fgetattr is not None:
setattr(mcls, '__getattr__', fgetattr)
return super(Meta, mcls).__new__(mcls, name, bases, dikt)
class Generator(object):
__metaclass__ = Meta
def __getattr__(obj, name):
def f(self):
return "Result of %s for %r" % (name, self)
f.__name__ = name
if isinstance(obj, type):
setattr(obj, name, f)
else:
setattr(type(obj), name, f)
return getattr(obj, name)
Rather than directly create the method via the dynamic function's __get__ descriptor method, I think it's better to store the function in the class dict and rely on getattr to return the proper bound/unbound method. Subsequent attribute access will use the function from the class. Since the same __getattr__ function is used for both the class and the instance, an isinstance check is required to ensure the dynamic function gets stored to the class and not the instance.
In Python 3, getting the function as an attribute of the class merely returns the function since unbound methods were removed from the language. Also, the metaclass syntax has changed to a keyword argument in the class definition line.
Test:
>>> g = Generator()
>>> g.foo
<bound method Generator.foo of <__main__.Generator object at 0xb7248c2c>>
>>> Generator.foo
<unbound method Generator.foo>
>>> g.bar
<bound method Generator.bar of <__main__.Generator object at 0xb7248c2c>>
>>> Generator.bar
<unbound method Generator.bar>
>>> g.foo()
'Result of foo for <__main__.Generator object at 0xb7248c2c>'
>>> Generator.foo(g)
'Result of foo for <__main__.Generator object at 0xb7248c2c>'
>>> 'foo' in vars(Generator), 'bar' in vars(Generator)
(True, True)

__getattr__() only works on instances, you will need to make the function on a metaclass on Generator to get the behavior your want.

You have two problems.
You only set the dynamic function on the instance g (not the class Generator), so Generator.bar is not defined.
You don't wrap it in a MethodType, so you get a function instead of a method.
For 1, if you always call g.foo before you call Generator.foo, you can just add the line
setattr(self.__class__, name, f)
inside __getattr__, which will bind the name as a method on the class. Otherwise, you will need a custom __getattr__ on the type object, which means that you have to make it an instance of a custom class i.e. write your own metaclass.
For 2, see #thg435's answer. Note that this is icky because of backwards compatibility in Python 2, and has been neatened considerably in Py3k -- now what you are doing would basically work. It's because of the automatic injection of self.

For the first case (g.bar), replace return f with return MethodType(f, self).