I'm working in Python 2.7 and I fond that issue that puzzling me.
That is the simplest example:
>>> class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
>>> a = A()
>>> del a
DEL
That is OK like expected... now I'm trying to change the a() method of object a and what happen is that after change it I can't delete a any more:
>>> a = A()
>>> a.a = a.a
>>> del a
Just to do some checks I've print the a.a reference before and after the assignment
>>> a = A()
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>
>>> a.a = a.a
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>
Finally I used objgraph module to try to understand why the object is not released:
>>> b = A()
>>> import objgraph
>>> objgraph.show_backrefs([b], filename='pre-backref-graph.png')
>>> b.a = b.a
>>> objgraph.show_backrefs([b], filename='post-backref-graph.png')
As you can see in the post-backref-graph.png image there is a __self__ references in b that have no sense for me because the self references of instance method should be ignored (as was before the assignment).
Somebody can explain why that behaviour and how can I work around it?
When you write a.a, it effectively runs:
A.a.__get__(a, A)
because you are not accessing a pre-bound method but the class' method that is being
bound at runtime.
When you do
a.a = a.a
you effectively "cache" the act of binding the method. As the bound method has a reference to the object (obviously, as it has to pass self to the function) this creates a circular reference.
So I'm modelling your problem like:
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls(function):
def inner(*args, **kwargs):
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls(a.a)
a.a()
You can use weak references to bind on demand inside log_all_calls like:
import weakref
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls_weakmethod(method):
cls = method.im_class
func = method.im_func
instance_ref = weakref.ref(method.im_self)
del method
def inner(*args, **kwargs):
instance = instance_ref()
if instance is None:
raise ValueError("Cannot call weak decorator with dead instance")
function = func.__get__(instance, cls)
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls_weakmethod(a.a)
a.a()
This is really ugly, so I would rather extract it out to make a weakmethod decorator:
import weakref
def weakmethod(method):
cls = method.im_class
func = method.im_func
instance_ref = weakref.ref(method.im_self)
del method
def inner(*args, **kwargs):
instance = instance_ref()
if instance is None:
raise ValueError("Cannot call weak method with dead instance")
return func.__get__(instance, cls)(*args, **kwargs)
return inner
class A(object):
def __del__(self):
print("DEL")
def a(self):
pass
def log_all_calls(function):
def inner(*args, **kwargs):
print("Calling {}".format(function))
try:
return function(*args, **kwargs)
finally:
print("Called {}".format(function))
return inner
a = A()
a.a = log_all_calls(weakmethod(a.a))
a.a()
Done!
FWIW, not only does Python 3.4 not have these issues, it also has WeakMethod pre-built for you.
Veedrac's answer about the bound method keeping a reference to the instance is only part of the answer. CPython's garbage collector knows how to detect and handle cyclic references - except when some object that's part of the cycle has a __del__ method, as mentioned here https://docs.python.org/2/library/gc.html#gc.garbage :
Objects that have __del__() methods and are part of a reference cycle
cause the entire reference cycle to be uncollectable, including
objects not necessarily in the cycle but reachable only from it.
Python doesn’t collect such cycles automatically because, in general,
it isn’t possible for Python to guess a safe order in which to run the
__del__() methods. (...) It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods, and
garbage can be examined in that case to verify that no such cycles are
being created.
IOW : remove your __del__ method and you should be fine.
EDIT: wrt/ your comment :
I use it on the object as function a.a = functor(a.a). When the test
is done I would like replace the functor by the original method.
Then the solution is plain and simple:
a = A()
a.a = functor(a.a)
test(a)
del a.a
Until you explicitely bind it, a has no 'a' instance atribute, so it's looked up on the class and a new method instance is returned (cf https://wiki.python.org/moin/FromFunctionToMethod for more on this). This method instance is then called, and (usually) discarded.
As to why Python does this. Technically all objects contain circular references if they have methods. However, garbage collection would take much longer if the garbage collector had to do explicit checks on an objects methods to make sure freeing the object wouldn't cause a problem. As such Python stores the methods separately from an object's __dict__. So when you write a.a = a.a, you are shadowing the method with itself in the a field on the object. And thus, there is an explicit reference to the method which prevents the object from being freed properly.
The solution to your problem is not bother to keep a "cache" of the original method and just delete the shadowed variable when you're done with it. This will unshadow the method and make it available again.
>>> class A(object):
... def __del__(self):
... print("del")
... def method(self):
... print("method")
>>> a = A()
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method = a.method
>>> vars(a)
{'method': <bound method A.method of <__main__.A object at 0x0000000001F07940>>}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a.method
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a
del
Here vars shows what's in the __dict__ attribute of an object. Note how __dict__ doesn't contain a reference to itself even though a.__dict__ is valid. dir produces a list of all the attributes reachable from the given object. Here we can see all the attributes and methods on an object and all the methods and attributes of its classes and their bases. This shows that the bound method of a is stored in place separate to where a's attributes are stored.
Related
I'm trying to "wrap" an existing classmethod, i.e.,
def Foo:
#classmethod
def bar(cls, x): return x + 2
old_bar = Foo.bar
def wrapped_bar(cls, x):
result = old_bar(cls, x) # Results in an error
return result
Foo.bar = wrapped_bar
It seems that Foo.bar is already bound with cls = Foo, how do I get the unbound version of the function bar?
[I'm not allowed to modify Foo, it exists in another codebase that I'm patching]
Suppose, you have:
>>> class Foo:
... #classmethod
... def bar(cls, x):
... return x*42
...
>>> Foo.bar(2)
84
Then one way is to access the name-space of your class directly. Then you should be able to access the classmethod object and obtain the decorated function available at the __func__ attribute:
>>> vars(Foo)['bar']
<classmethod object at 0x103eec520>
>>> vars(Foo)['bar'].__func__
<function Foo.bar at 0x1043e49d0>
Alternatively, it is accessible on the bound-method object itself:
>>> bound = Foo.bar
>>> bound
<bound method Foo.bar of <class '__main__.Foo'>>
>>> bound.__func__
<function Foo.bar at 0x1043e49d0>
The __func__ attribute holds the wrapped method. Note that your wrapper will also need the #classmethod decorator in order to attach properly (otherwise it'll act as an instance method).
#classmethod
def wrapped_bar(cls, x):
result = Foo.bar.__func__(cls, x)
return result
Foo.bar = wrapped_bar
Here __class__ should not be confused with self.__class__ which I am already able to access with the inspect module:
import inspect
class A:
def __init__(self):
print(__class__.__name__) # I want to move this statement inside f
f()
class B(A):
pass
def f():
prev_frame = inspect.currentframe().f_back
self = prev_frame.f_locals["self"]
print(self.__class__.__name__)
B() # prints A B
The implicit __class__ reference is created at compile-time only if you actually reference it within the method (or use super). For example this code:
class Foo:
def bar(self):
print('bar', locals())
def baz(self):
print('baz', locals())
if False:
__class__
if __name__ == '__main__':
foo = Foo()
foo.bar()
foo.baz()
Produces this output:
bar {'self': <__main__.Foo object at 0x10f45f978>}
baz {'self': <__main__.Foo object at 0x10f45f978>, '__class__': <class '__main__.Foo'>}
To find the calling function's class (in most cases) you could chain together a few CPython-specific inspect incantations:
Find the calling function: How to get current function into a variable?
Find that function's class: Get defining class of unbound method object in Python 3
I wouldn't recommend it.
I'm investigating if I can implement an easy callback functionality in python. I thought I might be able to use weakref.WeakSet for this, but there is clearly something I'm missing or have misunderstood. As you can see in the code I first tried with a list of call back methods in 'ClassA' objects, but realized that this would keep objects that have been added to the list of callbacks alive. Instead I tried using weakref.WeakSet but that doesnt do the trick either (at least not en this way). Comments in the last four lines of code explain what I want to happen.
Can anyone help me with this?
from weakref import WeakSet
class ClassA:
def __init__(self):
#self.destroyCallback=[]
self.destroyCallback=WeakSet()
def __del__(self):
print('ClassA object %d is being destroyed' %id(self))
for f in self.destroyCallback:
f(self)
class ClassB:
def destroyedObjectListener(self,obj):
print('ClassB object %d is called because obj %d is being destroyed'%(id(self),id(obj)))
a1=ClassA()
a2=ClassA()
b=ClassB()
a1.destroyCallback.add(b.destroyedObjectListener)
#a1.destroyCallback.append(b.destroyedObjectListener)
print('destroyCallback len() of obj: %d is: %d'%(id(a1),len(a1.destroyCallback))) # should be 1
a2.destroyCallback.add(b.destroyedObjectListener)
#a2.destroyCallback.append(b.destroyedObjectListener)
print('destroyCallback len() of obj: %d is: %d'%(id(a2),len(a2.destroyCallback))) # should be 1
del a1 # Should call b.destroyedObjectListener(self) in its __del__ method
del b # should result in no strong refs to b so a2's WeakSet should automatically remove added item
print('destroyCallback len() of obj: %d is: %d'%(id(a2),len(a2.destroyCallback))) # should be 0
del a2 # Should call __del__ method
UPDATE: solution based on the accepted answer can be found on github: git#github.com:thgis/PythonEvent.git
You cannot create weak references to method objects. Method objects are short lived; they are created on the fly as you access the name on the instance. See the descriptor howto how that works.
When you access a method name, a new method object is created for you, and when you then add that method to the WeakSet, no other references exist to it anymore, so garbage collection happily cleans it up again.
You'll have to store something less transient. Storing instance objects themselves would work, then call a predefined method on the registered callbacks:
def __del__(self):
for f in self.destroyCallback:
f.destroyedObjectListener(self)
and to register:
a1.destroyCallback.add(b)
You can also make b itself a callable by giving it a __call__ method:
class ClassB:
def __call__(self,obj):
print('ClassB object %d is called because obj %d '
'is being destroyed' % (id(self), id(obj)))
Another approach would be to store a reference to the underlying function object plus a reference to the instance:
import weakref
class ClassA:
def __init__(self):
self._callbacks = []
def registerCallback(self, callback):
try:
# methods
callback_ref = weakref.ref(callback.__func__), weakref.ref(callback.__self__)
except AttributeError:
callback_ref = weakref.ref(callback), None
self._callbacks.append(callback_ref)
def __del__(self):
for callback_ref in self._callbacks:
callback, arg = callback_ref[0](), callback_ref[1]
if arg is not None:
# method
arg = arg()
if arg is None:
# instance is gone
continue
callback(arg, self)
continue
else:
if callback is None:
# callback has been deleted already
continue
callback(self)
Demo:
>>> class ClassB:
... def listener(self, deleted):
... print('ClassA {} was deleted, notified ClassB {}'.format(id(deleted), id(self)))
...
>>> def listener1(deleted):
... print('ClassA {} was deleted, notified listener1'.format(id(deleted)))
...
>>> def listener2(deleted):
... print('ClassA {} was deleted, notified listener2'.format(id(deleted)))
...
>>> # setup, one ClassA and 4 listeners (2 methods, 2 functions)
...
>>> a = ClassA()
>>> b1 = ClassB()
>>> b2 = ClassB()
>>> a.registerCallback(b1.listener)
>>> a.registerCallback(b2.listener)
>>> a.registerCallback(listener1)
>>> a.registerCallback(listener2)
>>>
>>> # deletion, we delete one instance of ClassB, and one function
...
>>> del b1
>>> del listener1
>>>
>>> # Deleting the ClassA instance will only notify the listeners still remaining
...
>>> del a
ClassA 4435440336 was deleted, notified ClassB 4435541648
ClassA 4435440336 was deleted, notified listener2
Try the following changes:
To update the WeakSet:
a1.destroyCallback.add(b)
so the WeakSet holds a reference to b.
Then in the __del__ method of ClassA, trigger the callback like this:
for f in self.destroyCallback:
f.destroyedObjectListener(self)
I'm investigating if I can implement an easy callback functionality in python. I thought I might be able to use weakref.WeakSet for this, but there is clearly something I'm missing or have misunderstood. As you can see in the code I first tried with a list of call back methods in 'ClassA' objects, but realized that this would keep objects that have been added to the list of callbacks alive. Instead I tried using weakref.WeakSet but that doesnt do the trick either (at least not en this way). Comments in the last four lines of code explain what I want to happen.
Can anyone help me with this?
from weakref import WeakSet
class ClassA:
def __init__(self):
#self.destroyCallback=[]
self.destroyCallback=WeakSet()
def __del__(self):
print('ClassA object %d is being destroyed' %id(self))
for f in self.destroyCallback:
f(self)
class ClassB:
def destroyedObjectListener(self,obj):
print('ClassB object %d is called because obj %d is being destroyed'%(id(self),id(obj)))
a1=ClassA()
a2=ClassA()
b=ClassB()
a1.destroyCallback.add(b.destroyedObjectListener)
#a1.destroyCallback.append(b.destroyedObjectListener)
print('destroyCallback len() of obj: %d is: %d'%(id(a1),len(a1.destroyCallback))) # should be 1
a2.destroyCallback.add(b.destroyedObjectListener)
#a2.destroyCallback.append(b.destroyedObjectListener)
print('destroyCallback len() of obj: %d is: %d'%(id(a2),len(a2.destroyCallback))) # should be 1
del a1 # Should call b.destroyedObjectListener(self) in its __del__ method
del b # should result in no strong refs to b so a2's WeakSet should automatically remove added item
print('destroyCallback len() of obj: %d is: %d'%(id(a2),len(a2.destroyCallback))) # should be 0
del a2 # Should call __del__ method
UPDATE: solution based on the accepted answer can be found on github: git#github.com:thgis/PythonEvent.git
You cannot create weak references to method objects. Method objects are short lived; they are created on the fly as you access the name on the instance. See the descriptor howto how that works.
When you access a method name, a new method object is created for you, and when you then add that method to the WeakSet, no other references exist to it anymore, so garbage collection happily cleans it up again.
You'll have to store something less transient. Storing instance objects themselves would work, then call a predefined method on the registered callbacks:
def __del__(self):
for f in self.destroyCallback:
f.destroyedObjectListener(self)
and to register:
a1.destroyCallback.add(b)
You can also make b itself a callable by giving it a __call__ method:
class ClassB:
def __call__(self,obj):
print('ClassB object %d is called because obj %d '
'is being destroyed' % (id(self), id(obj)))
Another approach would be to store a reference to the underlying function object plus a reference to the instance:
import weakref
class ClassA:
def __init__(self):
self._callbacks = []
def registerCallback(self, callback):
try:
# methods
callback_ref = weakref.ref(callback.__func__), weakref.ref(callback.__self__)
except AttributeError:
callback_ref = weakref.ref(callback), None
self._callbacks.append(callback_ref)
def __del__(self):
for callback_ref in self._callbacks:
callback, arg = callback_ref[0](), callback_ref[1]
if arg is not None:
# method
arg = arg()
if arg is None:
# instance is gone
continue
callback(arg, self)
continue
else:
if callback is None:
# callback has been deleted already
continue
callback(self)
Demo:
>>> class ClassB:
... def listener(self, deleted):
... print('ClassA {} was deleted, notified ClassB {}'.format(id(deleted), id(self)))
...
>>> def listener1(deleted):
... print('ClassA {} was deleted, notified listener1'.format(id(deleted)))
...
>>> def listener2(deleted):
... print('ClassA {} was deleted, notified listener2'.format(id(deleted)))
...
>>> # setup, one ClassA and 4 listeners (2 methods, 2 functions)
...
>>> a = ClassA()
>>> b1 = ClassB()
>>> b2 = ClassB()
>>> a.registerCallback(b1.listener)
>>> a.registerCallback(b2.listener)
>>> a.registerCallback(listener1)
>>> a.registerCallback(listener2)
>>>
>>> # deletion, we delete one instance of ClassB, and one function
...
>>> del b1
>>> del listener1
>>>
>>> # Deleting the ClassA instance will only notify the listeners still remaining
...
>>> del a
ClassA 4435440336 was deleted, notified ClassB 4435541648
ClassA 4435440336 was deleted, notified listener2
Try the following changes:
To update the WeakSet:
a1.destroyCallback.add(b)
so the WeakSet holds a reference to b.
Then in the __del__ method of ClassA, trigger the callback like this:
for f in self.destroyCallback:
f.destroyedObjectListener(self)
class A:
def foo(self):
print "foo()"
getattr(A, foo) # True
A.foo() # error
getattr(A(), foo) # True
A().foo() # prints "foo()"
That being said, here is my problem:
I wish to store test case meta information as attributes of the Test Case class objects themselves, not on instances of them.
I have a list of attribute names to extract, but if there is an instance method of the same name, then getattr(class_obj, attr) will return True, but getattr(class_obj, attr)() raises an Error.
Is there a way to tell getattr not to include attributes of the instantiated class and only of the class object itself?
EDIT: I tried accessing class_obj.__dict__ directly (which I understand is bad practice), but it does not include some attributes like __name__
EDIT: Rephrase of the question. Is there a way to differentiate between methods of the class obj and the methods of an instance of the class?
Is this good enough?
import types
class Test(object):
#staticmethod
def foo():
print 'foo'
def bar(self):
print 'bar'
In combination with:
>>>(isinstance(getattr(Test, 'foo'), types.FunctionType),
isinstance(getattr(Test, 'bar'), types.FunctionType))
True, False
You can also use the inspect module:
>>> inspect.isfunction(Test.foo)
True
>>> inspect.isfunction(Test.bar)
False
With a little additional work you can even distinguish class methods from instance methods and static methods:
import inspect
def get_type(cls, attr):
try:
return [a.kind for a in inspect.classify_class_attrs(cls) if a.name == attr][0]
except IndexError:
return None
class Test(object):
#classmethod
def foo(cls):
print 'foo'
def bar(self):
print 'bar'
#staticmethod
def baz():
print 'baz'
You can use it as:
>>> get_type(Test, 'foo')
'class method'
>>> get_type(Test, 'bar')
'method'
>>> get_type(Test, 'baz')
'static method'
>>> get_type(Test, 'nonexistant')
None
Your results from an incorrect definition of foo, not any underlying semantics of class attributes. By default, a function declared inside a class is an instance method, which must take at least one argument, an instance of the class. Conventionally, it is referred to as self:
class A:
def foo(self):
print "foo()"
Normally, you would call such a method like this:
a = A()
a.foo() # passes the object 'a' implicitly as the value of the parameter 'self'
but this is legal as well
a = A()
A.foo(a) # pass the object 'a' explicitly as the value of the parameter 'self'
In order to define a function inside a class that doesn't take any such implicit arguments, you need to decorate it with the #staticmethod decorator:
class A:
#staticmethod
def foo():
print "foo()"
Now, you can call foo the way you tried to previously:
>>> A.foo()
foo()
You want something like this:
from inspect import ismethod
from collections import Callable
def can_i_call(func):
if not isinstance(func, Callable):
# not a callable at all
return False
if not ismethod(func):
# regular function or class or whatever
return True
# func is a method
return func.im_self is not None
Note: this will only test whether or not an attempt to call will error out because you're calling an unbound method without a self. It doesn't guarantee that func() will succeed, i.e. not fail for any other reason.