Is it really OK to do object closeing/disposing in __del__? - python

I have been thinking about how I write classes in Python. More specifically how the constructor is implemented and how the object should be destroyed. I don't want to rely on CPython's reference counting to do object cleanup. This basically tells me I should use with statements to manage my object life times and that I need an explicit close/dispose method (this method could be called from __exit__ if the object is also a context manager).
class Foo(object):
def __init__(self):
pass
def close(self):
pass
Now, if all my objects behave in this way and all my code uses with statements or explicit calls to close() (or dispose()) I don't realy see the need for me to put any code in __del__. Should we really use __del__ to dispose of our objects?

Short answer : No.
Long answer: Using __del__ is tricky, mainly because it's not guaranteed to be called. That means you can't do things there that absolutely has to be done. This in turn means that __del__ basically only can be used for cleanups that would happen sooner or later anyway, like cleaning up resources that would be cleaned up when the process exits, so it doesn't matter if __del__ doesn't get called. Of course, these are also generally the same things Python will do for you. So that kinda makes __del__ useless.
Also, __del__ gets called when Python garbage collects, and you didn't want to wait for Pythons garbage collecting, which means you can't use __del__ anyway.
So, don't use __del__. Use __enter__/__exit__ instead.
FYI: Here is an example of a non-circular situation where the destructor did not get called:
class A(object):
def __init__(self):
print('Constructing A')
def __del__(self):
print('Destructing A')
class B(object):
a = A()
OK, so it's a class attribute. Evidently that's a special case. But it just goes to show that making sure __del__ gets called isn't straightforward. I'm pretty sure I've seen more non-circular situations where __del__ isn't called.

Not necessarily. You'll encounter problems when you have cyclic references. Eli Bendersky does a good job of explaining this in his blog post:
Safely using destructors in Python

If you are sure you will not go into cyclic references, then using __del__ in that way is OK: as soon as the reference count goes to zero, the CPython VM will call that method and destroy the object.
If you plan to use cyclic references - please think it very thoroughly, and check if weak references may help; in many cases, cyclic references are a first symptom of bad design.
If you have no control on the way your object is going to be used, then using __del__ may not be safe.
If you plan to use JPython or IronPython, __del__ is unreliable at all, because final object destruction will happen at garbage collection, and that's something you cannot control.
In sum, in my opinion, __del__ is usually perfectly safe and good; however, in many situation it could be better to make a step back, and try to look at the problem from a different perspective; a good use of try/except and of with contexts may be a more pythonic solution.

Related

Is __del__ really a destructor?

I do things mostly in C++, where the destructor method is really meant for destruction of an acquired resource. Recently I started with python (which is really a fun and fantastic), and I came to learn it has GC like java.
Thus, there is no heavy emphasis on object ownership (construction and destruction).
As far as I've learned, the __init__() method makes more sense to me in python than it does for ruby too, but the __del__() method, do we really need to implement this built-in function in our class? Will my class lack something if I miss __del__()? The one scenario I could see __del__() useful is, if I want to log something when destroying an object. Is there anything other than this?
In the Python 3 docs the developers have now made clear that destructor is in fact not the appropriate name for the method __del__.
object.__del__(self)
Called when the instance is about to be destroyed. This is also called a finalizer or (improperly) a destructor.
Note that the OLD Python 3 docs used to suggest that 'destructor' was the proper name:
object.__del__(self)
Called when the instance is about to be destroyed. This is also called a destructor. If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance.
From other answers but also from the Wikipedia:
In a language with an automatic garbage collection mechanism, it would be difficult to deterministically ensure the invocation of a destructor, and hence these languages are generally considered unsuitable for RAII [Resource Acquisition Is Initialization]
So you should almost never be implementing __del__, but it gives you the opportunity to do so in some (rare?) use cases
As the other answers have already pointed out, you probably shouldn't implement __del__ in Python. If you find yourself in the situation thinking you'd really need a destructor (for example if your class wraps a resource that needs to be explicitly closed) then the Pythonic way to go is using context managers.
Is del really a destructor?
No, __del__ method is not a destructor, is just a normal method you can call whenever you want to perform any operation, but it is always called before the garbage collector destroys the object.
Think of it like a clean or last will method.
So uncommon it is that I have learned about it today (and I'm long ago into python).
Memory is deallocated, files closed, ... by the GC. But you could need to perform some task with effects outside of the class.
My use case is about implementing some sort of RAII regarding some temporal directories. I'd like it to be removed no matter what.
Instead of removing it after the processing (which, after some change, was no longer run) I've moved it to the __del__ method, and it works as expected.
This is a very specific case, where we don't really care about when the method is called, as long as it's called before leaving the program. So, use with care.

Destructor in metaclass Singleton object

I'm modifying a legacy library that uses the singleton pattern through the metaclass approach.
The Singleton class, inheriting from type, defines de __call__ function.
Right now, my singleton object using this library are never deleted. I defined the __del__ method in the singleton classes and that function is never called.
Clarification: I have implemented one (meta)class named Singleton, that is used by several classes, using Singleton as __metaclass__.
For example, I have class A(object), that has __metaclass__ = Singleton. The A class has several members that I want to be destroyed when my program ends and the A object (the only one that can exist) is destroyed.
I tried defining __del__ method in A class, but it doesn't work.
Point 1: __del__() may not be called at process exit
The first thing to say is that
It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
From the python data model docs. Therefore you should not be relying on it to tidy up state that you need to tidy up at exit, and at the highest level, that's why your __del__() may not be being called. That's what atexit is for.
Point 2: predictable object lifetimes is an implementation detail in python
The next thing to say is that while CPython uses reference counting to enable it to detect that an object can be released without having to use the garbage collector (leading to more predictable CPU impact and potentially more efficient applications), it only takes one circular reference, one uncleared exception, one forgotten closure or one different python implementation to break, and so you should think really really hard about whether you want to rely on __del__() being called at a particular point.
Point 3: Singleton implementations generally maintain a global reference to the singleton instance
By the sound of it, I would guess your singleton metaclass (itself a singleton...) is retaining your singleton instance the first time __call__() is called. Since the metaclass is not released since it belongs to the module, which is itself retained by sys.modules, that reference is not going to go away by the time the program terminates, so even given a guaranteed prompt tidy up of all external references to the singleton being freed, your __del__() is not going to get called.
What you could try
Add an atexit handler when you create your singleton instance to do your necessary tidy up at process exit.
Also do that tidy up in the __del__() method if you want. E.g, you may decide for neatness / future extensibility (e.g. pluralizing the singleton) that you would like the singleton instance to tidy up after itself when it is no longer being used.
And if you implement a __del__() method expecting to want tidy up to be done during normal program execution, you will probably want to remove the atexit handler too.
If you would like your singleton to be cleaned up when no one is using it anymore, consider storing it on your metaclass using weakref so that you don't retain it yourself.

Is relying on __del__() for cleanup in Python unreliable?

I was reading about different ways to clean up objects in Python, and I have stumbled upon these questions (1, 2) which basically say that cleaning up using __del__() is unreliable and the following code should be avoid:
def __init__(self):
rc.open()
def __del__(self):
rc.close()
The problem is, I'm using exactly this code, and I can't reproduce any of the issues cited in the questions above. As far as my knowledge goes, I can't go for the alternative with with statement, since I provide a Python module for a closed-source software (testIDEA, anyone?) This software will create instances of particular classes and dispose of them, these instances have to be ready to provide services in between. The only alternative to __del__() that I see is to manually call open() and close() as needed, which I assume will be quite bug-prone.
I understand that when I'll close the interpreter, there's no guarantee that my objects will be destroyed correctly (and it doesn't bother me much, heck, even Python authors decided it was OK). Apart from that, am I playing with fire by using __del__() for cleanup?
You observe the typical issue with finalizers in garbage collected languages. Java has it, C# has it, and they all provide a scope based cleanup method like the Python with keyword to deal with it.
The main issue is, that the garbage collector is responsible for cleaning up and destroying objects. In C++ an object gets destroyed when it goes out of scope, so you can use RAII and have well defined semantics. In Python the object goes out of scope and lives on as long as the GC likes. Depending on your Python implementation this may be different. CPython with its refcounting based GC is rather benign (so you rarely see issues), while PyPy, IronPython and Jython might keep an object alive for a very long time.
For example:
def bad_code(filename):
return open(filename, 'r').read()
for i in xrange(10000):
bad_code('some_file.txt')
bad_code leaks a file handle. In CPython it doesn't matter. The refcount drops to zero and it is deleted right away. In PyPy or IronPython you might get IOErrors or similar issues, as you exhaust all available file descriptors (up to ulimit on Unix or 509 handles on Windows).
Scope based cleanup with a context manager and with is preferable if you need to guarantee cleanup. You know exactly when your objects will be finalized. But sometimes you cannot enforce this kind of scoped cleanup easily. Thats when you might use __del__, atexit or similar constructs to do a best effort at cleaning up. It is not reliable but better than nothing.
You can either burden your users with explicit cleanup or enforcing explicit scopes or you can take the gamble with __del__ and see some oddities now and then (especially interpreter shutdown).
There are a few problems with using __del__ to run code.
For one, it only works if you're actively keeping track of references, and even then, there's no guarantee that it will be run immediately unless you're manually kicking off garbage collections throughout your code. I don't know about you, but automatic garbage collection has pretty much spoiled me in terms of accurately keeping track of references. And even if you are super diligent in your code, you're also relying on other users that use your code being just as diligent when it comes to reference counts.
Two, there are lots of instances where __del__ is never going to run. Was there an exception while objects were being initialized and created? Did the interpreter exit? Is there a circular reference somewhere? Yep, lots that could go wrong here and very few ways to cleanly and consistently deal with it.
Three, even if it does run, it won't raise exceptions, so you can't handle exceptions from them like you can with other code. It's also nearly impossible to guarantee that the __del__ methods from various objects will run in any particular order. So the most common use case for destructors - cleaning up and deleting a bunch of objects - is kind of pointless and unlikely to go as planned.
If you actually want code to run, there are much better mechanisms -- context managers, signals/slots, events, etc.
If you're using CPython, then __del__ fires perfectly reliably and predictably as soon as an object's reference count hits zero. The docs at https://docs.python.org/3/c-api/intro.html state:
When an object’s reference count becomes zero, the object is deallocated. If it contains references to other objects, their reference count is decremented. Those other objects may be deallocated in turn, if this decrement makes their reference count become zero, and so on.
You can easily test and see this immediate cleanup happening yourself:
>>> class Foo:
... def __del__(self):
... print('Bye bye!')
...
>>> x = Foo()
>>> x = None
Bye bye!
>>> for i in range(5):
... print(Foo())
...
<__main__.Foo object at 0x7f037e6a0550>
Bye bye!
<__main__.Foo object at 0x7f037e6a0550>
Bye bye!
<__main__.Foo object at 0x7f037e6a0550>
Bye bye!
<__main__.Foo object at 0x7f037e6a0550>
Bye bye!
<__main__.Foo object at 0x7f037e6a0550>
Bye bye!
>>>
(Though if you want to test stuff involving __del__ at the REPL, be aware that the last evaluated expression's result gets stored as _, which counts as a reference.)
In other words, if your code is strictly going to be run in CPython, relying on __del__ is safe.

How to use __del__ in a reliable way?

I have learned that python does not guarantee that __del__ is called whenever an object is deleted.
In other words, del x does not necessarily invoke its destructor x.__del__().
If I want to ensure proper object cleanup, I should use a context manager (in a with statement).
I know it's stupid, but for a couple of reasons (please don't ask why) I am tied to a system with Python 2.4; therefore context managers are out of question (they were introduced in Python 2.5)
So I need a an alternative solution, and hence my question: are there best practices that would help me to use __del__ reliably? I am thinking in the direction of "if python provides such functionality, there must be a way it can be efficiently used (I'm just to stupid to figure out how)",,,
Or I am just being naive, should forget about __del__ and move on to a completely different approach?
In short: No, there is no way to ensure it gets called.
The answer is to implement context managers yourself. A with statement roughly translates to:
x.__enter__()
try:
...
finally:
x.__exit__()
So just do it manually. It is a little more complex than that, so I recommend reading PEP 343 to fully understand how context managers work.
One option is to call your cleaning up function close(), and then in future versions of python, people can easily use contextlib.closing to turn it into a real context manager.
Instead of __del__, give your class a method called something like close, then call that explicitly:
foo = Foo()
try:
foo.do_interesting_stuff()
finally:
foo.close()
For extra safety and forward-compatibility, have __exit__ and __del__ call close as well.

Guaranteeing a file close

I have a class where I create a file object in the constructor. This class also implements a finish() method as part of its interface and in this method I close the file object. The problem is that if I get an exception before this point, the file will not be closed. The class in question has a number of other methods that use the file object. Do I need to wrap all of these in a try finally clause or is there a better approach?
Thanks,
Barry
You could make your class a context-manager, and then wrap object creation and use of that class in a with-statement. See PEP 343 for details.
To make your class a context-manager, it has to implement the methods __enter__() and __exit__(). __enter__() is called when you enter the with-statement, and __exit__() is guaranteed to be called when you leave it, no matter how.
You could then use your class like this:
with MyClass() as foo:
# use foo here
If you acquire your resources in the constructor, you can make __enter__() simply return self without doing anything. __exit__() should just call your finish()-method.
For short lived file objects, a try/finally pair or the more succinct with-statement is recommended as a clean way to make sure the files are flushed and the related resources are released.
For long lived file objects, you can register with atexit() for an explicit close or just rely on the interpreter cleaning up before it exits.
At the interactive prompt, most people don't bother for simple experiments where there isn't much of a downside to leaving files unclosed or relying on refcounting or GC to close for you.
Closing your files is considered good technique. In reality though, not explicitly closing files rarely has any noticeable effects.
You can either have a try...finally pair, or make your class a context manager suitable for use in the with statement.

Categories

Resources