I'm modifying a legacy library that uses the singleton pattern through the metaclass approach.
The Singleton class, inheriting from type, defines de __call__ function.
Right now, my singleton object using this library are never deleted. I defined the __del__ method in the singleton classes and that function is never called.
Clarification: I have implemented one (meta)class named Singleton, that is used by several classes, using Singleton as __metaclass__.
For example, I have class A(object), that has __metaclass__ = Singleton. The A class has several members that I want to be destroyed when my program ends and the A object (the only one that can exist) is destroyed.
I tried defining __del__ method in A class, but it doesn't work.
Point 1: __del__() may not be called at process exit
The first thing to say is that
It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
From the python data model docs. Therefore you should not be relying on it to tidy up state that you need to tidy up at exit, and at the highest level, that's why your __del__() may not be being called. That's what atexit is for.
Point 2: predictable object lifetimes is an implementation detail in python
The next thing to say is that while CPython uses reference counting to enable it to detect that an object can be released without having to use the garbage collector (leading to more predictable CPU impact and potentially more efficient applications), it only takes one circular reference, one uncleared exception, one forgotten closure or one different python implementation to break, and so you should think really really hard about whether you want to rely on __del__() being called at a particular point.
Point 3: Singleton implementations generally maintain a global reference to the singleton instance
By the sound of it, I would guess your singleton metaclass (itself a singleton...) is retaining your singleton instance the first time __call__() is called. Since the metaclass is not released since it belongs to the module, which is itself retained by sys.modules, that reference is not going to go away by the time the program terminates, so even given a guaranteed prompt tidy up of all external references to the singleton being freed, your __del__() is not going to get called.
What you could try
Add an atexit handler when you create your singleton instance to do your necessary tidy up at process exit.
Also do that tidy up in the __del__() method if you want. E.g, you may decide for neatness / future extensibility (e.g. pluralizing the singleton) that you would like the singleton instance to tidy up after itself when it is no longer being used.
And if you implement a __del__() method expecting to want tidy up to be done during normal program execution, you will probably want to remove the atexit handler too.
If you would like your singleton to be cleaned up when no one is using it anymore, consider storing it on your metaclass using weakref so that you don't retain it yourself.
Related
I have worked with Python for about 4 years and have recently started learning C++. In C++ you create a constructor method for each class I I was wondering if it is correct to think that this is equivalent to the __init__(self) function in Python? Are there any notable differences? Same question for a C++ destructor method vs. Python _exit__(self)
Yes, Python's __init__ is analogous to C++'s constructor. Both are typically where non-static data members are initialized. In both languages, these functions take the in-creation object as the first argument, explicit and by convention named self in Python and implicit and by language named this in C++. In both languages, these functions can return nothing. One notable difference between the languages is that in Python base-class __init__ must be called explicitly from an inherited class __init__ and in C++ it is implicit and automatic. C++ also has ways to declare data member initializers outside the body of the constructor, both by member initializer lists and non-static data member initializers. C++ will also generate a default constructor for you in some circumstances.
Python's __new__ is analogous to C++'s class-level operator new. Both are static class functions which must return a value for the creation to proceed. In C++, that something is a pointer to memory and in Python it is an uninitialized value of the class type being created.
Python's __del__ has no direct analogue in C++. It is an object finalizer, which exist also in other garbage collected languages like Java. It is not called at a lexically predetermined time, but the runtime calls it when it is time to deallocate the object.
__exit__ plays a role similar to C++'s destructor, in that it can provide for deterministic cleanup and a lexically predetermined point. In C++, this tends to be done through the C++ destructor of an RAII type. In Python, the same object can have __enter__ and __exit__ called multiple times. In C++, that would be accomplished with the constructor and destructor of a separate RAII resource holding type. For example, in Python given an instance lock of a mutual exclusion lock type, one can say with lock: to introduce a critical section. In C++, we create an instance of a different type taking the lock as a parameter std::lock_guard g{lock} to accomplish the same thing. The Python __enter__ and __exit__ calls map to the constructor and destructor of the C++ RAII type.
The best you can say is that __init__ and a C++ constructor are called at roughly the same point in the lifetime of a new object, and that __del__ and a C++ destructor are also called near the end of the lifetime of an object. The semantics, however, are markedly different, and the execution model of each language makes further comparison more difficult.
Suffice it to say that __init__ is used to initialize an object after it has been created. __del__ is like a destructor that may be called at some unspecified point in time after the last reference to an object goes away, and __exit__ is more like a callback invoked at the end of a with statement, whether or not the object's reference count reaches zero.
I was wondering if it is correct to think that this is equivalent to
the init(self) function in Python?
No. Just by looking at the structure of the statement you can understand. Indeed, self is a reference to the instance. Therefore, the instance must be constructed before __init__ is called.
See this for more information (__new__ is actually what you're looking for)
Same question for a C++ destructor method vs. Python _exit__(self)
No. __exit__ only exit the Runtime context related to the object. In this case, what you are really looking for is __del__.
See this, which clearly state:
Called when the instance is about to be destroyed. This is also called
a destructor.
I do things mostly in C++, where the destructor method is really meant for destruction of an acquired resource. Recently I started with python (which is really a fun and fantastic), and I came to learn it has GC like java.
Thus, there is no heavy emphasis on object ownership (construction and destruction).
As far as I've learned, the __init__() method makes more sense to me in python than it does for ruby too, but the __del__() method, do we really need to implement this built-in function in our class? Will my class lack something if I miss __del__()? The one scenario I could see __del__() useful is, if I want to log something when destroying an object. Is there anything other than this?
In the Python 3 docs the developers have now made clear that destructor is in fact not the appropriate name for the method __del__.
object.__del__(self)
Called when the instance is about to be destroyed. This is also called a finalizer or (improperly) a destructor.
Note that the OLD Python 3 docs used to suggest that 'destructor' was the proper name:
object.__del__(self)
Called when the instance is about to be destroyed. This is also called a destructor. If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance.
From other answers but also from the Wikipedia:
In a language with an automatic garbage collection mechanism, it would be difficult to deterministically ensure the invocation of a destructor, and hence these languages are generally considered unsuitable for RAII [Resource Acquisition Is Initialization]
So you should almost never be implementing __del__, but it gives you the opportunity to do so in some (rare?) use cases
As the other answers have already pointed out, you probably shouldn't implement __del__ in Python. If you find yourself in the situation thinking you'd really need a destructor (for example if your class wraps a resource that needs to be explicitly closed) then the Pythonic way to go is using context managers.
Is del really a destructor?
No, __del__ method is not a destructor, is just a normal method you can call whenever you want to perform any operation, but it is always called before the garbage collector destroys the object.
Think of it like a clean or last will method.
So uncommon it is that I have learned about it today (and I'm long ago into python).
Memory is deallocated, files closed, ... by the GC. But you could need to perform some task with effects outside of the class.
My use case is about implementing some sort of RAII regarding some temporal directories. I'd like it to be removed no matter what.
Instead of removing it after the processing (which, after some change, was no longer run) I've moved it to the __del__ method, and it works as expected.
This is a very specific case, where we don't really care about when the method is called, as long as it's called before leaving the program. So, use with care.
Say I have some class that manages a database connection. The user is supposed to call close() on instances of this class so that the db connection is terminated cleanly.
Is there any way in python to get this object to call close() if the interpreter is closed or the object is otherwise picked up by the garbage collector?
Edit: This question assumes the user of the object failed to instantiate it within a with block, either because he forgot or isn't concerned about closing connections.
The only way to ensure such a method is called if you don't trust users is using __del__ (docs). From the docs:
Called when the instance is about to be destroyed.
Note that there are lots of issues that make using del tricky. For example, at the moment it is called, the interpreter may be shutting down already - meaning other objects and modules may have been destroyed already. See the notes and warnings for details.
If you really cannot rely on users to be consenting adults, I would prevent them from implicitly avoiding close - don't give them a public open in the first place. Only supply the methods to support with. If anybody explicitly digs into your code to do otherwise, they probably have a good reason for it.
Define __enter__ and __exit__ methods on your class and then use it with the with statement:
with MyClass() as c:
# Do stuff
When the with block ends your __exit__() method will be called automatically.
In contextlib.py, I see the ExitStack class is calling __enter__() method via the type object (type(cm)) instead of direct method calls to the given object (cm).
I wonder why or why not.
e.g.,
does it give better exception traces when an error occurs?
is it just specific to some module author's coding style?
does it have any performance benefits?
does it avoid some artifacts/side-effects with complicated type hierarchies?
First of all, this is what happens when you do with something, it's not just contextlib that looks up special method on the type. Also, it's worth noting that the same happens with other special methods too: e.g. a + b results in type(a).__add__(a, b).
But why does it happen? This is a question that is often fired up on the python-dev and python-ideas mailing lists. And when I say "often", I mean "very often".
The last one were these: Missing Core Feature: + - * / | & do not call getattr and Eliminating special method lookup.
Here are some interesting points:
The current behaviour is by design - special methods are looked up as
slots on the object's class, not as instance attributes. This allows
the interpreter to bypass several steps in the normal instance
attribute lookup process.
(Source)
It is worth noting that the behavior is even more magical than this.
Even when looked up on the class, implicit special method lookup
bypasses __getattr__ and __getattribute__ of the metaclass. So the
special method lookup is not just an ordinary lookup that happens to
start on the class instead of the instance; it is a fully magic lookup
that does not engage the usual attribute-access-customization hooks at
any level.
(Source)
This behavior is also documented on the reference documentation: Special method lookup, which says:
Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
In short, performance is the main concern. But let's take a closer look at this.
What's the difference between type(obj).__enter__() and obj.__enter__()?
When you write obj.attr, type(obj).__getattribute__('attr') gets called. The default implementation of __getattribute__() looks for attr into the instance dictionary (i.e. obj.__dict__) and into the class namespace and, failing that, calls type(obj).__getattr__('attr').
Now, this was a quick explanation and I have omitted some details, however it should give you an idea of how complicated an attribute lookup can be, and how slow it can become. Short circuiting special method lookup surely provides performance improvements, as looking up obj.__enter__() in the "classical" way may be too slow.
I have been thinking about how I write classes in Python. More specifically how the constructor is implemented and how the object should be destroyed. I don't want to rely on CPython's reference counting to do object cleanup. This basically tells me I should use with statements to manage my object life times and that I need an explicit close/dispose method (this method could be called from __exit__ if the object is also a context manager).
class Foo(object):
def __init__(self):
pass
def close(self):
pass
Now, if all my objects behave in this way and all my code uses with statements or explicit calls to close() (or dispose()) I don't realy see the need for me to put any code in __del__. Should we really use __del__ to dispose of our objects?
Short answer : No.
Long answer: Using __del__ is tricky, mainly because it's not guaranteed to be called. That means you can't do things there that absolutely has to be done. This in turn means that __del__ basically only can be used for cleanups that would happen sooner or later anyway, like cleaning up resources that would be cleaned up when the process exits, so it doesn't matter if __del__ doesn't get called. Of course, these are also generally the same things Python will do for you. So that kinda makes __del__ useless.
Also, __del__ gets called when Python garbage collects, and you didn't want to wait for Pythons garbage collecting, which means you can't use __del__ anyway.
So, don't use __del__. Use __enter__/__exit__ instead.
FYI: Here is an example of a non-circular situation where the destructor did not get called:
class A(object):
def __init__(self):
print('Constructing A')
def __del__(self):
print('Destructing A')
class B(object):
a = A()
OK, so it's a class attribute. Evidently that's a special case. But it just goes to show that making sure __del__ gets called isn't straightforward. I'm pretty sure I've seen more non-circular situations where __del__ isn't called.
Not necessarily. You'll encounter problems when you have cyclic references. Eli Bendersky does a good job of explaining this in his blog post:
Safely using destructors in Python
If you are sure you will not go into cyclic references, then using __del__ in that way is OK: as soon as the reference count goes to zero, the CPython VM will call that method and destroy the object.
If you plan to use cyclic references - please think it very thoroughly, and check if weak references may help; in many cases, cyclic references are a first symptom of bad design.
If you have no control on the way your object is going to be used, then using __del__ may not be safe.
If you plan to use JPython or IronPython, __del__ is unreliable at all, because final object destruction will happen at garbage collection, and that's something you cannot control.
In sum, in my opinion, __del__ is usually perfectly safe and good; however, in many situation it could be better to make a step back, and try to look at the problem from a different perspective; a good use of try/except and of with contexts may be a more pythonic solution.