I think the main purpose of __slots__ is to save the memory usage by allowing to specify properties explicitly, instead of using __dict__ allowing dynamic property assignment on the instances. So I somehow understand why __dict__ is removed by default when using __slots__. But why does it meanwhile remove __weakref__ by default?
Reference: https://docs.python.org/3/reference/datamodel.html#slots
I can't read minds, but I suspect the rationale goes like this:
If __weakref__ wasn't disabled by default when using __slots__, providing a way to save the associated memory explicitly would require yet another special opt-out mechanism
More special cases add complexity to the language, and this one would provide no real benefit
Given how infrequently weak references are used at all, it was probably deemed simpler to simpler have it disabled by default, with the option to opt back in.
Diving to implementation details, in a sense, unslotted user-defined classes have precisely two "slots" (one for __dict__, one for __weakref__) over and above the base object header, so having __slots__ say "Replace the default with this explicit list" makes it natural to remove both __dict__ and __weakref__ when __slots__ comes into play.
Related
I've been programming in Python for a long time, but I still can't understand why classes base their attribute lookup on the __dict__ dictionary by default instead of the faster __slots__ tuple.
Wouldn't it make more sense to use the more efficient and less flexible __slots__ method as the default implementation and instead make the more flexible, but slower __dict__ method optional?
Also, if a class uses __slots__ to store its attributes, there's no chance of mistakenly creating new attributes like this:
class Object:
__slots__ = ("name",)
def __init__(self, name):
self.name = name
obj = Object()
# Note the typo here
obj.namr = "Karen"
So, I was wondering if there's a valid reason why Python defaults to accessing instance attributes through __dict__ instead of through __slots__.
Python is designed to be an extremely flexible language, and allows objects to modify themselves in many interesting ways at runtime. Making a change to prevent that kind of flexibility would break a massive amount of other people's code, so for the sake of backwards compatibility I don't think it will happen any time soon (if at all).
As well as this, due to the way Python code is interpreted, it is very difficult to design a system that can look ahead and determine exactly what variables a particular class will use ahead of time, especially given the existence of setattr() and other similar functions, which can modify the state of other objects in unpredictable ways.
In summary, Python is designed to value flexibility over performance, and as such, having __slots__ be an optional technique to speed up parts of your code is a trade-off that you choose to make if you wish to write your code in Python. I can't answer whether this is a worthwhile design decision for you, since it's entirely based on opinion.
If you wish to have a bit more safety to prevent issues such as the one you described, there are tools such as mypy and pylint which can catch that sort of error.
Should you declare private instance variables of a class in the init function? My code works perfectly fine without doing this, but PyCharm tells me to do this when highlighting warnings.
It's generally considered good practice to assign to all instance variables in __init__, even if some of them are lazily given real values and all you can do in __init__ is give them a sentinel that means "No value here yet" (e.g. None). There are two reasons for this:
Maintainer benefit: If you don't follow this guideline, determining the complete set of attributes the class may have involves reading the entire class to look for lazily added attributes. It's a lot easier if maintainers can count on __init__ to provide the complete set of attributes, even if some of them are given real values elsewhere.
(On modern CPython, as an implementation detail) Reduced memory usage: When all instances of a class are given the same set of attributes, in the same order, and the set of attributes is not modified unpredictably after __init__ (it's okay to reassign an attribute, just not to add or delete attributes), CPython uses a key-sharing dictionary to hold the attributes for each instance. The hash table itself that stores the keys ends up shared, tied to the class itself, and only the cheap array containing the values for the instance's attributes ends up costing memory. For the case of a class with a single attribute, this reduces the per-instance __dict__ size from 232 bytes to 104, and the ratio remains similar as the number of attributes grows (the key-sharing __dict__ costs less than half as much memory as a non-key-sharing __dict__).
According to this post:
Python memoising/deferred lookup property decorator
A mnemonic decorator can be used to declare a lazy property in a class. There is even an 'official' package that can be used out of the box:
https://pypi.python.org/pypi/lazy
however, both of these implementation has a severe problem: any memorized values will be attempted to be pickled by python. If these values are unpicklable it will cause the program to break down.
My question is: is there an easy way to implement scala's "#transient lazy val" declaration without too much tinkering? This declaration should remember the property in case of multiple invocation, and drop it once the class/object is serialized.
Unaware of scala implementation details, but the easiest solution comes to my mind, if you're satisfied with other aspects of the 'lazy property' library you've found, would be implementing __getstate__ and __setstate__ object methods, as described in Pickling and unpickling normal class instances
These methods are called by pickle/unpickle handler during object instance (de)serialization.
This way you can have fine-grained control of how/which attributes of your object serialized.
You should read corresponding documentation on another two pickle-related methods as well (take care of __getinitargs__ specifically).
Python deserialized objects initialization differes from common __new__ & __init__ sequence
In contextlib.py, I see the ExitStack class is calling __enter__() method via the type object (type(cm)) instead of direct method calls to the given object (cm).
I wonder why or why not.
e.g.,
does it give better exception traces when an error occurs?
is it just specific to some module author's coding style?
does it have any performance benefits?
does it avoid some artifacts/side-effects with complicated type hierarchies?
First of all, this is what happens when you do with something, it's not just contextlib that looks up special method on the type. Also, it's worth noting that the same happens with other special methods too: e.g. a + b results in type(a).__add__(a, b).
But why does it happen? This is a question that is often fired up on the python-dev and python-ideas mailing lists. And when I say "often", I mean "very often".
The last one were these: Missing Core Feature: + - * / | & do not call getattr and Eliminating special method lookup.
Here are some interesting points:
The current behaviour is by design - special methods are looked up as
slots on the object's class, not as instance attributes. This allows
the interpreter to bypass several steps in the normal instance
attribute lookup process.
(Source)
It is worth noting that the behavior is even more magical than this.
Even when looked up on the class, implicit special method lookup
bypasses __getattr__ and __getattribute__ of the metaclass. So the
special method lookup is not just an ordinary lookup that happens to
start on the class instead of the instance; it is a fully magic lookup
that does not engage the usual attribute-access-customization hooks at
any level.
(Source)
This behavior is also documented on the reference documentation: Special method lookup, which says:
Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
In short, performance is the main concern. But let's take a closer look at this.
What's the difference between type(obj).__enter__() and obj.__enter__()?
When you write obj.attr, type(obj).__getattribute__('attr') gets called. The default implementation of __getattribute__() looks for attr into the instance dictionary (i.e. obj.__dict__) and into the class namespace and, failing that, calls type(obj).__getattr__('attr').
Now, this was a quick explanation and I have omitted some details, however it should give you an idea of how complicated an attribute lookup can be, and how slow it can become. Short circuiting special method lookup surely provides performance improvements, as looking up obj.__enter__() in the "classical" way may be too slow.
I'm having a tricky problem in the game I'm working on. I'm using Pygame to develop it. I happen to be one of those developers who never uses the default__dict__ object variable; I always define __slots__ to clarify the variables an object can have (I have a classmethod that reads the slots to determine the variables needed from a config file).
Anyway, I just realized that this effort isn't working in some of my classes; they still have a __dict__ variable and can have arbitrary attributes assigned to, even though they explicitly define their __slots__. I think this is because they are inheriting from pygame.sprite.Sprite, which has a __dict__. If this is the case, how do I suppress creation of this dict? (I though explicitly defining __slots__ was supposed to) Or could I be mistaken about the cause? Thanks for any insight; it's hard to find information about this particular problem via searches.
The only way to suppress arbitrary attributes and the __dict__ container of them, is to use __slots__ as you are and inherit from a class that does the same. A subclass of a class that has a __dict__ will always have a __dict__. The only way around it is to not inherit from this class (but, for example, use composition instead.)