Develop Reference

Modifying class __dict__ when shadowed by a property

I am attempting to modify a value in a class __dict__ directly using something like X.__dict__['x'] += 1. It is impossible to do the modification like that because a class __dict__ is actually a mappingproxy object that does not allow direct modification of values. The reason for attempting direct modification or equivalent is that I am trying to hide the class attribute behind a property defined on the metaclass with the same name. Here is an example:
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = 0
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x']
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
__class__.__dict__['x'] += 1
This is example shows a scheme for creating an auto-incremented ID for each instance of Class. The line __class__.__dict__['x'] += 1 can not be replaced by setattr(__class__, 'x', __class__.x + 1) because x is a property with no setter in Meta. It would just change a TypeError from mappingproxy into an AttributeError from property.
I have tried messing with __prepare__, but that has no effect. The implementation in type already returns a mutable dict for the namespace. The immutable mappingproxy seems to get set in type.__new__, which I don't know how to avoid.
I have also attempted to rebind the entire __dict__ reference to a mutable version, but that failed as well: https://ideone.com/w3HqNf, implying that perhaps the mappingproxy is not created in type.__new__.
How can I modify a class dict value directly, even when shadowed by a metaclass property? While it may be effectively impossible, setattr is able to do it somehow, so I would expect that there is a solution.
My main requirement is to have a class attribute that appears to be read only and does not use additional names anywhere. I am not absolutely hung up on the idea of using a metaclass property with an eponymous class dict entry, but that is usually how I hide read only values in regular instances.
EDIT
I finally figured out where the class __dict__ becomes immutable. It is described in the last paragraph of the "Creating the Class Object" section of the Data Model reference:
When a new class is created by type.__new__, the object provided as the namespace parameter is copied to a new ordered mapping and the original object is discarded. The new copy is wrapped in a read-only proxy, which becomes the __dict__ attribute of the class object.

Probably the best way: just pick another name. Call the property x and the dict key '_x', so you can access it the normal way.
Alternative way: add another layer of indirection:
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = [0]
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x'][0]
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
__class__.__dict__['x'][0] += 1
That way you don't have to modify the actual entry in the class dict.
Super-hacky way that might outright segfault your Python: access the underlying dict through the gc module.
import gc
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = 0
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x']
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
gc.get_referents(__class__.__dict__)[0]['x'] += 1
This bypasses critical work type.__setattr__ does to maintain internal invariants, particularly in things like CPython's type attribute cache. It is a terrible idea, and I'm only mentioning it so I can put this warning here, because if someone else comes up with it, they might not know that messing with the underlying dict is legitimately dangerous.
It is very easy to end up with dangling references doing this, and I have segfaulted Python quite a few times experimenting with this. Here's one simple case that crashed on Ideone:
import gc
class Foo(object):
x = []
Foo().x
gc.get_referents(Foo.__dict__)[0]['x'] = []
print(Foo().x)
Output:
*** Error in `python3': double free or corruption (fasttop): 0x000055d69f59b110 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x2b32d5977bcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x2b32d597df96]
/lib/x86_64-linux-gnu/libc.so.6(+0x7778e)[0x2b32d597e78e]
python3(+0x2011f5)[0x55d69f02d1f5]
python3(+0x6be7a)[0x55d69ee97e7a]
python3(PyCFunction_Call+0xd1)[0x55d69efec761]
python3(PyObject_Call+0x47)[0x55d69f035647]
... [it continues like that for a while]
And here's a case with wrong results and no noisy error message to alert you to the fact that something has gone wrong:
import gc
class Foo(object):
x = 'foo'
print(Foo().x)
gc.get_referents(Foo.__dict__)[0]['x'] = 'bar'
print(Foo().x)
Output:
foo
foo
I make absolutely no guarantees as to any safe way to use this, and even if things happen to work out on one Python version, they may not work on future versions. It can be fun to fiddle with, but it's not something to actually use. Seriously, don't do it. Do you want to explain to your boss that your website went down or your published data analysis will need to be retracted because you took this bad idea and used it?

This probably counts as an "additional name" you don't want, but I've implemented this using a dictionary in the metaclass where the keys are the classes. The __next__ method on the metaclass makes the class itself iterable, such that you can just do next() to get the next ID. The dunder method also keeps the method from being available through the instances. The dictionary storing the next id has a name starting with a double underscore, so it's not easily discoverable from any of the classes that use it. The incrementing ID functionality is thus entirely contained in the metaclass.
I tucked the assignment of the id into a __new__ method on a base class, so you don't have to worry about it in __init__. This also allows you to del Meta so all the machinery is a little harder to get to.
class Meta(type):
__ids = {}
#property
def id(cls):
return __class__.__ids.setdefault(cls, 0)
def __next__(cls):
id = __class__.__ids.setdefault(cls, 0)
__class__.__ids[cls] += 1
return id
class Base(metaclass=Meta):
def __new__(cls, *args, **kwargs):
self = object.__new__(cls)
self.id = next(cls)
return self
del Meta
class Class(Base):
pass
class Brass(Base):
pass
c0 = Class()
c1 = Class()
b0 = Brass()
b1 = Brass()
assert (b0.id, b1.id, c0.id, c1.id) == (0, 1, 0, 1)
assert (Class.id, Brass.id) == (2, 2)
assert not hasattr(Class, "__ids")
assert not hasattr(Brass, "__ids")
Note that I've used the same name for the attribute on both the class and the object. That way Class.id is the number of instances you've created, while c1.id is the ID of that specific instance.

My main requirement is to have a class attribute that appears to be read only and does not use additional names anywhere. I am not absolutely hung up on the idea of using a metaclass property with an eponymous class dict entry, but that is usually how I hide read only values in regular instances.
What you are asking for is a contradiction: If your example worked, then __class__.__dict__['x'] would be an "additional name" for the attribute. So clearly we need a more specific definition of "additional name." But to come up with that definition, we need to know what you are trying to accomplish (NB: The following goals are not mutually exclusive, so you may want to do all of these things):
You want to make the value completely untouchable, except within the Class.__init__() method (and the same method of any subclasses): This is unPythonic and quite impossible. If __init__() can modify the value, then so can anyone else. You might be able to accomplish something like this if the modifying code lives in Class.__new__(), which the metaclass dynamically creates in Meta.__new__(), but that's extremely ugly and hard to understand.
You want the code that manipulates the value to be "nicely encapsulated": Write a method in the metaclass that increments the private value (or does whatever other modification you need), and provide a read-only metaclass property that accesses it under the public name.
You are concerned about a subclass accidentally clashing names with the private name: Prefix the private name with a double underscore to invoke automatic name mangling. While this is usually seen as a bit unPythonic, it is appropriate for cases where name collisions may be less obvious to subclass authors, such as the internal names of a metaclass colliding with the internal names of a regular class instantiated from it.

Use cases for property vs. descriptor vs. __getattribute__

The question refers to which one is preferable to be used in which use case, not about the technical background.
In python, you can control the access of attributes via a property, a descriptor, or magic methods. Which one is most pythonic in which use case? All of them seem to have the same effect (see the examples below).
I am looking for an answer like:
Property: Should be used in case of …
Descriptor: In the case of … it should be used instead of a property.
Magic method: Only use if ….
Example
A use case would be an attribute that might not be able to be set in the __init__ method, for example because the object is not present in the database yet, but at a later time. Each time the attribute is accessed, it should be tried to be set and returned.
As an example that works with Copy&Paste in the Python shell, there is a class that wants to present its attribute only the second time it is asked for it. So, which one is the best way, or are there different situations one of them is preferable? Here are the three ways to implement it:
With Property::
class ContactBook(object):
intents = 0
def __init__(self):
self.__first_person = None
def get_first_person(self):
ContactBook.intents += 1
if self.__first_person is None:
if ContactBook.intents > 1:
value = 'Mr. First'
self.__first_person = value
else:
return None
return self.__first_person
def set_first_person(self, value):
self.__first_person = value
first_person = property(get_first_person, set_first_person)
With __getattribute__::
class ContactBook(object):
intents = 0
def __init__(self):
self.first_person = None
def __getattribute__(self, name):
if name == 'first_person' \
and object.__getattribute__(self, name) is None:
ContactBook.intents += 1
if ContactBook.intents > 1:
value = 'Mr. First'
self.first_person = value
else:
value = None
else:
value = object.__getattribute__(self, name)
return value
Descriptor::
class FirstPerson(object):
def __init__(self, value=None):
self.value = None
def __get__(self, instance, owner):
if self.value is None:
ContactBook.intents += 1
if ContactBook.intents > 1:
self.value = 'Mr. First'
else:
return None
return self.value
class ContactBook(object):
intents = 0
first_person = FirstPerson()
Each one of it has this behavior::
book = ContactBook()
print(book.first_person)
# >>None
print(book.first_person)
# >>Mr. First

Basically, use the simplest one you can. Roughly speaking, the order of complexity/heavy-duty-ness goes: regular attribute, property, __getattr__, __getattribute__/descriptor. (__getattribute__ and custom descriptors are both things you probably won't need to do very often.) This leads to some simple rules of thumb:
Don't use a property if a normal attribute will work.
Don't write your own descriptor if a property will work.
Don't use __getattr__ if a property will work.
Don't use __getattribute__ if __getattr__ will work.
Stated a bit more specifically: use a property to customize handling of one or a small set of attributes; use __getattr__ to customize handling of all attributes, or all except a small set; use __getattribute__ if you were hoping to use __getattr__ but it doesn't quite work; write your own descriptor class if you are doing something very complicated.
You use a property when you have one or a small set of attributes whose getting/setting you want to hook into. That is, you want things like obj.prop and obj.prop = 2 to secretly call a function that you write to customize what happens.
You would use __getattr__ when you want to do that for so many attributes that you don't actually want to define them all individually, but rather want to customize the whole attribute-access process as a whole. In other words, instead of hooking into obj.prop1, and obj.prop2, etc., you have so many that you want to be able to hook into obj.<anything>, and handle that in general.
However, __getattr__ still won't let you override what happens for attributes that really do exist, it just lets you hook in with a blanket handling for any use of attributes that would otherwise raise an AttributeError. Using __getattribute__ lets you hook in to handle everything, even normal attributes that would have worked without messing with __getattribute__. Because of this, using __getattribute__ has the potential to break fairly basic behavior, so you should only use it if you considered using __getattr__ and it wasn't enough. It also can have a noticeable performance impact. You might for instance need to use __getattribute__ if you're wrapping a class that defines some attributes, and you want to be able to wrap those attributes in a custom way, so that they work as usual in some situations but get custom behavior in other situations.
Finally, I would say writing your own descriptor is a fairly advanced task. property is a descriptor, and for probably 95% of cases it's the only one you'll need. A good simple example of why you might write your own descriptor is given here: basically, you might do it if you would otherwise have to write several propertys with similar behavior; a descriptor lets you factor out the common behavior to avoid code repetition. Custom descriptors are used, for instance, to drive systems like like Django and SQLAlchemy. If you find yourself writing something at that level of complexity you might need to write a custom descriptor.
In your example, property would be the best choice. It is usually (not always) a red flag if you're doing if name == 'somespecificname' inside __getattribute__. If you only need to specially handle one specific name, you can probably do it without stooping to the level of __getattribute__. Likewise, it doesn't make sense to write your own descriptor if all you write for its __get__ is something you could have written in a property's getter method.

__getattribute__ is the hook that enables property (and other descriptors) to work in the first place and is called for all attribute access on an object. Consider it a lower-level API when a property or even a custom descriptor is not enough for your needs. __getattr__ is called by __getattribute__ when no attribute has been located through other means, as a fallback.
Use property for dynamic attributes with a fixed name, __getattr__ for attributes of a more dynamic nature (e.g. a series of attributes that map to values in an algorithmic manner).
Descriptors are used when you need to bind arbitrary objects to an instance. When you need to replace method objects with something more advanced for example; a recent example is a class-based decorator wrapping methods that needed to support additional attributes and methods on the method object. Generally, when you are still thinking in terms of scalar attributes, you don't need descriptors.

"public" or "private" attribute in Python ? What is the best way?

In Python, I have the following example class :
class Foo:
self._attr = 0
#property
def attr(self):
return self._attr
#attr.setter
def attr(self, value):
self._attr = value
#attr.deleter
def attr(self):
del self._attr
As you can see, I have a simple "private" attribute "_attr" and a property to access it. There is a lot of codes to declare a simple private attribute and I think that it's not respecting the "KISS" philosophy to declare all attributes like that.
So, why not declare all my attributes as public attributes if I don't need a particular getter/setter/deleter ?
My answer will be :
Because the principle of encapsulation (OOP) says otherwise!
What is the best way ?

Typically, Python code strives to adhere to the Uniform Access Principle. Specifically, the accepted approach is:
Expose your instance variables directly, allowing, for instance, foo.x = 0, not foo.set_x(0)
If you need to wrap the accesses inside methods, for whatever reason, use #property, which preserves the access semantics. That is, foo.x = 0 now invokes foo.set_x(0).
The main advantage to this approach is that the caller gets to do this:
foo.x += 1
even though the code might really be doing:
foo.set_x(foo.get_x() + 1)
The first statement is infinitely more readable. Yet, with properties, you can add (at the beginning, or later on) the access control you get with the second approach.
Note, too, that instance variables starting with a single underscore are conventionally private. That is, the underscore signals to other developers that you consider the value to be private, and they shouldn't mess with it directly; however, nothing in the language prevents them from messing with it directly.
If you use a double leading underscore (e.g., __x), Python does a little obfuscation of the name. The variable is still accessible from outside the class, via its obfuscated name, however. It's not truly private. It's just kind of ... more opaque. And there are valid arguments against using the double underscore; for one thing, it can make debugging more difficult.

The "dunder" (double underscore, __) prefix prevents access to attribute, except through accessors.
class Foo():
def __init__(self):
self.__attr = 0
#property
def attr(self):
return self.__attr
#attr.setter
def attr(self, value):
self.__attr = value
#attr.deleter
def attr(self):
del self.__attr
Some examples:
>>> f = Foo()
>>> f.__attr # Not directly accessible.
Traceback (most recent call last):
File "<input>", line 1, in <module>
AttributeError: 'Foo' object has no attribute '__attr'
>>> '__attr' in f.__dir__() # Not listed by __dir__()
False
>>> f.__getattribute__('__attr') # Not listed by __getattribute__()
Traceback (most recent call last):
File "<input>", line 1, in <module>
AttributeError: 'Foo' object has no attribute '__attr'
>>> f.attr # Accessible by implemented getter.
0
>>> f.attr = 'Presto' # Can be set by implemented setter.
>>> f.attr
'Presto'
>>> f.__attr = 'Tricky?' # Can we set it explicitly?
>>> f.attr # No. By doing that we have created a
'Presto' # new but unrelated attribute, same name.
However, you can access this type of attribute through name mangling (_classname__attribute), which Python does in the background:
>>> f._Foo__attr
0
>>> f.__getattribute__('_Foo__attr')
0

Quite simply, the OOP principles are wrong. Why this is is a long discussion which leads to flamewars and is probably off topic for this site. :-)
In Python there is not private attributes, you can't protect them, and this is never a real problem. So don't. Easy! :)
Then comes the question: Should you have a leading underscore or not. And in the example you have here you should definitely not. A leading underscore in Python is a convention to show that something is internal, and not a part of the API, and that you should use it on your own risk. This is obviously not the case here, but it's a common and useful convention.

Python doesn't have public OR private attributes. All attributes are accessible to all code.
self.attr = 0 #Done
Your method isn't in any way making _attr private, it's just a bit of obfuscation.

See this link:https://docs.python.org/2/tutorial/classes.html
9.6. Private Variables and Class-local References
“Private” instance variables that cannot be accessed except from inside an object don’t exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice.
Since there is a valid use-case for class-private members (namely to avoid name clashes of names with names defined by subclasses), there is limited support for such a mechanism, called name mangling. Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam, where classname is the current class name with leading underscore(s) stripped. This mangling is done without regard to the syntactic position of the identifier, as long as it occurs within the definition of a class.

As others have said, private attributes in Python are merely a convention. The use of property syntax should be used for special processing when attributes are bound, modified or deleted. The beauty of Python is that you can start off by just using normal attribute binding, e.g., self.attr = 0 and if at some later date you decide you want to restrict the value of attr to say 0 <= attr <=100, you can make attr a property and define a method to make sure this condition is true without ever having to change any user code.

To make an attribute private, you just have to do self.__attr
class Foo:
self.__attr = 0
#property
def attr(self):
return self._attr
#attr.setter
def attr(self, value):
self._attr = value
#attr.deleter
def attr(self):
del self._attr

In Python, unless you need special behavior out of an attribute, there's no need to hide it behind accessor methods. If an attribute is for internal use only, prepend it with an underscore.

The nice thing about properties is that they given you a really cool interface to work with. Sometimes it's handy to derive a property based on some other (ie. BMI is defined by weight and height). The user of the interface doesn't have to know this of course.
I prefer this way over having explicit getters and setters like in Java ie. Way nicer. :)

Dynamically adding #property in python

I know that I can dynamically add an instance method to an object by doing something like:
import types
def my_method(self):
# logic of method
# ...
# instance is some instance of some class
instance.my_method = types.MethodType(my_method, instance)
Later on I can call instance.my_method() and self will be bound correctly and everything works.
Now, my question: how to do the exact same thing to obtain the behavior that decorating the new method with #property would give?
I would guess something like:
instance.my_method = types.MethodType(my_method, instance)
instance.my_method = property(instance.my_method)
But, doing that instance.my_method returns a property object.

The property descriptor objects needs to live in the class, not in the instance, to have the effect you desire. If you don't want to alter the existing class in order to avoid altering the behavior of other instances, you'll need to make a "per-instance class", e.g.:
def addprop(inst, name, method):
cls = type(inst)
if not hasattr(cls, '__perinstance'):
cls = type(cls.__name__, (cls,), {})
cls.__perinstance = True
inst.__class__ = cls
setattr(cls, name, property(method))
I'm marking these special "per-instance" classes with an attribute to avoid needlessly making multiple ones if you're doing several addprop calls on the same instance.
Note that, like for other uses of property, you need the class in play to be new-style (typically obtained by inheriting directly or indirectly from object), not the ancient legacy style (dropped in Python 3) that's assigned by default to a class without bases.

Since this question isn't asking about only adding to a spesific instance,
the following method can be used to add a property to the class, this will expose the properties to all instances of the class YMMV.
cls = type(my_instance)
cls.my_prop = property(lambda self: "hello world")
print(my_instance.my_prop)
# >>> hello world
Note: Adding another answer because I think #Alex Martelli, while correct, is achieving the desired result by creating a new class that holds the property, this answer is intended to be more direct/straightforward without abstracting whats going on into its own method.

Constant instance variables?

I use #property to ensure that changes to an objects instance variables are wrapped by methods where I need to.
What about when an instance has an variable that logically should not be changed? Eg, if I'm making a class for a Process, each Process instance should have a PID attribute that will frequently be accessed but should not be changed.
What's the most Pythonic way to handle someone attempting to modify that instance variable?
Simply trust the user not to try and change
something they shouldn't?
Use property but raise an
exception if the instance variable is
changed?
Something else?

Prepend name of the variable with __, and create read-only property, Python will take care of exceptions, and variable itself will be protected from accidental overwrite.
class foo(object):
def __init__(self, bar):
self.__bar = bar
#property
def bar(self):
return self.__bar
f = foo('bar')
f.bar # => bar
f.bar = 'baz' # AttributeError; would have to use f._foo__bar

Simply trusting the user is not necessarily a bad thing; if you are just writing a quick Python program to be used once and thrown away, you might very well just trust that the user not alter the pid field.
IMHO the most Pythonic way to enforce the read-only field is to use a property that raises an exception on an attempt to set the field.
So, IMHO you have good instincts about this stuff.

Maybe you can override __setattr__? In the line of,
def __setattr__(self, name, value):
if self.__dict__.has_key(name):
raise TypeError, 'value is read only'
self.__dict__[name] = value

Simply use a property and a hidden attribute (prefixed with one underscore).
Simple properties are read-only!
>>> class Test (object):
... #property
... def bar(self):
... return self._bar
...
>>> t = Test()
>>> t._bar = 2
>>> t.bar
2
>>> t.bar = 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't set attribute
Hiding with double underscore is not used to hide the implementation, but to make sure you don't collide with a subclass' attributes; consider a mixin for example, it has to be very careful!
If you just want to hide the attribute, use a single underscore. And as you see there is no extra magic to add -- if you don't define a set function, your property is just as read-only as the return value of a method.