Why are properties class attributes in Python? - python

I'm reading Fluent Python chapter 19 > A Proper Look at Properties, and I'm confused about the following words:
Properties are always class attributes, but they actually manage attribute access in the instances of the class.
The example code is:
class LineItem:
def __init__(self, description, weight, price):
self.description = description
self.weight = weight # <1>
self.price = price
def subtotal(self):
return self.weight * self.price
#property # <2>
def weight(self): # <3>
return self.__weight # <4>
#weight.setter # <5>
def weight(self, value):
if value > 0:
self.__weight = value # <6>
else:
raise ValueError('value must be > 0') # <7>
From my previous experiences, class attributes are belong to the class itself and shared by all the instances. But here, weight, the property, is an instance method and the value returned by it is different between instances. How is it eligible to be a class attribute? Doesn't it that all the class attributes should be the same for any instances?
I think I misunderstand something, so I hope to get a correct explanation. Thanks!

A distinction is made because when you define a #property on a class, that property object becomes an attribute on the class. Whereas when you define attributes against an instance of your class (in your __init__ method), that attribute only exists against that object. This might be confusing if you do:
>>> dir(LineItem)
['__class__', ..., '__weakref__', 'subtotal', 'weight']
>>> item = LineItem("an item", 3, 1.12)
>>> dir(item)
['__class__', ..., '__weakref__', 'description', 'price', 'subtotal', 'weight']
Notice how both subtotal and weight exist as attributes on your class.
I think it's also worth noting that when you define a class, code under that class is executed. This includes defining variables (which then become class attributes), defining functions, and anything else.
>>> import requests
>>> class KindOfJustANamespace:
... text = requests.get("https://example.com").text
... while True:
... break
... for x in range(2):
... print(x)
...
0
1
>>> KindOfJustANamespace.text
'<!doctype html>\n<html>\n<head>\n <title>Example Domain...'
A #decorator is just "syntactic sugar". Meaning #property over a function if the same as function = property(function). This applies to functions defined inside a class as well, but now the function is part of the class's namespace.
class TestClass:
#property
def foo(self):
return "foo"
# ^ is the same as:
def bar(self):
return "bar"
bar = property(bar)
A good explanation of property in Python can be found here: https://stackoverflow.com/a/17330273/7220776

From my previous experiences, class attributes are belong to the class itself and shared by all the instances.
That's right.
But here, weight, the property, is an instance method
No, it's a property object. When you do:
#decorator
def func():
return 42
it's actually syntactic sugar for
def func():
return 42
func = decorator(func)
IOW the def statement is executed, the function object created, but instead of beeing bound to it's name, it's passed to the decorator callable, and the name is bound to whatever decorator() returned.
In this case the decorator is the property class itself, so the weight attribute is a property instance. You can check this out by yourself by inspecting LineItem.weight (which will return the property object itself).
and the value returned by it is different between instances.
Well yes of course, how is this surprising ? LineItem.subtotal is a class attribute also (like all methods), yet it returns values from the instance it's called on (which is passed to the function as the self param).
How is it eligible to be a class attribute? Doesn't it that all the class attributes should be the same for any instances?
The class attributes ARE the same for all instances of a class, yes. There's only one single subtotal function for all instances of LineItem.
A property is mainly a shortcut to make a function (or a pair of functions if you specify a setter) look like it's a plain attribute, so when you type mylinitem.weight, what is really executed is LineItem.weight.fget(mylineitem), where fget is the getter function you decorated with #property. The mechanism behind this is known as the "descriptor protocol", which is also used to turn mylineitem.subtotal() into LineItem.subtotal(mylineitem) (python functions implement the descriptor protocol to return "method" objects, which are themselves wrappers around the function and the current instance and insert the instance as first argument to the function call).
So it's not suprising that properties are class attributes - you only need one property instance to "serve" all instances of the class -, and moreover, properties - like all descriptors FWIW - MUST actually be class attributes to work as expected, since the descriptor protocol is only invoked on class attributes (there's no use case for a "per instance" computed attribute since the function in charge of the "computation" will get the instance as parameter).

I finally understand the descriptor and property concept through Simeon Franklin's excellent presentation, the following contents can be seen as a summary on his lecture notes. Thanks to him!
To understand properties, you first need to understand descriptors, because a property is implemented by a descriptor and python's decorator syntactic sugar. Don't worry, it's not that difficult.
What is a descriptor:
a descriptor is any object that implements at least one of methods named __get__(), __set__(), and __delete__().
Descriptor can be divided into two categories:
A data descriptor implements both __get__() and __set__().
A non-data descriptor implements only __get__().
According to python's HowTo:
a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol.
Then what is the descriptor protocol? Basically speaking, it's just says that when Python interpreter comes across an attribute access like obj.attr,it will search in some order to resolve this .attr , and if this attr is a descriptor attribute, then this descriptor will take some precedence in this specific order and this attribute access will be translated into a method call on this descriptor according to the descriptor protocol, possibly shadowing a namesake instance attribute or class attribute. More concretely, if attr is a data descriptor, then obj.attr will be translated into the calling result of this descriptor's __get__ method; if attr is not a data descriptor and is an instance attribute, this instance attribute will be matched; if attr is not in above, and it is a non-data descriptor, we get the calling result of this non-data descriptor's __get__ method. Full rules on attribute resolution can be found here .
Now let's talk about property. If you have looked at Python' descriptor HowTo, you can find a pure Python version implementation of property:
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
Apparently,property is a data descriptor!
#property just uses python's decorator syntactic sugar.
#property
def attr(self):
pass
is equivalent to:
attr = property(attr)
So, attr is no longer an instance method as I posted in thie question, but is translated into a class attribute by the decorator syntactic sugar as the author said. It's a descriptor object attribute.
How is it eligible to be a class attribute?
OK, we solved it now.
Then:
Doesn't it that all the class attributes should be the same for any instances?
No!
I steal an example from Simeon Franklin's excellent presentation .
>>> class MyDescriptor(object):
... def __get__(self, obj, type):
... print self, obj, type
... def __set__(self, obj, val):
... print "Got %s" % val
...
>>> class MyClass(object):
... x = MyDescriptor() # Attached at class definition time!
...
>>> obj = MyClass()
>>> obj.x # a function call is hiding here
<...MyDescriptor object ...> <....MyClass object ...> <class '__main__.MyClass'>
>>>
>>> MyClass.x # and here!
<...MyDescriptor object ...> None <class '__main__.MyClass'>
>>>
>>> obj.x = 4 # and here
Got 4
Pay attention to obj.x and its output. The second element in its output is <....MyClass object ...> . It's the specific instance obj . Shortly speaking, because this attribute access has been translated into a __get__ method call, and this __get__ method get the specific instance argument as its method signature descr.__get__(self, obj, type=None) demands, it can return different values according to which instance it is been called by.
Note: my English explanation maybe not clear enough, so I highly recommend you to look at Simeon Franklin's notes and Python's descriptor HowTo.

You didn't misunderstand. Don't worry, just read on. It will become clear in the next chapter.
The same book explains in chapter 20 that they can be a class attributes because of the descriptor protocol. The documentation explains how properties are implemented as descriptors.
As you see from the example, properties are really class attributes (methods). When called, they get a reference to the instance, and writes/reads to its underlying __dict__.

I think the example is wrong, the init shoul look like this:
def __init__(self, description, weight, price):
self.description = description
self.__weight = weight # <1>
self.__price = price
self.__weight and self.__price are the internal attributes hidden in the class by the properties

Related

Is there a recommended way of ensuring immutability

I am observing following behavior since python passes object by reference?
class Person(object):
pass
person = Person()
person.name = 'UI'
def test(person):
person.name = 'Test'
test(person)
print(person.name)
>>> Test
I found copy.deepcopy() to deepcopy object to prevent modifying the passed object. Are there any other recommendations ?
import copy
class Person(object):
pass
person = Person()
person.name = 'UI'
def test(person):
person_copy = copy.deepcopy(person)
person_copy.name = 'Test'
test(person)
print(person.name)
>>> UI
I am observing following behavior since python passes object by reference?
Not really. it's a subtle question. you can look at python - How do I pass a variable by reference? - Stack Overflow
Personally, I don't fully agree with the accepted answer and recommend you google call by sharing. Then, you can make your own decision on this subtle question.
I found copy.deepcopy() to deepcopy object to prevent modifying the passed object. Are there any other recommendations ?
As far as I know, there no other better way, if you don't use third package.
You can use the __setattr__ magic method to implement a base class that allows you to "freeze" an object after you're done with it.
This is not bullet-proof; you can still access __dict__ to mutate the object, and you can also unfreeze the object by unsetting _frozen, and if the attribute's value itself is mutable, this doesn't help much (x.things.append('x') would work for a list of things).
class Freezable:
def freeze(self):
self._frozen = True
def __setattr__(self, key, value):
if getattr(self, "_frozen", False):
raise RuntimeError("%r is frozen" % self)
super().__setattr__(key, value)
class Person(Freezable):
def __init__(self, name):
self.name = name
p = Person("x")
print(p.name)
p.name = "y"
print(p.name)
p.freeze()
p.name = "q"
outputs
x
y
Traceback (most recent call last):
File "freezable.py", line 21, in <module>
p.name = 'q'
RuntimeError: <__main__.Person object at 0x10f82f3c8> is frozen
There are no really 100% watertight way, but you can make it difficult to inadvertently mutate an object that you want to keep frozen; the recommended way for most people is probably to use a frozen DataClass, or a frozen attrs class
In his talk on DataClasses (2018), #RaymonHettinger mentions three approaches: one way, is with a metaclass, another, like in the fractions module is to give attributes a read only property; the DataClass module extends __setattr__ and __delattr__, and overrides __hash__:
-> use a metaclass.
Good resources include #DavidBeasley books and talks at python.
-> give attributes a read only property
class SimpleFrozenObject:
def __init__(self, x=0):
self._x = x
#property
def x(self):
return self._x
f = SimpleFrozenObject()
f.x = 2 # raises AttributeError: can't set attribute
-> extend __setattr__ and __delattr__, and override `hash
class FrozenObject:
...
def __setattr__(self, name, value):
if type(self) is cls or name in (tuple of attributes to freeze,):
raise FrozenInstanceError(f'cannot assign to field {name}')
super(cls, self).__setattr__(name, value)
def __delattr__(self, name):
if type(self) is cls or name in (tuple of attributes to freeze,):
raise FrozenInstanceError(f'cannot delete field {name}')
super(cls, self).__delattr__(name, value)
def __hash__(self):
return hash((tuple of attributes to freeze,))
...
The library attrs also offers options to create immutable objects.

Remove a property (getter/setter) from an attribute in subclass

This may have been answered somewhere else, but I was wondering if there was any way to remove an attribute/method decorated with #property in a subclass.
Example:
from datetime import datetime
class A():
def __init__(self, num):
self._num = num
#property
def id(self):
return self._num * datetime.now().timestamp()
class B(A):
def __init__(self, id, num):
super().__init__(num)
self.id = id
The above code does not run if you attempt to create an instance of class B. AttributeError: can't set attribute
The base class uses a property because it needs to evaluate its ID on the fly, while my sub class is able to know its ID when it is created. The id attribute is accessed OFTEN, and I am seeing a significant performance hit because I have to use a property to serve this attribute, instead of just accessing it directly. (From what I have read, properties increase time-to-access by 5x). My application is currently spending around 10% of runtime getting this property.
Is there any way I can short-circuit the property in a sub class?
I'm going to go through several possibilities here. Some of them do what you literally asked. Some of them don't, but they may be better options anyway.
First, your example base class changes the value of obj.id on every access due to the passage of time. That's really bizarre and doesn't seem like a useful concept of "ID". If your real use case has a stable obj.id return value, then you can cache it to avoid the expense of recomputation:
def __init__(self):
...
self._id = None
#property
def id(self):
if self._id is not None:
return self._id
retval = self._id = expensive_computation()
return retval
This may mitigate the expense of the property. If you need more mitigation, look for places where you access id repeatedly, and instead, access it once and save it in a variable. Local variable lookup outperforms attribute access no matter how the attribute is implemented. (Of course, if you actually do have weird time-variant IDs, then this sort of refactoring may not be valid.)
Second, you can't override a property with a "regular" attribute, but you can create your own version of property that can be overridden this way. Your property blocks attribute setting, and takes priority over "regular" attributes even if you force an entry into the instance __dict__, because property has a __set__ method (even if you don't write a setter). Writing your own descriptor without a __set__ would allow overriding. You could do it with a generic LowPriorityProperty:
class LowPriorityProperty(object):
"""
Like #property, but no __set__ or __delete__, and does not take priority
over the instance __dict__.
"""
def __init__(self, fget):
self.fget = fget
def __get__(self, instance, owner=None):
if instance is None:
return self
return self.fget(instance)
class Foo(object):
...
#LowPriorityProperty
def id(self):
...
class Bar(Foo):
def __init__(self):
super(Bar, self).__init__()
self.id = whatever
...
Or with a role-specific descriptor class:
class IDDescriptor(object):
def __get__(self, instance, owner=None):
if instance is None:
return self
# Remember, self is the descriptor. instance is the object you're
# trying to compute the id attribute of.
return whatever(instance)
class Foo(object):
id = IDDescriptor()
...
class Bar(Foo):
def __init__(self):
super(Bar, self).__init__()
self.id = whatever
...
The role-specific descriptor performs better than the generic LowPriorityProperty, but both perform worse than property due to implementing more logic in Python instead of C.
Finally, you can't override a property with a "regular" attribute, but you can override it with another descriptor, such as another property, or such as the descriptors created for __slots__. If you're really, really pressed for performance, __slots__ is probably more performant than any descriptor you could implement manually, but the interaction between __slots__ and the property is weird and obscure and you'll probably want to leave a comment explaining what you're doing.
class Foo(object):
#property
def id(self):
...
class Bar(Foo):
__slots__ = ('id',)
def __init__(self):
super(Bar, self).__init__()
self.id = whatever
...
add a class C as common ancestor, without id. inherit A and B from it and implement id there as needed. Python wont care that id doesn’t exist on C.
refactor non-id code/attributes from A to C.
Suitability depends on whether OP controls class hierarchy and instantiation mechanisms.
I also found a workaround to get it working as is:
from datetime import datetime
class A():
def __init__(self, num):
self._num = num
#property
def id(self):
return self._num * datetime.now().timestamp()
class B(A):
#this fixes the problem
id = None
def __init__(self, id, num):
super().__init__(num)
self.id = id
b = B("id", 3)
print(vars(b))
This will output:
{'_num': 3, 'id': 'id'}
The trick is id = None on class B. Basically, Python's attribute/method lookup mechanism will stop at the first class with id as an attribute in the MRO. With id = None on class B, the lookup stops there and it never gets as far as that pesky #property on A.
If I comment it back out, as per the OP:
self.id = id
AttributeError: can't set attribute

How to make a class attribute exclusive to the super class

I have a master class for a planet:
class Planet:
def __init__(self,name):
self.name = name
(...)
def destroy(self):
(...)
I also have a few classes that inherit from Planet and I want to make one of them unable to be destroyed (not to inherit the destroy function)
Example:
class Undestroyable(Planet):
def __init__(self,name):
super().__init__(name)
(...)
#Now it shouldn't have the destroy(self) function
So when this is run,
Undestroyable('This Planet').destroy()
it should produce an error like:
AttributeError: Undestroyable has no attribute 'destroy'
The mixin approach in other answers is nice, and probably better for most cases. But nevertheless, it spoils part of the fun - maybe obliging you to have separate planet-hierarchies - like having to live with two abstract classes each ancestor of "destroyable" and "non-destroyable".
First approach: descriptor decorator
But Python has a powerful mechanism, called the "descriptor protocol", which is used to retrieve any attribute from a class or instance - it is even used to ordinarily retrieve methods from instances - so, it is possible to customize the method retrieval in a way it checks if it "should belong" to that class, and raise attribute error otherwise.
The descriptor protocol mandates that whenever you try to get any attribute from an instance object in Python, Python will check if the attribute exists in that object's class, and if so, if the attribute itself has a method named __get__. If it has, __get__ is called (with the instance and class where it is defined as parameters) - and whatever it returns is the attribute. Python uses this to implement methods: functions in Python 3 have a __get__ method that when called, will return another callable object that, in turn, when called will insert the self parameter in a call to the original function.
So, it is possible to create a class whose __get__ method will decide whether to return a function as a bound method or not depending on the outer class been marked as so - for example, it could check an specific flag non_destrutible. This could be done by using a decorator to wrap the method with this descriptor functionality
class Muteable:
def __init__(self, flag_attr):
self.flag_attr = flag_attr
def __call__(self, func):
"""Called when the decorator is applied"""
self.func = func
return self
def __get__(self, instance, owner):
if instance and getattr(instance, self.flag_attr, False):
raise AttributeError('Objects of type {0} have no {1} method'.format(instance.__class__.__name__, self.func.__name__))
return self.func.__get__(instance, owner)
class Planet:
def __init__(self, name=""):
pass
#Muteable("undestroyable")
def destroy(self):
print("Destroyed")
class BorgWorld(Planet):
undestroyable = True
And on the interactive prompt:
In [110]: Planet().destroy()
Destroyed
In [111]: BorgWorld().destroy()
...
AttributeError: Objects of type BorgWorld have no destroy method
In [112]: BorgWorld().destroy
AttributeError: Objects of type BorgWorld have no destroy method
Perceive that unlike simply overriding the method, this approach raises the error when the attribute is retrieved - and will even make hasattr work:
In [113]: hasattr(BorgWorld(), "destroy")
Out[113]: False
Although, it won't work if one tries to retrieve the method directly from the class, instead of from an instance - in that case the instance parameter to __get__ is set to None, and we can't say from which class it was retrieved - just the owner class, where it was declared.
In [114]: BorgWorld.destroy
Out[114]: <function __main__.Planet.destroy>
Second approach: __delattr__ on the metaclass:
While writting the above, it occurred me that Pythn does have the __delattr__ special method. If the Planet class itself implements __delattr__ and we'd try to delete the destroy method on specifc derived classes, it wuld nt work: __delattr__ gards the attribute deletion of attributes in instances - and if you'd try to del the "destroy" method in an instance, it would fail anyway, since the method is in the class.
However, in Python, the class itself is an instance - of its "metaclass". That is usually type . A proper __delattr__ on the metaclass of "Planet" could make possible the "disinheitance" of the "destroy" method by issuing a `del UndestructiblePlanet.destroy" after class creation.
Again, we use the descriptor protocol to have a proper "deleted method on the subclass":
class Deleted:
def __init__(self, cls, name):
self.cls = cls.__name__
self.name = name
def __get__(self, instance, owner):
raise AttributeError("Objects of type '{0}' have no '{1}' method".format(self.cls, self.name))
class Deletable(type):
def __delattr__(cls, attr):
print("deleting from", cls)
setattr(cls, attr, Deleted(cls, attr))
class Planet(metaclass=Deletable):
def __init__(self, name=""):
pass
def destroy(self):
print("Destroyed")
class BorgWorld(Planet):
pass
del BorgWorld.destroy
And with this method, even trying to retrieve or check for the method existense on the class itself will work:
In [129]: BorgWorld.destroy
...
AttributeError: Objects of type 'BorgWorld' have no 'destroy' method
In [130]: hasattr(BorgWorld, "destroy")
Out[130]: False
metaclass with a custom __prepare__ method.
Since metaclasses allow one to customize the object that contains the class namespace, it is possible to have an object that responds to a del statement within the class body, adding a Deleted descriptor.
For the user (programmer) using this metaclass, it is almost the samething, but for the del statement been allowed into the class body itself:
class Deleted:
def __init__(self, name):
self.name = name
def __get__(self, instance, owner):
raise AttributeError("No '{0}' method on class '{1}'".format(self.name, owner.__name__))
class Deletable(type):
def __prepare__(mcls,arg):
class D(dict):
def __delitem__(self, attr):
self[attr] = Deleted(attr)
return D()
class Planet(metaclass=Deletable):
def destroy(self):
print("destroyed")
class BorgPlanet(Planet):
del destroy
(The 'deleted' descriptor is the correct form to mark a method as 'deleted' - in this method, though, it can't know the class name at class creation time)
As a class decorator:
And given the "deleted" descriptor, one could simply inform the methods to be removed as a class decorator - there is no need for a metaclass in this case:
class Deleted:
def __init__(self, cls, name):
self.cls = cls.__name__
self.name = name
def __get__(self, instance, owner):
raise AttributeError("Objects of type '{0}' have no '{1}' method".format(self.cls, self.name))
def mute(*methods):
def decorator(cls):
for method in methods:
setattr(cls, method, Deleted(cls, method))
return cls
return decorator
class Planet:
def destroy(self):
print("destroyed")
#mute('destroy')
class BorgPlanet(Planet):
pass
Modifying the __getattribute__ mechanism:
For sake of completeness - what really makes Python reach methods and attributes on the super-class is what happens inside the __getattribute__ call. n the object version of __getattribute__ is where the algorithm with the priorities for "data-descriptor, instance, class, chain of base-classes, ..." for attribute retrieval is encoded.
So, changing that for the class is an easy an unique point to get a "legitimate" attribute error, without need for the "non-existent" descritor used on the previous methods.
The problem is that object's __getattribute__ does not make use of type's one to search the attribute in the class - if it did so, just implementing the __getattribute__ on the metaclass would suffice. One have to do that on the instance to avoid instance lookp of an method, and on the metaclass to avoid metaclass look-up. A metaclass can, of course, inject the needed code:
def blocker_getattribute(target, attr, attr_base):
try:
muted = attr_base.__getattribute__(target, '__muted__')
except AttributeError:
muted = []
if attr in muted:
raise AttributeError("object {} has no attribute '{}'".format(target, attr))
return attr_base.__getattribute__(target, attr)
def instance_getattribute(self, attr):
return blocker_getattribute(self, attr, object)
class M(type):
def __init__(cls, name, bases, namespace):
cls.__getattribute__ = instance_getattribute
def __getattribute__(cls, attr):
return blocker_getattribute(cls, attr, type)
class Planet(metaclass=M):
def destroy(self):
print("destroyed")
class BorgPlanet(Planet):
__muted__=['destroy'] # or use a decorator to set this! :-)
pass
If Undestroyable is a unique (or at least unusual) case, it's probably easiest to just redefine destroy():
class Undestroyable(Planet):
# ...
def destroy(self):
cls_name = self.__class__.__name__
raise AttributeError("%s has no attribute 'destroy'" % cls_name)
From the point of view of the user of the class, this will behave as though Undestroyable.destroy() doesn't exist … unless they go poking around with hasattr(Undestroyable, 'destroy'), which is always a possibility.
If it happens more often that you want subclasses to inherit some properties and not others, the mixin approach in chepner's answer is likely to be more maintainable. You can improve it further by making Destructible an abstract base class:
from abc import abstractmethod, ABCMeta
class Destructible(metaclass=ABCMeta):
#abstractmethod
def destroy(self):
pass
class BasePlanet:
# ...
pass
class Planet(BasePlanet, Destructible):
def destroy(self):
# ...
pass
class IndestructiblePlanet(BasePlanet):
# ...
pass
This has the advantage that if you try to instantiate the abstract class Destructible, you'll get an error pointing you at the problem:
>>> Destructible()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Destructible with abstract methods destroy
… similarly if you inherit from Destructible but forget to define destroy():
class InscrutablePlanet(BasePlanet, Destructible):
pass
>>> InscrutablePlanet()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class InscrutablePlanet with abstract methods destroy
Rather than remove an attribute that is inherited, only inherit destroy in the subclasses where it is applicable, via a mix-in class. This preserves the correct "is-a" semantics of inheritance.
class Destructible(object):
def destroy(self):
pass
class BasePlanet(object):
...
class Planet(BasePlanet, Destructible):
...
class IndestructiblePlanet(BasePlanet): # Does *not* inherit from Destructible
...
You can provide suitable definitions for destroy in any of Destructible, Planet, or any class that inherits from Planet.
Metaclasses and descriptor protocols are fun, but perhaps overkill. Sometimes, for raw functionality, you can't beat good ole' __slots__.
class Planet(object):
def __init__(self, name):
self.name = name
def destroy(self):
print("Boom! %s is toast!\n" % self.name)
class Undestroyable(Planet):
__slots__ = ['destroy']
def __init__(self,name):
super().__init__(name)
print()
x = Planet('Pluto') # Small, easy to destroy
y = Undestroyable('Jupiter') # Too big to fail
x.destroy()
y.destroy()
Boom! Pluto is toast!
Traceback (most recent call last):
File "planets.py", line 95, in <module>
y.destroy()
AttributeError: destroy
You cannot inherit only a portion of a class. Its all or nothing.
What you can do is to put the destroy function in a second level of the class, such you have the Planet-class without the destry-function, and then you make a DestroyablePlanet-Class where you add the destroy-function, which all the destroyable planets use.
Or you can put a flag in the construct of the Planet-Class which determines if the destroy function will be able to succeed or not, which is then checked in the destroy-function.

Python metaclasses: Why isn't __setattr__ called for attributes set during class definition?

I have the following python code:
class FooMeta(type):
def __setattr__(self, name, value):
print name, value
return super(FooMeta, self).__setattr__(name, value)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.
What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.
A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.
This:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Comes out pretty much as if you'd written:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).
The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.
Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:
from collections import MutableMapping
class SingleAssignDict(MutableMapping):
def __init__(self, *args, **kwargs):
self._d = dict(*args, **kwargs)
def __getitem__(self, key):
return self._d[key]
def __setitem__(self, key, value):
if key in self._d:
raise ValueError(
'Key {!r} already exists in SingleAssignDict'.format(key)
)
else:
self._d[key] = value
def __delitem__(self, key):
del self._d[key]
def __iter__(self):
return iter(self._d)
def __len__(self):
return len(self._d)
def __contains__(self, key):
return key in self._d
def __repr__(self):
return '{}({!r})'.format(type(self).__name__, self._d)
class RedefBlocker(type):
#classmethod
def __prepare__(metacls, name, bases, **kwargs):
return SingleAssignDict()
def __new__(metacls, name, bases, sad):
return super().__new__(metacls, name, bases, dict(sad))
class Okay(metaclass=RedefBlocker):
a = 1
b = 2
class Boom(metaclass=RedefBlocker):
a = 1
b = 2
a = 3
Running this gives me:
Traceback (most recent call last):
File "/tmp/redef.py", line 50, in <module>
class Boom(metaclass=RedefBlocker):
File "/tmp/redef.py", line 53, in Boom
a = 3
File "/tmp/redef.py", line 15, in __setitem__
'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict
Some notes:
__prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.
Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.
Compare Python 2:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Where the metaclass to invoke is determined from the dictionary, versus Python 3:
class Foo(metaclass=FooMeta):
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
pass
d['a'] = a
Foo = FooMeta('Foo', (), d)
Where the dictionary to use is determined from the metaclass.
There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:
class FooMeta(type):
def __new__(self, name, bases, attrs):
print attrs
return type.__new__(self, name, bases, attrs)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.
Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().
It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.
During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.
Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.

Accessing the class that owns a decorated method from the decorator

I'm writing a decorator for methods that must inspect the parent methods (the methods of the same name in the parents of the class in which I'm decorating).
Example (from the fourth example of PEP 318):
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
# here I want to reach the class owning the decorated method f,
# it should give me the class A
return new_f
return check_returns
class A(object):
#returns(int)
def compute(self, value):
return value * 3
So I'm looking for the code to type in place of # here I want...
Thanks.
As bobince said it, you can't access the surrounding class, because at the time the decorator is invoked, the class does not exist yet. If you need access to the full dictionary of the class and the bases, you should consider a metaclass:
__metaclass__
This variable can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().
Basically, we convert the returns decorator into something that just tells the metaclass to do some magic on class construction:
class CheckedReturnType(object):
def __init__(self, meth, rtype):
self.meth = meth
self.rtype = rtype
def returns(rtype):
def _inner(f):
return CheckedReturnType(f, rtype)
return _inner
class BaseInspector(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, CheckedReturnType):
# do your wrapping & checking here, base classes are in bases
# reassign to dct
return type.__new__(mcs, name, bases, dct)
class A(object):
__metaclass__ = BaseInspector
#returns(int)
def compute(self, value):
return value * 3
Mind that I have not tested this code, please leave comments if I should update this.
There are some articles on metaclasses by the highly recommendable David Mertz, which you might find interesting in this context.
here I want to reach the class owning the decorated method f
You can't because at the point of decoration, no class owns the method f.
class A(object):
#returns(int)
def compute(self, value):
return value * 3
Is the same as saying:
class A(object):
pass
#returns(int)
def compute(self, value):
return value*3
A.compute= compute
Clearly, the returns() decorator is built before the function is assigned to an owner class.
Now when you write a function to a class (either inline, or explicitly like this) it becomes an unbound method object. Now it has a reference to its owner class, which you can get by saying:
>>> A.compute.im_class
<class '__main__.A'>
So you can read f.im_class inside ‘new_f’, which is executed after the assignment, but not in the decorator itself.
(And even then it's a bit ugly relying on a CPython implementation detail if you don't need to. I'm not quite sure what you're trying to do, but things involving “get the owner class” are often doable using metaclasses.)

Categories

Resources