Related
I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
What is instance and owner here? (in __get__). What is the purpose of these parameters?
How would I call/use this example?
The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example:
temp=Temperature()
temp.celsius #calls celsius.__get__
Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method.
instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature).
You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class.
An article about descriptors can be found here.
EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.
Why do I need the descriptor class?
It gives you extra control over how attributes work. If you're used to getters and setters in Java, for example, then it's Python's way of doing that. One advantage is that it looks to users just like an attribute (there's no change in syntax). So you can start with an ordinary attribute and then, when you need to do something fancy, switch to a descriptor.
An attribute is just a mutable value. A descriptor lets you execute arbitrary code when reading or setting (or deleting) a value. So you could imagine using it to map an attribute to a field in a database, for example – a kind of ORM.
Another use might be refusing to accept a new value by throwing an exception in __set__ – effectively making the "attribute" read only.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
This is pretty subtle (and the reason I am writing a new answer here - I found this question while wondering the same thing and didn't find the existing answer that great).
A descriptor is defined on a class, but is typically called from an instance. When it's called from an instance both instance and owner are set (and you can work out owner from instance so it seems kinda pointless). But when called from a class, only owner is set – which is why it's there.
This is only needed for __get__ because it's the only one that can be called on a class. If you set the class value you set the descriptor itself. Similarly for deletion. Which is why the owner isn't needed there.
How would I call/use this example?
Well, here's a cool trick using similar classes:
class Celsius:
def __get__(self, instance, owner):
return 5 * (instance.fahrenheit - 32) / 9
def __set__(self, instance, value):
instance.fahrenheit = 32 + 9 * value / 5
class Temperature:
celsius = Celsius()
def __init__(self, initial_f):
self.fahrenheit = initial_f
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
(I'm using Python 3; for python 2 you need to make sure those divisions are / 5.0 and / 9.0). That gives:
100.0
32.0
Now there are other, arguably better ways to achieve the same effect in python (e.g. if celsius were a property, which is the same basic mechanism but places all the source inside the Temperature class), but that shows what can be done...
I am trying to understand what Python's descriptors are and what they can be useful for.
Descriptors are objects in a class namespace that manage instance attributes (like slots, properties, or methods). For example:
class HasDescriptors:
__slots__ = 'a_slot' # creates a descriptor
def a_method(self): # creates a descriptor
"a regular method"
#staticmethod # creates a descriptor
def a_static_method():
"a static method"
#classmethod # creates a descriptor
def a_class_method(cls):
"a class method"
#property # creates a descriptor
def a_property(self):
"a property"
# even a regular function:
def a_function(some_obj_or_self): # creates a descriptor
"create a function suitable for monkey patching"
HasDescriptors.a_function = a_function # (but we usually don't do this)
Pedantically, descriptors are objects with any of the following special methods, which may be known as "descriptor methods":
__get__: non-data descriptor method, for example on a method/function
__set__: data descriptor method, for example on a property instance or slot
__delete__: data descriptor method, again used by properties or slots
These descriptor objects are attributes in other object class namespaces. That is, they live in the __dict__ of the class object.
Descriptor objects programmatically manage the results of a dotted lookup (e.g. foo.descriptor) in a normal expression, an assignment, or a deletion.
Functions/methods, bound methods, property, classmethod, and staticmethod all use these special methods to control how they are accessed via the dotted lookup.
A data descriptor, like property, can allow for lazy evaluation of attributes based on a simpler state of the object, allowing instances to use less memory than if you precomputed each possible attribute.
Another data descriptor, a member_descriptor created by __slots__, allows memory savings (and faster lookups) by having the class store data in a mutable tuple-like datastructure instead of the more flexible but space-consuming __dict__.
Non-data descriptors, instance and class methods, get their implicit first arguments (usually named self and cls, respectively) from their non-data descriptor method, __get__ - and this is how static methods know not to have an implicit first argument.
Most users of Python need to learn only the high-level usage of descriptors, and have no need to learn or understand the implementation of descriptors further.
But understanding how descriptors work can give one greater confidence in one's mastery of Python.
In Depth: What Are Descriptors?
A descriptor is an object with any of the following methods (__get__, __set__, or __delete__), intended to be used via dotted-lookup as if it were a typical attribute of an instance. For an owner-object, obj_instance, with a descriptor object:
obj_instance.descriptor invokes
descriptor.__get__(self, obj_instance, owner_class) returning a value
This is how all methods and the get on a property work.
obj_instance.descriptor = value invokes
descriptor.__set__(self, obj_instance, value) returning None
This is how the setter on a property works.
del obj_instance.descriptor invokes
descriptor.__delete__(self, obj_instance) returning None
This is how the deleter on a property works.
obj_instance is the instance whose class contains the descriptor object's instance. self is the instance of the descriptor (probably just one for the class of the obj_instance)
To define this with code, an object is a descriptor if the set of its attributes intersects with any of the required attributes:
def has_descriptor_attrs(obj):
return set(['__get__', '__set__', '__delete__']).intersection(dir(obj))
def is_descriptor(obj):
"""obj can be instance of descriptor or the descriptor class"""
return bool(has_descriptor_attrs(obj))
A Data Descriptor has a __set__ and/or __delete__.
A Non-Data-Descriptor has neither __set__ nor __delete__.
def has_data_descriptor_attrs(obj):
return set(['__set__', '__delete__']) & set(dir(obj))
def is_data_descriptor(obj):
return bool(has_data_descriptor_attrs(obj))
Builtin Descriptor Object Examples:
classmethod
staticmethod
property
functions in general
Non-Data Descriptors
We can see that classmethod and staticmethod are Non-Data-Descriptors:
>>> is_descriptor(classmethod), is_data_descriptor(classmethod)
(True, False)
>>> is_descriptor(staticmethod), is_data_descriptor(staticmethod)
(True, False)
Both only have the __get__ method:
>>> has_descriptor_attrs(classmethod), has_descriptor_attrs(staticmethod)
(set(['__get__']), set(['__get__']))
Note that all functions are also Non-Data-Descriptors:
>>> def foo(): pass
...
>>> is_descriptor(foo), is_data_descriptor(foo)
(True, False)
Data Descriptor, property
However, property is a Data-Descriptor:
>>> is_data_descriptor(property)
True
>>> has_descriptor_attrs(property)
set(['__set__', '__get__', '__delete__'])
Dotted Lookup Order
These are important distinctions, as they affect the lookup order for a dotted lookup.
obj_instance.attribute
First the above looks to see if the attribute is a Data-Descriptor on the class of the instance,
If not, it looks to see if the attribute is in the obj_instance's __dict__, then
it finally falls back to a Non-Data-Descriptor.
The consequence of this lookup order is that Non-Data-Descriptors like functions/methods can be overridden by instances.
Recap and Next Steps
We have learned that descriptors are objects with any of __get__, __set__, or __delete__. These descriptor objects can be used as attributes on other object class definitions. Now we will look at how they are used, using your code as an example.
Analysis of Code from the Question
Here's your code, followed by your questions and answers to each:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
Your descriptor ensures you always have a float for this class attribute of Temperature, and that you can't use del to delete the attribute:
>>> t1 = Temperature()
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
Otherwise, your descriptors ignore the owner-class and instances of the owner, instead, storing state in the descriptor. You could just as easily share state across all instances with a simple class attribute (so long as you always set it as a float to the class and never delete it, or are comfortable with users of your code doing so):
class Temperature(object):
celsius = 0.0
This gets you exactly the same behavior as your example (see response to question 3 below), but uses a Pythons builtin (property), and would be considered more idiomatic:
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
What is instance and owner here? (in get). What is the purpose of these parameters?
instance is the instance of the owner that is calling the descriptor. The owner is the class in which the descriptor object is used to manage access to the data point. See the descriptions of the special methods that define descriptors next to the first paragraph of this answer for more descriptive variable names.
How would I call/use this example?
Here's a demonstration:
>>> t1 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1
>>>
>>> t1.celsius
1.0
>>> t2 = Temperature()
>>> t2.celsius
1.0
You can't delete the attribute:
>>> del t2.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
And you can't assign a variable that can't be converted to a float:
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __set__
ValueError: invalid literal for float(): 0x02
Otherwise, what you have here is a global state for all instances, that is managed by assigning to any instance.
The expected way that most experienced Python programmers would accomplish this outcome would be to use the property decorator, which makes use of the same descriptors under the hood, but brings the behavior into the implementation of the owner class (again, as defined above):
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
Which has the exact same expected behavior of the original piece of code:
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1.0
>>> t2.celsius
1.0
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't delete attribute
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in celsius
ValueError: invalid literal for float(): 0x02
Conclusion
We've covered the attributes that define descriptors, the difference between data- and non-data-descriptors, builtin objects that use them, and specific questions about use.
So again, how would you use the question's example? I hope you wouldn't. I hope you would start with my first suggestion (a simple class attribute) and move on to the second suggestion (the property decorator) if you feel it is necessary.
Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).
The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".
This represents the lookup of an attribute foobar on an instance of Class:
Two conditions are important here:
If the class of instance has an entry for the attribute name and it has __get__ and __set__.
If the instance has no entry for the attribute name but the class has one and it has __get__.
That's where descriptors come into it:
Data descriptors which have both __get__ and __set__.
Non-data descriptors which only have __get__.
In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.
The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).
Let's move to your specific questions:
Why do I need the descriptor class?
In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.
def test_function(self):
return self
class TestClass(object):
def test_method(self):
...
If you look up test_method on an instance you'll get back a "bound method":
>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>
Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):
>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>
You can even call this "self-bound method":
>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>
Note that I did not provide any arguments and the function did return the instance I had bound!
Functions are Non-data descriptors!
Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):
class Property(object):
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with #property, #name.setter, and #name.deleter (if present).
There are several other descriptors in the standard library, for example staticmethod, classmethod.
The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).
Another example would be a class property.
One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:
class TypedProperty(object):
__slots__ = ('_name', '_type')
def __init__(self, typ):
self._type = typ
def __get__(self, instance, klass=None):
if instance is None:
return self
return instance.__dict__[self._name]
def __set__(self, instance, value):
if not isinstance(value, self._type):
raise TypeError(f"Expected class {self._type}, got {type(value)}")
instance.__dict__[self._name] = value
def __delete__(self, instance):
del instance.__dict__[self._name]
def __set_name__(self, klass, name):
self._name = name
Then you can use the descriptor in a class:
class Test(object):
int_prop = TypedProperty(int)
And playing a bit with it:
>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10
>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>
Or a "lazy property":
class LazyProperty(object):
__slots__ = ('_fget', '_name')
def __init__(self, fget):
self._fget = fget
def __get__(self, instance, klass=None):
if instance is None:
return self
try:
return instance.__dict__[self._name]
except KeyError:
value = self._fget(instance)
instance.__dict__[self._name] = value
return value
def __set_name__(self, klass, name):
self._name = name
class Test(object):
#LazyProperty
def lazy(self):
print('calculating')
return 10
>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10
These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
It depends on how you look up the attribute. If you look up the attribute on an instance then:
the second argument is the instance on which you look up the attribute
the third argument is the class of the instance
In case you look up the attribute on the class (assuming the descriptor is defined on the class):
the second argument is None
the third argument is the class where you look up the attribute
So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).
How would I call/use this example?
Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius = 20 # setting it on one instance
>>> t2.celsius # looking it up on another instance
20.0
>>> Temperature.celsius # looking it up on the class
20.0
That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.
Why do I need the descriptor class?
Inspired by Fluent Python by Buciano Ramalho
Imaging you have a class like this
class LineItem:
price = 10.9
weight = 2.1
def __init__(self, name, price, weight):
self.name = name
self.price = price
self.weight = weight
item = LineItem("apple", 2.9, 2.1)
item.price = -0.9 # it's price is negative, you need to refund to your customer even you delivered the apple :(
item.weight = -0.8 # negative weight, it doesn't make sense
We should validate the weight and price in avoid to assign them a negative number, we can write less code if we use descriptor as a proxy as this
class Quantity(object):
__index = 0
def __init__(self):
self.__index = self.__class__.__index
self._storage_name = "quantity#{}".format(self.__index)
self.__class__.__index += 1
def __set__(self, instance, value):
if value > 0:
setattr(instance, self._storage_name, value)
else:
raise ValueError('value should >0')
def __get__(self, instance, owner):
return getattr(instance, self._storage_name)
then define class LineItem like this:
class LineItem(object):
weight = Quantity()
price = Quantity()
def __init__(self, name, weight, price):
self.name = name
self.weight = weight
self.price = price
and we can extend the Quantity class to do more common validating
You'd see https://docs.python.org/3/howto/descriptor.html#properties
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?
Some points to mug up before diving in:
__get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.
1. Conside this code:
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'
So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'
2. how to solve it?
Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?
Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:
from types import MethodType
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
def __set__(self, instance, value):
self.value = value
instance.__dict__["method"] = value
def __get__(self, instance, owner):
if instance is None:
return self
print (instance, owner)
return MethodType(self, instance)
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second
forget about set for now according to docs:
__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."
if you do: instance.method.__get__(instance)
Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>
this means instance: object of MyClass which is instance
and Owner is MyClass itself
3. __set__ Explaination:
__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case
(instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)
So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method.
Do:
instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}
Or you can check the __dict__ value using vars(instance.method)
prints: {'name': 'Internal call', 'value': 'value'}
I hope things are clear now:)
I tried (with minor changes as suggested) the code from Andrew Cooke's answer. (I am running python 2.7).
The code:
#!/usr/bin/env python
class Celsius:
def __get__(self, instance, owner): return 9 * (instance.fahrenheit + 32) / 5.0
def __set__(self, instance, value): instance.fahrenheit = 32 + 5 * value / 9.0
class Temperature:
def __init__(self, initial_f): self.fahrenheit = initial_f
celsius = Celsius()
if __name__ == "__main__":
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
The result:
C:\Users\gkuhn\Desktop>python test2.py
<__main__.Celsius instance at 0x02E95A80>
212
With Python prior to 3, make sure you subclass from object which will make the descriptor work correctly as the get magic does not work for old style classes.
I'd like to do something like this:
class X:
#classmethod
def id(cls):
return cls.__name__
def id(self):
return self.__class__.__name__
And now call id() for either the class or an instance of it:
>>> X.id()
'X'
>>> X().id()
'X'
Obviously, this exact code doesn't work, but is there a similar way to make it work?
Or any other workarounds to get such behavior without too much "hacky" stuff?
Class and instance methods live in the same namespace and you cannot reuse names like that; the last definition of id will win in that case.
The class method will continue to work on instances however, there is no need to create a separate instance method; just use:
class X:
#classmethod
def id(cls):
return cls.__name__
because the method continues to be bound to the class:
>>> class X:
... #classmethod
... def id(cls):
... return cls.__name__
...
>>> X.id()
'X'
>>> X().id()
'X'
This is explicitly documented:
It can be called either on the class (such as C.f()) or on an instance (such as C().f()). The instance is ignored except for its class.
If you do need distinguish between binding to the class and an instance
If you need a method to work differently based on where it is being used on; bound to a class when accessed on the class, bound to the instance when accessed on the instance, you'll need to create a custom descriptor object.
The descriptor API is how Python causes functions to be bound as methods, and bind classmethod objects to the class; see the descriptor howto.
You can provide your own descriptor for methods by creating an object that has a __get__ method. Here is a simple one that switches what the method is bound to based on context, if the first argument to __get__ is None, then the descriptor is being bound to a class, otherwise it is being bound to an instance:
class class_or_instancemethod(classmethod):
def __get__(self, instance, type_):
descr_get = super().__get__ if instance is None else self.__func__.__get__
return descr_get(instance, type_)
This re-uses classmethod and only re-defines how it handles binding, delegating the original implementation for instance is None, and to the standard function __get__ implementation otherwise.
Note that in the method itself, you may then have to test, what it is bound to. isinstance(firstargument, type) is a good test for this:
>>> class X:
... #class_or_instancemethod
... def foo(self_or_cls):
... if isinstance(self_or_cls, type):
... return f"bound to the class, {self_or_cls}"
... else:
... return f"bound to the instance, {self_or_cls"
...
>>> X.foo()
"bound to the class, <class '__main__.X'>"
>>> X().foo()
'bound to the instance, <__main__.X object at 0x10ac7d580>'
An alternative implementation could use two functions, one for when bound to a class, the other when bound to an instance:
class hybridmethod:
def __init__(self, fclass, finstance=None, doc=None):
self.fclass = fclass
self.finstance = finstance
self.__doc__ = doc or fclass.__doc__
# support use on abstract base classes
self.__isabstractmethod__ = bool(
getattr(fclass, '__isabstractmethod__', False)
)
def classmethod(self, fclass):
return type(self)(fclass, self.finstance, None)
def instancemethod(self, finstance):
return type(self)(self.fclass, finstance, self.__doc__)
def __get__(self, instance, cls):
if instance is None or self.finstance is None:
# either bound to the class, or no instance method available
return self.fclass.__get__(cls, None)
return self.finstance.__get__(instance, cls)
This then is a classmethod with an optional instance method. Use it like you'd use a property object; decorate the instance method with #<name>.instancemethod:
>>> class X:
... #hybridmethod
... def bar(cls):
... return f"bound to the class, {cls}"
... #bar.instancemethod
... def bar(self):
... return f"bound to the instance, {self}"
...
>>> X.bar()
"bound to the class, <class '__main__.X'>"
>>> X().bar()
'bound to the instance, <__main__.X object at 0x10a010f70>'
Personally, my advice is to be cautious about using this; the exact same method altering behaviour based on the context can be confusing to use. However, there are use-cases for this, such as SQLAlchemy's differentiation between SQL objects and SQL values, where column objects in a model switch behaviour like this; see their Hybrid Attributes documentation. The implementation for this follows the exact same pattern as my hybridmethod class above.
I have no idea what's your actual use case is, but you can do something like this using a descriptor:
class Desc(object):
def __get__(self, ins, typ):
if ins is None:
print 'Called by a class.'
return lambda : typ.__name__
else:
print 'Called by an instance.'
return lambda : ins.__class__.__name__
class X(object):
id = Desc()
x = X()
print x.id()
print X.id()
Output
Called by an instance.
X
Called by a class.
X
It can be done, quite succinctly, by binding the instance-bound version of your method explicitly to the instance (rather than to the class). Python will invoke the instance attribute found in Class().__dict__ when Class().foo() is called (because it searches the instance's __dict__ before the class'), and the class-bound method found in Class.__dict__ when Class.foo() is called.
This has a number of potential use cases, though whether they are anti-patterns is open for debate:
class Test:
def __init__(self):
self.check = self.__check
#staticmethod
def check():
print('Called as class')
def __check(self):
print('Called as instance, probably')
>>> Test.check()
Called as class
>>> Test().check()
Called as instance, probably
Or... let's say we want to be able to abuse stuff like map():
class Str(str):
def __init__(self, *args):
self.split = self.__split
#staticmethod
def split(sep=None, maxsplit=-1):
return lambda string: string.split(sep, maxsplit)
def __split(self, sep=None, maxsplit=-1):
return super().split(sep, maxsplit)
>>> s = Str('w-o-w')
>>> s.split('-')
['w', 'o', 'w']
>>> Str.split('-')(s)
['w', 'o', 'w']
>>> list(map(Str.split('-'), [s]*3))
[['w', 'o', 'w'], ['w', 'o', 'w'], ['w', 'o', 'w']]
"types" provides something quite interesting since Python 3.4: DynamicClassAttribute
It is not doing 100% of what you had in mind, but it seems to be closely related, and you might need to tweak a bit my metaclass but, rougly, you can have this;
from types import DynamicClassAttribute
class XMeta(type):
def __getattr__(self, value):
if value == 'id':
return XMeta.id # You may want to change a bit that line.
#property
def id(self):
return "Class {}".format(self.__name__)
That would define your class attribute. For the instance attribute:
class X(metaclass=XMeta):
#DynamicClassAttribute
def id(self):
return "Instance {}".format(self.__class__.__name__)
It might be a bit overkill especially if you want to stay away from metaclasses. It's a trick I'd like to explore on my side, so I just wanted to share this hidden jewel, in case you can polish it and make it shine!
>>> X().id
'Instance X'
>>> X.id
'Class X'
Voila...
In your example, you could simply delete the second method entirely, since both the staticmethod and the class method do the same thing.
If you wanted them to do different things:
class X:
def id(self=None):
if self is None:
# It's being called as a static method
else:
# It's being called as an instance method
(Python 3 only) Elaborating on the idea of a pure-Python implementation of #classmethod, we can declare an #class_or_instance_method as a decorator, which is actually a class implementing the attribute descriptor protocol:
import inspect
class class_or_instance_method(object):
def __init__(self, f):
self.f = f
def __get__(self, instance, owner):
if instance is not None:
class_or_instance = instance
else:
class_or_instance = owner
def newfunc(*args, **kwargs):
return self.f(class_or_instance, *args, **kwargs)
return newfunc
class A:
#class_or_instance_method
def foo(self_or_cls, a, b, c=None):
if inspect.isclass(self_or_cls):
print("Called as a class method")
else:
print("Called as an instance method")
I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:
class ShowMeList(object):
def __init__(self, it):
self._data = list(it)
def __getattr__(self, name):
attr = object.__getattribute__(self._data, name)
if callable(attr):
def wrapper(*a, **kw):
print "before the call"
result = attr(*a, **kw)
print "after the call"
return result
return wrapper
return attr
If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
If I don't inherit from object (first line class ShowMeList:) everything works as expected:
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
How do I accomplish this intercept with new style classes?
For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.
class Wrapper(object):
"""Wrapper class that provides proxy access to some internal instance."""
__wraps__ = None
__ignore__ = "class mro new init setattr getattr getattribute"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
# provide proxy access to regular attributes of wrapped object
def __getattr__(self, name):
return getattr(self._obj, name)
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Usage:
class DictWrapper(Wrapper):
__wraps__ = dict
wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))
# make sure it worked....
assert "b" in wrapped_dict # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3) # __eq__
assert "'a': 1" in str(wrapped_dict) # __str__
assert wrapped_dict.__doc__.startswith("dict()") # __doc__
Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.
Consider the following:
>>> class C:
x = 1
def __init__(self):
self.y = 2
def __getattr__(self, attr):
print(attr)
>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z
The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).
On your example, the __repr__ and many other magic methods are already defined in the object class.
One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.
As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.
Cut and copy from the documentation:
For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
It bugs me that the default __repr__() for a class is so uninformative:
>>> class Opaque(object): pass
...
>>> Opaque()
<__main__.Opaque object at 0x7f3ac50eba90>
... so I've been thinking about how to improve it. After a little consideration, I came up with this abstract base class which leverages the pickle protocol's __getnewargs__() method:
from abc import abstractmethod
class Repro(object):
"""Abstract base class for objects with informative ``repr()`` behaviour."""
#abstractmethod
def __getnewargs__(self):
raise NotImplementedError
def __repr__(self):
signature = ", ".join(repr(arg) for arg in self.__getnewargs__())
return "%s(%s)" % (self.__class__.__name__, signature)
Here's a trivial example of its usage:
class Transparent(Repro):
"""An example of a ``Repro`` subclass."""
def __init__(self, *args):
self.args = args
def __getnewargs__(self):
return self.args
... and the resulting repr() behaviour:
>>> Transparent("an absurd signature", [1, 2, 3], str)
Transparent('an absurd signature', [1, 2, 3], <type 'str'>)
>>>
Now, I can see one reason Python doesn't do this by default straight away - requiring every class to define a __getnewargs__() method would be more burdensome than expecting (but not requiring) that it defines a __repr__() method.
What I'd like to know is: how dangerous and/or fragile is it? Off-hand, I can't think of anything that could go terribly wrong except that if a Repro instance contained itself, you'd get infinite recursion ... but that's solveable, at the cost of making the code above uglier.
What else have I missed?
If you're into this sort of thing, why not have the arguments automatically picked up from __init__ by using a decorator? Then you don't need to burden the user with manually handling them, and you can transparently handle normal method signatures with multiple arguments. Here's a quick version I came up with:
def deco(f):
def newFunc(self, *args, **kwargs):
self._args = args
self._kwargs = kwargs
f(self, *args, **kwargs)
return newFunc
class AutoRepr(object):
def __repr__(self):
args = ', '.join(repr(arg) for arg in self._args)
kwargs = ', '.join('{0}={1}'.format(k, repr(v)) for k, v in self._kwargs.iteritems())
allArgs = ', '.join([args, kwargs]).strip(', ')
return '{0}({1})'.format(self.__class__.__name__, allArgs)
Now you can define subclasses of AutoRepr normally, with normal __init__ signatures:
class Thingy(AutoRepr):
#deco
def __init__(self, foo, bar=88):
self.foo = foo
self.bar = bar
And the __repr__ automatically works:
>>> Thingy(1, 2)
Thingy(1, 2)
>>> Thingy(10)
Thingy(10)
>>> Thingy(1, bar=2)
Thingy(1, bar=2)
>>> Thingy(bar=1, foo=2)
Thingy(foo=2, bar=1)
>>> Thingy([1, 2, 3], "Some junk")
Thingy([1, 2, 3], 'Some junk')
Putting #deco on your __init__ is much easier than writing a whole __getnewargs__. And if you don't even want to have to do that, you could write a metaclass that automatically decorates the __init__ method in this way.
One problem with this whole idea is that there can be some kinds of objects who's state is not fully dependent on the arguments given to its constructor. For a trivial case, consider a class with random state:
import random
def A(object):
def __init__(self):
self.state = random.random()
There's no way for this class to correctly implement __getnewargs__, and so your implantation of __repr__ is also impossible. It may be that a class like the one above is not well designed. But pickle can handle it with no problems (I assume using the __reduce__ method inherited from object, but my pickle-fu is not enough to say so with certainty).
This is why it is nice that __repr__ can be coded to do whatever you want. If you want the internal state to be visible, you can make your class's __repr__ do that. If the object should be opaque, you can do that too. For the class above, I'd probably implement __repr__ like this:
def __repr__(self):
return "<A object with state=%f>" % self.state
I am using the python mock framework for testing (http://www.voidspace.org.uk/python/mock/) and I want to mock out a superclass and focus on testing the subclasses' added behavior.
(For those interested I have extended pymongo.collection.Collection and I want to only test my added behavior. I do not want to have to run mongodb as another process for testing purposes.)
For this discussion, A is the superclass and B is the subclass. Furthermore, I define direct and indirect superclass calls as shown below:
class A(object):
def method(self):
...
def another_method(self):
...
class B(A):
def direct_superclass_call(self):
...
A.method(self)
def indirect_superclass_call(self):
...
super(A, self).another_method()
Approach #1
Define a mock class for A called MockA and use mock.patch to substitute it for the test at runtime. This handles direct superclass calls. Then manipulate B.__bases__ to handle indirect superclass calls. (see below)
The issue that arises is that I have to write MockA and in some cases (as in the case for pymongo.collection.Collection) this can involve a lot of work to unravel all of the internal calls to mock out.
Approach #2
The desired approach is to somehow use a mock.Mock() class to handle calls on the the mock just in time, as well as defined return_value or side_effect in place in the test. In this manner, I have to do less work by avoiding the definition of MockA.
The issue that I am having is that I cannot figure out how to alter B.__bases__ so that an instance of mock.Mock() can be put in place as a superclass (I must need to somehow do some direct binding here). Thus far I have determined, that super() examines the MRO and then calls the first class that defines the method in question. I cannot figure out how to get a superclass to handle the check to it and succeed if it comes across a mock class. __getattr__ does not seem to be used in this case. I want super to to think that the method is defined at this point and then use the mock.Mock() functionality as usual.
How does super() discover what attributes are defined within the class in the MRO sequence? And is there a way for me to interject here and to somehow get it to utilize a mock.Mock() on the fly?
import mock
class A(object):
def __init__(self, value):
self.value = value
def get_value_direct(self):
return self.value
def get_value_indirect(self):
return self.value
class B(A):
def __init__(self, value):
A.__init__(self, value)
def get_value_direct(self):
return A.get_value_direct(self)
def get_value_indirect(self):
return super(B, self).get_value_indirect()
# approach 1 - use a defined MockA
class MockA(object):
def __init__(self, value):
pass
def get_value_direct(self):
return 0
def get_value_indirect(self):
return 0
B.__bases__ = (MockA, ) # - mock superclass
with mock.patch('__main__.A', MockA):
b2 = B(7)
print '\nApproach 1'
print 'expected result = 0'
print 'direct =', b2.get_value_direct()
print 'indirect =', b2.get_value_indirect()
B.__bases__ = (A, ) # - original superclass
# approach 2 - use mock module to mock out superclass
# what does XXX need to be below to use mock.Mock()?
#B.__bases__ = (XXX, )
with mock.patch('__main__.A') as mymock:
b3 = B(7)
mymock.get_value_direct.return_value = 0
mymock.get_value_indirect.return_value = 0
print '\nApproach 2'
print 'expected result = 0'
print 'direct =', b3.get_value_direct()
print 'indirect =', b3.get_value_indirect() # FAILS HERE as the old superclass is called
#B.__bases__ = (A, ) # - original superclass
is there a way for me to interject here and to somehow get it to utilize a mock.Mock() on the fly?
There may be better approaches, but you can always write your own super() and inject it into the module that contains the class you're mocking. Have it return whatever it should based on what's calling it.
You can either just define super() in the current namespace (in which case the redefinition only applies to the current module after the definition), or you can import __builtin__ and apply the redefinition to __builtin__.super, in which case it will apply globally in the Python session.
You can capture the original super function (if you need to call it from your implementation) using a default argument:
def super(type, obj=None, super=super):
# inside the function, super refers to the built-in
I played around with mocking out super() as suggested by kindall. Unfortunately, after a great deal of effort it became quite complicated to handle complex inheritance cases.
After some work I realized that super() accesses the __dict__ of classes directly when resolving attributes through the MRO (it does not do a getattr type of call). The solution is to extend a mock.MagicMock() object and wrap it with a class to accomplish this. The wrapped class can then be placed in the __bases__ variable of a subclass.
The wrapped object reflects all defined attributes of the target class to the __dict__ of the wrapping class so that super() calls resolve to the properly patched in attributes within the internal MagicMock().
The following code is the solution that I have found to work thus far. Note that I actually implement this within a context handler. Also, care has to be taken to patch in the proper namespaces if importing from other modules.
This is a simple example illustrating the approach:
from mock import MagicMock
import inspect
class _WrappedMagicMock(MagicMock):
def __init__(self, *args, **kwds):
object.__setattr__(self, '_mockclass_wrapper', None)
super(_WrappedMagicMock, self).__init__(*args, **kwds)
def wrap(self, cls):
# get defined attribtues of spec class that need to be preset
base_attrs = dir(type('Dummy', (object,), {}))
attrs = inspect.getmembers(self._spec_class)
new_attrs = [a[0] for a in attrs if a[0] not in base_attrs]
# pre set mocks for attributes in the target mock class
for name in new_attrs:
setattr(cls, name, getattr(self, name))
# eat up any attempts to initialize the target mock class
setattr(cls, '__init__', lambda *args, **kwds: None)
object.__setattr__(self, '_mockclass_wrapper', cls)
def unwrap(self):
object.__setattr__(self, '_mockclass_wrapper', None)
def __setattr__(self, name, value):
super(_WrappedMagicMock, self).__setattr__(name, value)
# be sure to reflect to changes wrapper class if activated
if self._mockclass_wrapper is not None:
setattr(self._mockclass_wrapper, name, value)
def _get_child_mock(self, **kwds):
# when created children mocks need only be MagicMocks
return MagicMock(**kwds)
class A(object):
x = 1
def __init__(self, value):
self.value = value
def get_value_direct(self):
return self.value
def get_value_indirect(self):
return self.value
class B(A):
def __init__(self, value):
super(B, self).__init__(value)
def f(self):
return 2
def get_value_direct(self):
return A.get_value_direct(self)
def get_value_indirect(self):
return super(B, self).get_value_indirect()
# nominal behavior
b = B(3)
assert b.get_value_direct() == 3
assert b.get_value_indirect() == 3
assert b.f() == 2
assert b.x == 1
# using mock class
MockClass = type('MockClassWrapper', (), {})
mock = _WrappedMagicMock(A)
mock.wrap(MockClass)
# patch the mock in
B.__bases__ = (MockClass, )
A = MockClass
# set values within the mock
mock.x = 0
mock.get_value_direct.return_value = 0
mock.get_value_indirect.return_value = 0
# mocked behavior
b = B(7)
assert b.get_value_direct() == 0
assert b.get_value_indirect() == 0
assert b.f() == 2
assert b.x == 0