In the rare cases that an object's __getattribute__ method must be overridden, a common mistake is to try and return the attribute in question like so:
class WrongWay:
def __getattribute__(self, name):
# Custom functionality goes here
return self.__dict__[name]
This code always yields a RuntimeError due to a recursive loop, since self.__dict__ is in itself an attribute reference that calls upon the same __getattribute__ method.
According to this answer, the correct solution to this problem is to replace last line with:
...
return super().__getattribute__(self, name) # Defer responsibility to the superclass
This solution works when run through the Python 3 interpreter, but it also seems to violate __getattribute__'s promised functionality. Even if the superclass chain is traversed up to object, at the end of the line somebody will eventually have to return self.something, and by definition that attribute reference must first get through the child's __getattribute__ method.
How does Python get around this recursion issue? In object.__getattribute__, how is anything returned without looping into another request?
at the end of the line somebody will eventually have to return self.something, and by definition that attribute reference must first get through the child's __getattribute__() method.
That's not correct. object.__getattribute__ is not defined as returning self.anything, and it does not respect descendant class implementations of __getattribute__. object.__getattribute__ is the default attribute access implementation, and it always performs its job through the default attribute access mechanism.
Similarly, object.__eq__ is not defined as returning self == other_thing, and it does not respect descendant class implementations of __eq__. object.__str__ is not defined as returning str(self), and it does not respect descendant class implementations of __str__. object's methods are the default implementations of those methods, and they always do the default thing.
Related
I have a class as follows:
class Lz:
def __init__(self, b):
self.b = b
def __getattr__(self, item):
return self.b.__getattribute__(item)
And I create an instance and print :
a = Lz('abc')
print(a)
Result is: abc
I have set a breakpoint at line return self.b.__getattribute__(item), item show __str__
I don't know why it calls __getattr__, and item is __str__ when I access the instance.
print calls __str__ (see this question for details), but as Lz does not have a __str__ method, a lookup for an attribute named '__str__' takes place using __getattr__.
So if you add a __str__ method, __getattr__ should not be called anymore when printing objects of the Lz class.
print(obj) invokes str(obj) (to get a printable representation), which in turn tries to invokes obj.__str__() (and fallback to something else if this fails, but that's not the point here).
You defined Lz as an old-style class, so it's doesn't by default have a __str__ method (new-style classes inherit this from object), but you defined a __getattr__() method, so this is what gets invoked in the end (__getattr__() is the last thing the attribute lookup will invoke when everything else has failed).
NB: in case you don't already know, since everything in Python is an object - include classes, functions, methods etc - Python doesn't make difference between "data" attributes and "method" attributes - those are all attributes, period.
NB2: directly accessing __magic__ names is considered bad practice. Those names are implementation support for operators or operator-like generic functions (ie len(), type etc), and you are supposed to use the operator or generic function instead. IOW, this:
return self.b.__getattribute__(item)
should be written as
return getattr(self.b, item)
(getattr() is the generic function version of the "dot" attribute lookup operator (.))
Why are python instance methods callable, but static methods and class methods not callable?
I did the following:
class Test():
class_var = 42
#classmethod
def class_method(cls):
pass
#staticmethod
def static_method():
pass
def instance_method(self):
pass
for attr, val in vars(Test).items():
if not attr.startswith("__"):
print (attr, "is %s callable" % ("" if callable(val) else "NOT"))
The result is:
static_method is NOT callable
instance_method is callable
class_method is NOT callable
class_var is NOT callable
Technically this may be because instance method object might have a particular attribute (not) set in a particular way (possibly __call__). Why such asymmetry, or what purpose does it serve?
I came across this while learning python inspection tools.
Additional remarks from comments:
The SO answer linked in the comments says that the static/class methods are descriptors , which are not callable. Now I am curious, why are descriptors made not callable, since descriptors are class with particular attributes (one of __get__, __set__, __del___) defined.
Why are descriptors not callable? Basically because they don't need to be. Not every descriptor represents a callable either.
As you correctly note, the descriptor protocol consists of __get__, __set__ and __del__. Note no __call__, that's the technical reason why it's not callable. The actual callable is the return value of your static_method.__get__(...).
As for the philosophical reason, let's look at the class. The contents of the __dict__, or in your case results of vars(), are basically locals() of the class block. If you define a function, it gets dumped as a plain function. If you use a decorator, such as #staticmethod, it's equivalent to something like:
def _this_is_not_stored_anywhere():
pass
static_method = staticmethod(_this_is_not_stored_anywhere)
I.e., static_method is assigned a return value of the staticmethod() function.
Now, function objects actually implement the descriptor protocol - every function has a __get__ method on it. This is where the special self and the bound-method behavior comes from. See:
def xyz(what):
print(what)
repr(xyz) # '<function xyz at 0x7f8f924bdea0>'
repr(xyz.__get__("hello")) # "<bound method str.xyz of 'hello'>"
xyz.__get__("hello")() # "hello"
Because of how the class calls __get__, your test.instance_method binds to the instance and gets it pre-filled as it first argument.
But the whole point of #classmethod and #staticmethod is that they do something special to avoid the default "bound method" behavior! So they can't return a plain function. Instead they return a descriptor object with a custom __get__ implementation.
Of course, you could put a __call__ method on this descriptor object, but why? It's code that you don't need in practice; you can almost never touch the descriptor object itself. If you do (in code similar to yours), you still need special handling for descriptors, because a general descriptor doesn't have to be(have like a) callable - properties are descriptors too. So you don't want __call__ in the descriptor protocol. So if a third party "forgets" to implement __call__ on something you consider a "callable", your code will miss it.
Also, the object is a descriptor, not a function. Putting a __call__ method on it would be masking its true nature :) I mean, it's not wrong per se, it's just ... something that you should never need for anything.
BTW, in case of classmethod/staticmethod, you can get back the original function from their __func__ attribute.
In Python when I call the hasattr on a #property decorator the hasattr function actually runs the #property code block.
E.g. a class:
class GooglePlusUser(object):
def __init__(self, master):
self.master = master
def get_user_id(self):
return self.master.google_plus_service.people().get(userId='me').execute()['id']
#property
def profile(self):
# this runs with hasattr
return self.master.google_plus_service.people().get(userId='me').execute()
Running the following code calls the profile property and actually makes the call:
#Check if the call is an attribute
if not hasattr(google_plus_user, call):
self.response.out.write('Unknown call')
return
Why? How can I solve this without making the api call?
hasattr() works by actually retrieving the attribute; if an exception is thrown hasattr() returns False. That's because that is the only reliable way of knowing if an attribute exists, since there are so many dynamic ways to inject attributes on Python objects (__getattr__, __getattribute__, property objects, meta classes, etc.).
From the hasattr() documentation:
This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.
If you don't want a property to be invoked when doing this, then don't use hasattr. Use vars() (which returns the instance dictionary) or dir() (which gives you a list of names on the class as well). This won't let you discover dynamic attributes handled by __getattr__ or __getattribute__ hooks however.
hasattr is basically implemented like this (except in C):
def hasattr(obj, attrname):
try:
getattr(obj, attname)
except AttributeError:
return False
return True
So in true "easier to ask for forgiveness than permission" (EAFP) fashion, to find out if an object has a given attribute, Python simply tries to get the attribute, and converts failure to a return value of False. Since it's really getting the attribute in the success case, hasattr() can trigger code for property and other descriptors.
To check for an attribute without triggering descriptors, you can write your own hasattr that traverses the object's method resolution order and checks to see whether the name is in each class's __dict__ (or __slots__). Since this isn't attribute access, it won't trigger properties.
Conveniently, Python already has a way to walk the method resolution order and gather the names of attributes from an instance's classes: dir(). A simple way to write such a method, then, would be:
# gingerly test whether an attribute exists, avoiding triggering descriptor code
def gentle_hasattr(obj, name):
return name in dir(obj) or hasattr(obj, name)
Note that we fall back to using hasattr() if we can't find the desired name in dir(), because dir() won't find dynamic attributes (i.e., where __getattr__ is overridden). Code for these will still be triggered, of course, so if you don't care that you don't find them, you could omit the or clause.
On balance, this is wasteful, since it gets all relevant attribute names when we're interested only in whether a specific one exists, but it'll do in a pinch. I'm not even sure that doing the loop yourself rather than calling dir() would be faster on average, since it'll be in Python rather than in C.
Making the variable as a class variable and then calling hasattr on it did the trick for me.
if not hasattr(google_plus_user, GooglePlusUser.call):
self.response.out.write('Unknown call')
return
What are some general rules of thumb for choosing which of these to implement in a given class, in a given situation?
I have read the docs, and so understand the difference between them. Rather, I am looking for guidance on how to best integrate their usage into my workflow by being better able to notice more subtle opportunities to use them, and which to use when. That kind of thing. The methods in question are (to my knowledge):
## fallback
__getattr__
__setattr__
__delattr__
## full control
__getattribute__
##(no __setattribute__ ? What's the deal there?)
## (the descriptor protocol)
__get__
__set__
__delete__
__setattribute__ does not exist because __setattr__ is always called. __getattr__ is only called for f.x if the attribute lookup fails via the normal channel (which is provided by __getattribute__, so that function is similarly always called).
The descriptor protocol is slightly orthogonal to the others. Given
class Foo(object):
def __init__(self):
self.x = 5
f = Foo()
The following are true:
f.x would invoke f.__getattribute__('x') if it were defined.
f.x would not invoke f.__getattr__('x') if it were defined.
f.y would invoke f.__getattr__('y') if it were defined, or else
f.__getattribute__('y') if it were defined.
The descriptor is invoked by an attribute, rather than for an attribute. That is:
class MyDescriptor(object):
def __get__(...):
pass
def __set__(...):
pass
class Foo(object):
x = MyDescriptor()
f = Foo()
Now, f.x would cause type(f).__dict__['x'].__get__ to be called, and f.x = 3 would call type(f).__dict__['x'].__set__(3).
That is, Foo.__getattr__ and Foo.__getattribute__ would be used to find what f.x references; once you have that, f.x produces the result of type(f.x).__get__() if defined, and f.x = y invokes f.x.__set__(y) if defined.
(The above calls to __get__ and __set__ are only approximately correct, since I've left out the details of what arguments __get__ and __set__ actually receive, but this should be enough to explain the difference between __get__ and __getattr[ibute]__.)
Put yet another way, if MyDescriptor did not define __get__, then f.x would simply return the instance of MyDescriptor.
For __getattr__ vs __getattribute__, see for example Difference between __getattr__ vs __getattribute__ .
__get__ is not really related. I'm going to quote from the official documentation for the descriptor protocol here:
The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses. If the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods were defined.
The purpose of __get__ is to control what happens once a.x is found through that "lookup chain" (for example, to create a method object instead of returning a plain function found via type(a).__dict__['x']); the purpose of __getattr__ and __getattribute__ is to alter the lookup chain itself.
There is no __setattribute__ because there is only one way to actually set an attribute of an object, under the hood. You might want to cause other things to happen "magically" when an attribute is set; but __setattr__ covers that. __setattribute__ couldn't possibly provide any functionality that __setattr__ doesn't already.
However - the best answer, in the overwhelming majority of cases, is to not even think of using any of these. First look to higher-level abstractions, such as propertys, classmethods, and staticmethods. If you think you need specialized tools like this, and can't figure it out for yourself, there's a pretty good chance you're wrong in your thinking; but regardless, it's better to post a more specific question in that case.
class Foo(object):
pass
foo = Foo()
def bar(self):
print 'bar'
Foo.bar = bar
foo.bar() #bar
Coming from JavaScript, if a "class" prototype was augmented with a certain attribute. It is known that all instances of that "class" would have that attribute in its prototype chain, hence no modifications has to be done on any of its instances or "sub-classes".
In that sense, how can a Class-based language like Python achieve Monkey patching?
The real question is, how can it not? In Python, classes are first-class objects in their own right. Attribute access on instances of a class is resolved by looking up attributes on the instance, and then the class, and then the parent classes (in the method resolution order.) These lookups are all done at runtime (as is everything in Python.) If you add an attribute to a class after you create an instance, the instance will still "see" the new attribute, simply because nothing prevents it.
In other words, it works because Python doesn't cache attributes (unless your code does), because it doesn't use negative caching or shadowclasses or any of the optimization techniques that would inhibit it (or, when Python implementations do, they take into account the class might change) and because everything is runtime.
I just read through a bunch of documentation, and as far as I can tell, the whole story of how foo.bar is resolved, is as follows:
Can we find foo.__getattribute__ by the following process? If so, use the result of foo.__getattribute__('bar').
(Looking up __getattribute__ will not cause infinite recursion, but the implementation of it might.)
(In reality, we will always find __getattribute__ in new-style objects, as a default implementation is provided in object - but that implementation is of the following process. ;) )
(If we define a __getattribute__ method in Foo, and access foo.__getattribute__, foo.__getattribute__('__getattribute__') will be called! But this does not imply infinite recursion - if you are careful ;) )
Is bar a "special" name for an attribute provided by the Python runtime (e.g. __dict__, __class__, __bases__, __mro__)? If so, use that. (As far as I can tell, __getattribute__ falls into this category, which avoids infinite recursion.)
Is bar in the foo.__dict__ dict? If so, use foo.__dict__['bar'].
Does foo.__mro__ exist (i.e., is foo actually a class)? If so,
For each base-class base in foo.__mro__[1:]:
(Note that the first one will be foo itself, which we already searched.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
For each base-class base of foo.__class__.__mro__:
(Note that this recursion is not a problem: those attributes should always exist, and fall into the "provided by the Python runtime" case. foo.__class__.__mro__[0] will always be foo.__class__, i.e. Foo in our example.)
(Note that we do this even if foo.__mro__ exists. This is because classes have a class, too: its name is type, and it provides, among other things, the method used to calculate __mro__ attributes in the first place.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
If we still haven't found something to use: can we find foo.__getattr__ by the preceding process? If so, use the result of foo.__getattr__('bar').
If everything failed, raise AttributeError.
bar.__get__ is not really a function - it's a "method-wrapper" - but you can imagine it being implemented vaguely like this:
# Somewhere in the Python internals
class __method_wrapper(object):
def __init__(self, func):
self.func = func
def __call__(self, obj, cls):
return lambda *args, **kwargs: func(obj, *args, **kwargs)
# Except it actually returns a "bound method" object
# that uses cls for its __repr__
# and there is a __repr__ for the method_wrapper that I *think*
# uses the hashcode of the underlying function, rather than of itself,
# but I'm not sure.
# Automatically done after compiling bar
bar.__get__ = __method_wrapper(bar)
The "binding" that happens within the __get__ automatically attached to bar (called a descriptor), by the way, is more or less the reason why you have to specify self parameters explicitly for Python methods. In Javascript, this itself is magical; in Python, it is merely the process of binding things to self that is magical. ;)
And yes, you can explicitly set a __get__ method on your own objects and have it do special things when you set a class attribute to an instance of the object and then access it from an instance of that other class. Python is extremely reflective. :) But if you want to learn how to do that, and get a really full understanding of the situation, you have a lot of reading to do. ;)