I have a class as follows:
class Lz:
def __init__(self, b):
self.b = b
def __getattr__(self, item):
return self.b.__getattribute__(item)
And I create an instance and print :
a = Lz('abc')
print(a)
Result is: abc
I have set a breakpoint at line return self.b.__getattribute__(item), item show __str__
I don't know why it calls __getattr__, and item is __str__ when I access the instance.
print calls __str__ (see this question for details), but as Lz does not have a __str__ method, a lookup for an attribute named '__str__' takes place using __getattr__.
So if you add a __str__ method, __getattr__ should not be called anymore when printing objects of the Lz class.
print(obj) invokes str(obj) (to get a printable representation), which in turn tries to invokes obj.__str__() (and fallback to something else if this fails, but that's not the point here).
You defined Lz as an old-style class, so it's doesn't by default have a __str__ method (new-style classes inherit this from object), but you defined a __getattr__() method, so this is what gets invoked in the end (__getattr__() is the last thing the attribute lookup will invoke when everything else has failed).
NB: in case you don't already know, since everything in Python is an object - include classes, functions, methods etc - Python doesn't make difference between "data" attributes and "method" attributes - those are all attributes, period.
NB2: directly accessing __magic__ names is considered bad practice. Those names are implementation support for operators or operator-like generic functions (ie len(), type etc), and you are supposed to use the operator or generic function instead. IOW, this:
return self.b.__getattribute__(item)
should be written as
return getattr(self.b, item)
(getattr() is the generic function version of the "dot" attribute lookup operator (.))
Related
Assume that we are using python 3.x or newer, not python 2.x
Python, like many languages, has a dot-operator:
# Create a new instance of the Rectangle class
robby = Rectangle(3, 10)
# INVOKE THE DOT OPERATOR
x = robby.length
Python's dot-operator is sometimes implemented as __getattribute__.
The following is equivalent to x = robby.length:
x = Rectangle.__getattribute__(robby, "length")
However, the dot-operator is not always implemented as __getattribute__.
Python has "magic methods"
A magic methods is any method whose name begins and ends with two underscore characters.
__len__() is an example of a magic method.
You can get a list of most of python's magic methods by executing the following code:
print("\n".join(filter(lambda s: s.startswith("__"), dir(int))))
The output is:
__abs__
__add__
__and__
__bool__
__ceil__
__class__
__delattr__
__dir__
__divmod__
__doc__
__eq__
__float__
[... truncated / abridged ...]
__rtruediv__
__rxor__
__setattr__
__sizeof__
__str__
__sub__
__subclasshook__
__truediv__
__trunc__
__xor__
Suppose we write a class named Rectangle, sub-class of object.
Then my attempts to override object.__getattribute__ inside of the Rectangle class, usually fail.
The following shows an example of a class where python sometimes ignores an overridden dot-operator:
class Klass:
def __getattribute__(self, attr_name):
return print
obj = Klass()
obj.append() # WORKS FINE. `obj.append == print`
obj.delete() # WORKS FINE. `obj.delete == print`
obj.length # WORKS FINE
obj.x # WORKS FINE
# None of the following work, because they
# invoke magic methods.
# The following line is similar to:
# x = Klass.__len__(obj)
len(obj)
# obj + 5
# is similar to:
# x = Klass.__add__(obj, 5)
x = obj + 5
# The following line is similair to:
# x = Klass.__radd__(obj, 2)
x = 2 + obj
There is more than one way to override python's dot operator.
What is an example of one way to do it which is readable, clean, and consistent?
By consistent, I mean that our custom dot operator gets called whenever . is used in source code, no matter whether the method is a magic method or not.
I am unwilling to manually type-in every single magic method under the sun.
I don't want to see thousands of lines of code which looks like:
def __len__(*args, **kwargs):
return getattr(args[0], "__len__")(*args, **kwargs)
I understand the distinction between __getattr__ and __getattribute__
Overriding __getattribute__ instead of __getattr__ is not the issue at hand.
__getattribute__ already does what you're literally asking for - overriding __getattribute__ is all you need to handle all uses of the . operator. (Strictly speaking, Python will fall back to __getattr__ if __getattribute__ fails, but you don't have to worry about that as long as you don't implement __getattr__.)
You say you want your operator to be called "whenever . is used in source code", but len, +, and all the other things you're worried about don't use .. There is no . in len(obj), obj + 5, or 2 + obj.
Most magic method lookups don't use attribute access. If you actually look up yourobj.__len__ or yourobj.__add__, that will go through attribute access, and your __getattribute__ will be invoked, but when Python looks up a magic method to implement language functionality, it does a direct search of the object's type's MRO. The . operator is not involved.
There is no way to override magic method lookup. That's a hardcoded process with no override hook. The closest thing you can do is override individual magic methods to delegate to __getattribute__, but that's not the same thing as overriding magic method lookup (or overriding .), and it's easy to get infinite recursion bugs that way.
If all you really want to do is avoid repetitive individual magic method overrides, you could put them in a class decorator or mixin.
In the rare cases that an object's __getattribute__ method must be overridden, a common mistake is to try and return the attribute in question like so:
class WrongWay:
def __getattribute__(self, name):
# Custom functionality goes here
return self.__dict__[name]
This code always yields a RuntimeError due to a recursive loop, since self.__dict__ is in itself an attribute reference that calls upon the same __getattribute__ method.
According to this answer, the correct solution to this problem is to replace last line with:
...
return super().__getattribute__(self, name) # Defer responsibility to the superclass
This solution works when run through the Python 3 interpreter, but it also seems to violate __getattribute__'s promised functionality. Even if the superclass chain is traversed up to object, at the end of the line somebody will eventually have to return self.something, and by definition that attribute reference must first get through the child's __getattribute__ method.
How does Python get around this recursion issue? In object.__getattribute__, how is anything returned without looping into another request?
at the end of the line somebody will eventually have to return self.something, and by definition that attribute reference must first get through the child's __getattribute__() method.
That's not correct. object.__getattribute__ is not defined as returning self.anything, and it does not respect descendant class implementations of __getattribute__. object.__getattribute__ is the default attribute access implementation, and it always performs its job through the default attribute access mechanism.
Similarly, object.__eq__ is not defined as returning self == other_thing, and it does not respect descendant class implementations of __eq__. object.__str__ is not defined as returning str(self), and it does not respect descendant class implementations of __str__. object's methods are the default implementations of those methods, and they always do the default thing.
Why are python instance methods callable, but static methods and class methods not callable?
I did the following:
class Test():
class_var = 42
#classmethod
def class_method(cls):
pass
#staticmethod
def static_method():
pass
def instance_method(self):
pass
for attr, val in vars(Test).items():
if not attr.startswith("__"):
print (attr, "is %s callable" % ("" if callable(val) else "NOT"))
The result is:
static_method is NOT callable
instance_method is callable
class_method is NOT callable
class_var is NOT callable
Technically this may be because instance method object might have a particular attribute (not) set in a particular way (possibly __call__). Why such asymmetry, or what purpose does it serve?
I came across this while learning python inspection tools.
Additional remarks from comments:
The SO answer linked in the comments says that the static/class methods are descriptors , which are not callable. Now I am curious, why are descriptors made not callable, since descriptors are class with particular attributes (one of __get__, __set__, __del___) defined.
Why are descriptors not callable? Basically because they don't need to be. Not every descriptor represents a callable either.
As you correctly note, the descriptor protocol consists of __get__, __set__ and __del__. Note no __call__, that's the technical reason why it's not callable. The actual callable is the return value of your static_method.__get__(...).
As for the philosophical reason, let's look at the class. The contents of the __dict__, or in your case results of vars(), are basically locals() of the class block. If you define a function, it gets dumped as a plain function. If you use a decorator, such as #staticmethod, it's equivalent to something like:
def _this_is_not_stored_anywhere():
pass
static_method = staticmethod(_this_is_not_stored_anywhere)
I.e., static_method is assigned a return value of the staticmethod() function.
Now, function objects actually implement the descriptor protocol - every function has a __get__ method on it. This is where the special self and the bound-method behavior comes from. See:
def xyz(what):
print(what)
repr(xyz) # '<function xyz at 0x7f8f924bdea0>'
repr(xyz.__get__("hello")) # "<bound method str.xyz of 'hello'>"
xyz.__get__("hello")() # "hello"
Because of how the class calls __get__, your test.instance_method binds to the instance and gets it pre-filled as it first argument.
But the whole point of #classmethod and #staticmethod is that they do something special to avoid the default "bound method" behavior! So they can't return a plain function. Instead they return a descriptor object with a custom __get__ implementation.
Of course, you could put a __call__ method on this descriptor object, but why? It's code that you don't need in practice; you can almost never touch the descriptor object itself. If you do (in code similar to yours), you still need special handling for descriptors, because a general descriptor doesn't have to be(have like a) callable - properties are descriptors too. So you don't want __call__ in the descriptor protocol. So if a third party "forgets" to implement __call__ on something you consider a "callable", your code will miss it.
Also, the object is a descriptor, not a function. Putting a __call__ method on it would be masking its true nature :) I mean, it's not wrong per se, it's just ... something that you should never need for anything.
BTW, in case of classmethod/staticmethod, you can get back the original function from their __func__ attribute.
In Python when I call the hasattr on a #property decorator the hasattr function actually runs the #property code block.
E.g. a class:
class GooglePlusUser(object):
def __init__(self, master):
self.master = master
def get_user_id(self):
return self.master.google_plus_service.people().get(userId='me').execute()['id']
#property
def profile(self):
# this runs with hasattr
return self.master.google_plus_service.people().get(userId='me').execute()
Running the following code calls the profile property and actually makes the call:
#Check if the call is an attribute
if not hasattr(google_plus_user, call):
self.response.out.write('Unknown call')
return
Why? How can I solve this without making the api call?
hasattr() works by actually retrieving the attribute; if an exception is thrown hasattr() returns False. That's because that is the only reliable way of knowing if an attribute exists, since there are so many dynamic ways to inject attributes on Python objects (__getattr__, __getattribute__, property objects, meta classes, etc.).
From the hasattr() documentation:
This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.
If you don't want a property to be invoked when doing this, then don't use hasattr. Use vars() (which returns the instance dictionary) or dir() (which gives you a list of names on the class as well). This won't let you discover dynamic attributes handled by __getattr__ or __getattribute__ hooks however.
hasattr is basically implemented like this (except in C):
def hasattr(obj, attrname):
try:
getattr(obj, attname)
except AttributeError:
return False
return True
So in true "easier to ask for forgiveness than permission" (EAFP) fashion, to find out if an object has a given attribute, Python simply tries to get the attribute, and converts failure to a return value of False. Since it's really getting the attribute in the success case, hasattr() can trigger code for property and other descriptors.
To check for an attribute without triggering descriptors, you can write your own hasattr that traverses the object's method resolution order and checks to see whether the name is in each class's __dict__ (or __slots__). Since this isn't attribute access, it won't trigger properties.
Conveniently, Python already has a way to walk the method resolution order and gather the names of attributes from an instance's classes: dir(). A simple way to write such a method, then, would be:
# gingerly test whether an attribute exists, avoiding triggering descriptor code
def gentle_hasattr(obj, name):
return name in dir(obj) or hasattr(obj, name)
Note that we fall back to using hasattr() if we can't find the desired name in dir(), because dir() won't find dynamic attributes (i.e., where __getattr__ is overridden). Code for these will still be triggered, of course, so if you don't care that you don't find them, you could omit the or clause.
On balance, this is wasteful, since it gets all relevant attribute names when we're interested only in whether a specific one exists, but it'll do in a pinch. I'm not even sure that doing the loop yourself rather than calling dir() would be faster on average, since it'll be in Python rather than in C.
Making the variable as a class variable and then calling hasattr on it did the trick for me.
if not hasattr(google_plus_user, GooglePlusUser.call):
self.response.out.write('Unknown call')
return
I have a class, and I wish to write a __hash__() method for this class. The method I want to write will return the default hash of the object in some cases, and a hash of one of its attributes in some others. So, as a simple test case:
class Foo:
def __init__(self, bar):
self.bar = bar
def __hash__(self):
if self.bar < 10:
return hash(self) # <- this line doesn't work
else:
return hash(self.bar)
The problem with this is that hash(self) simply calls self.__hash__(), leading to infinite recursion.
I gather that the hash of an object is based on the id() of that object, so I could rewrite return hash(self) as return id(self), or return id(self) / 16, but it seems bad form to me to recreate the default implementation in my own code.
It also occurred to me that I could rewrite it as return object.__hash__(self). This works, but seems even worse, as special methods are not intended to be called directly.
So, what I'm asking is; is there a way to use the default hash of an object without implicitly calling the __hash__() method of the class that object is an instance of?
To call parent implementation use:
super(Foo, self).__hash__()
It also occurred to me that I could rewrite it as return
object.__hash__(self). This works, but seems even worse, as special
methods are not intended to be called directly.
You are overriding a magic method, so it's ok to call parent's implementation directly.