I have the following class that tries to define a custom __getattribute__:
class Item:
def __init__(self):
self.x = 1
def __dir__(self):
return ['x']
def __getattribute__(self, i):
if i not in dir(self):
return "NOT DEFINED!"
else:
return super().__getattribute__(a)
And then I can run it fine:
>>> i=Item()
>>> i.x
1
>>> i.a
'NOT DEFINED!'
However, if I change this line:
if i not in dir(self):
To:
if i not in self.__dir__():
I get a RecursionError as it seems to be calling itself. Why is this so, and why does dir() and differently than __dir__()?
The __getattribute__ method is called whenever the dot notation is used on an object, so self.__dir__() would call the __getattribute__ method of self with '__dir__' as the argument, which, with your code, would then call self.__dir__() again, which calls the __getattribute__ method again with the same argument, resulting in endless recursions.
Related
Assume A is the parent class of B and b is an instance of B. Then an overriden method of A can be called with super: super(B, b).method().
The docs state "str(object) returns object.__str__()" in its basic invocation.
It should follow that str(super(B, b)) == super(B, b).__str__(), but that's not the case (interactive version):
class A:
def __str__(self):
return "A"
class B(A):
def __str__(self):
return "B"
b = B()
b_super = super(B, b)
print(str(b_super)) # "<super: <class 'B'>, <B object>>"
print(b_super.__str__()) # "A"
So where did I go wrong? Does the super mechanism not work for magic methods? Does str not invoke __str__ in this case? Is it related to this paragraph:
Note that super() is implemented as part of the binding process for explicit dotted attribute lookups such as super().__getitem__(name). It does so by implementing its own __getattribute__() method for searching classes in a predictable order that supports cooperative multiple inheritance. Accordingly, super() is undefined for implicit lookups using statements or operators such as super()[name].
str() doesn't look up the __str__ method through the normal attribute lookup procedure. Instead, it performs a direct search for the __str__ method in the __dict__s of its argument's class hierarchy, in MRO order. This finds super.__str__, which gives "<super: <class 'B'>, <B object>>".
However, when you look up b_super.__str__ manually, that goes through super.__getattribute__, the hook super uses to provide its special attribute lookup behavior. The lookup through __getattribute__ will resolve to A.__str__ and call that.
Consider this class, which illustrates the difference (I hope):
class B(object):
def __init__(self, other):
self.other = other
def __getattribute__(self, name):
if name == 'other':
return object.__getattribute__(self, 'other')
elif name == '__str__':
return getattr(self.other, name)
else:
return name
def __str__(self):
return 'fun'
>>> str(B(1)) # calls B.__str__ because it doesn't invoke __getattribute__
'fun'
>>> B(1).__str__() # calls B.__getattribute__ to look up the __str__ method which returns (1).__str__
'1'
The problem in this case and likewise for super is that these are proxies that rely on __getattribute__ to forward it. So any function or method that doesn't go through __getattribute__ doesn't forward. And str() is such a function.
Just for completeness because it was mentioned in the comments and the other answer.
But str(x) isn't equivalent to type(x).__str__(x) because str() even avoids the normal attribute lookup procedure of the "function on the class". It only checks the tp_str (or if that's NULL the tp_repr) slot of the class. So it doesn't even invoke __getattribute__ of the metaclass, which type(x).__str__(x) would do:
class A(type):
def __getattribute__(self, name):
print(name)
if name == '__str__':
return lambda self: 'A'
else:
return type.__getattribute__(self, name)
class B(metaclass=A):
def __str__(self):
return 'B'
>>> b = B()
>>> str(b)
'B'
>>> type(b).__str__(b)
__str__
'A'
However in the absense of a metaclass it might be helpful to think of str(x) as equivalent to type(x).__str__(x). But while (potentially) helpful it's not correct.
The docs are wrong.
str(x) is actually equivalent to type(x).__str__(x).
If you print(type(b_super).__str__(b_super)), you get the obvious result.
(even this might be oversimplified in the case of weird metaclasses)
I'm curious about the #property annotation in Python 3.
I know I can manipulate state between calls to a property like this ...
class Obj:
_x = 0
#property
def x(self):
self._x += 1
return self._x
obj = Obj()
print(obj.x)
print(obj.x)
... which prints:
1
2
However, is it possible keep this mechanism after it has been passed to a function?
Take the following function:
def f(x):
print(x)
print(x)
Say we cannot change it, meaning we can't simply pass the object to the function and use the properties directly. Is it possible to pass it only the property, or something to that effect, such that each time x is "used" in the function x increments? Is there maybe a way to do this by manipulating and passing the class or object itself? Something similar to __call__(self, ...), or adding #property to the class itself or the __init__(self, ...) of the class?
I've tried the following naive approaches ...
obj = Obj()
f(obj.x)
... and ...
obj = Obj()
f(getattr(obj, 'x'))
Both print ...
1
1
... which makes sense seeing as it is evaluated before it is passed to f. I also tried to extend Obj:
class Obj:
_x = 0
#property
def x(self):
self._x += 1
return self._x
def y(self):
return x
... and then go ...
obj = Obj()
f(obj.y)
... but it produces ...
<bound method Obj.y of <__main__.Obj object at 0x000002379E015748>>
<bound method Obj.y of <__main__.Obj object at 0x000002379E015748>>
This also makes sense seeing as we're simply getting the method itself and passing it to the f function.
Just a note: This isn't a real world problem/example. I'm simply trying to understand the principles and limits of python itself.
The property() wrapper is a descriptor, meaning that it is only invoked by dotted access, like a.x, and not by plain variable access.
To trigger a method invocation or function call with just a variable reference, I think it would be necessary to exec() code in a custom locals namespace:
>>> def add_ten(x):
return x + 10
>>> class NS(dict):
def __getitem__(self, key):
value = dict.__getitem__(self, key)
if key == 'x':
return add_ten(value)
return value
>>> exec('print(x + 2)', globals(), NS(x=5))
17
In the above example, just a reference to the variable x is enough to trigger a call to add_ten().
This is likely more trouble than its worth, but as you say, you just wanted to see what is possible :-)
I am in the process of learning Python 3 and just ran into the getattr function. From what I can tell, it is invoked when the attribute call is not found in the class definition as a function or a variable.
In order to understand the behaviour, I wrote the following test class (based on what I've read):
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm is 'test':
return lambda x: "%s%s" % (x.foo, x.bar)
raise AttributeError(itm)
And I then initate my object and call the non-existent function test which, expectedly, returns the reference to the function:
t = Test("Foo", "Bar")
print(t.test)
<function Test.__getattr__.<locals>.<lambda> at 0x01A138E8>
However, if I call the function, the result is not the expected "FooBar", but an error:
print(t.test())
TypeError: <lambda>() missing 1 required positional argument: 'x'
In order to get my expected results, I need to call the function with the same object as the first parameter, like this:
print(t.test(t))
FooBar
I find this behaviour rather strange, as when calling p.some_function(), is said to add p as the first argument.
I would be grateful if someone could shine some light over this headache of mine. I am using PyDev in Eclipse.
__getattr__ return values are "raw", they don't behave like class attributes, invoking the descriptor protocol that plain methods involve that causes the creation of bound methods (where self is passed implicitly). To bind the function as a method, you need to perform the binding manually:
import types
...
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# Explicitly bind function to self
return types.MethodType(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
types.MethodType is poorly documented (the interactive help is more helpful), but basically, you pass it a user-defined function and an instance of a class and it returns a bound method that, when called, implicitly passes that instance as the first positional argument (the self argument).
Note that in your specific case, you could just rely on closure scope to make a zero-argument function continue to work:
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# No binding, but referring to self captures it in closure scope
return lambda: "%s%s" % (self.foo, self.bar)
raise AttributeError(itm)
Now it's not a bound method at all, just a function that happens to have captured self from the scope in which it was defined (the __getattr__ call). Which solution is best depends on your needs; creating a bound method is slightly slower, but gets a true bound method, while relying on closure scope is (trivially, ~10ns out of >400ns) faster, but returns a plain function (which may be a problem if, for example, it's passed as a callback to code that assumes it's a bound method and can have __self__ and __func__ extracted separately for instance).
To get what you want, you need a lambda that doesn't take arguments:
return lambda: "%s%s" % (self.foo, self.bar)
But you should really use a property for this, instead.
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test)
# FooBar
Note the lack of parentheses.
If you're absolutely determined that it must be a function, do this:
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return lambda: "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test())
# FooBar
You need to create something that behaves like a bound method, you could simply use functools.partial to bind the instance to the function:
from functools import partial
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm == 'test': # you shouldn't use "is" for comparisons!
return partial(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
The test:
t = Test("Foo", "Bar")
print(t.test)
# functools.partial(<function Test.__getattr__.<locals>.<lambda> at 0x0000020C70CA6510>, <__main__.Test object at 0x0000020C7217F8D0>)
print(t.test())
# FooBar
"I find this behaviour rather strange, as when calling
p.some_function(), is said to add p as the first argument."
some_function is actually a method, which is why it gets passed an instance implicitly when the method is "bound to an object." But plain functions don't work that way, only functions defined in the class body have this magic applied to them automagically. And actually, unbound methods (accessed via the class directly) function the same as normal functions! The terminology "bound and unbound" methods no longer applies, because in Python 3 we only have methods and functions (getting rid of the distinction between unbound methods and plain functions). When an instance is instantiated, accessing the attribute returns a method which implicitly calls the instance on invocation.
>>> class A:
... def method(self, x):
... return x
...
>>> a.method
<bound method A.method of <__main__.A object at 0x101a5b3c8>>
>>> type(a.method)
<class 'method'>
However, if you access the attribute of the class you'll see it's just a function:
>>> A.method
<function A.method at 0x101a64950>
>>> type(A.method)
<class 'function'>
>>> a = A()
Now, observe:
>>> bound = a.method
>>> bound(42)
42
>>> unbound = A.method
>>> unbound(42)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() missing 1 required positional argument: 'x'
But this is the magic of classes. Note, you can even add functions to classes dynamically, and they get magically turned into methods when you invoke them on an instance:
>>> A.method2 = lambda self, x: x*2
>>> a2 = A()
>>> a2.method2(4)
8
And, as one would hope, the behavior still applies to objects already created!
>>> a.method2(2)
4
Note, this doesn't work if you dynamically add to an instance:
>>> a.method3 = lambda self, x: x*3
>>> a.method3(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: <lambda>() missing 1 required positional argument: 'x'
You have to do the magic yourself:
>>> from types import MethodType
>>> a.method4 = MethodType((lambda self, x: x*4), a)
>>> a.method4(4)
16
>>>
Notice that if you do print(t.__getattr__) you get something like <bound method Test.__getattr__ of <__main__.Test object at 0x00000123FBAE4DA0>>. The key point is that methods defined on an object are said to be 'bound' and so always take the object as the first parameter. Your lambda function is just an anonymous function not 'bound' to anything, so for it to access the object it needs to be explicitly passed in.
I presume you are only doing this to experiment with using `__getattr__', as what you are doing could be much more easily achieved by making your lambda a method on the object.
Is it possible to have a method and call it as either function or property?
def Foo:
def bar(self,x=1,y=2):
return x+y
foo=Foo()
foo.bar #should return 3
foo.bar(4,5) #should return 9
It seems to be impossible, because:
foo.bar will call __getattribute__; if it's a descriptior, __get__ will be called from within __getattribute__, meaning that bar() is evaluated before it's even returned to the caller
if an attribute is a function, it only returns a function and then () is applied to that function
because of that, it's impossible to detect in __getattribute__ if caller is calling a property or function
hence, dual behaviour is not possible
What I hopped to implement is something like (pseudocode):
def __getattribute__():
if len(args)>1: return function
else: return property
But because args are not passed to __getattribute__ or __get__, I don't know where and how to switch between property and function.
Just use:
foo.bar() # 3
foo.bar(4, 5) # 9
If you do insist, here is a very ugly, unusable solution, that wil actually return 3 and 9. Works on python 3.6. Don't use it:
class CallableInt(int):
"""But, why????"""
def __call__(self, x, y):
return x + y
class Foo:
#property
def bar(self):
return CallableInt(3)
foo = Foo()
print(foo.bar)
print(foo.bar(4, 5))
I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:
class ShowMeList(object):
def __init__(self, it):
self._data = list(it)
def __getattr__(self, name):
attr = object.__getattribute__(self._data, name)
if callable(attr):
def wrapper(*a, **kw):
print "before the call"
result = attr(*a, **kw)
print "after the call"
return result
return wrapper
return attr
If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
If I don't inherit from object (first line class ShowMeList:) everything works as expected:
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
How do I accomplish this intercept with new style classes?
For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.
class Wrapper(object):
"""Wrapper class that provides proxy access to some internal instance."""
__wraps__ = None
__ignore__ = "class mro new init setattr getattr getattribute"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
# provide proxy access to regular attributes of wrapped object
def __getattr__(self, name):
return getattr(self._obj, name)
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Usage:
class DictWrapper(Wrapper):
__wraps__ = dict
wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))
# make sure it worked....
assert "b" in wrapped_dict # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3) # __eq__
assert "'a': 1" in str(wrapped_dict) # __str__
assert wrapped_dict.__doc__.startswith("dict()") # __doc__
Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.
Consider the following:
>>> class C:
x = 1
def __init__(self):
self.y = 2
def __getattr__(self, attr):
print(attr)
>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z
The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).
On your example, the __repr__ and many other magic methods are already defined in the object class.
One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.
As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.
Cut and copy from the documentation:
For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.