My hope is to make attributes case-insensitive. But overwriting __getattr__ and __setattr__ are somewhat different, as indicated by the following toy example:
class A(object):
x = 10
def __getattr__(self, attribute):
return getattr(self, attribute.lower())
## following alternatives don't work ##
# def __getattr__(self, attribute):
# return self.__getattr__(attribute.lower())
# def __getattr__(self, attribute):
# return super().__getattr__(attribute.lower())
def __setattr__(self, attribute, value):
return super().__setattr__(attribute.lower(), value)
## following alternative doesn't work ##
# def __setattr__(self, attribute, value):
# return setattr(self, attribute.lower(), value)
a = A()
print(a.x) ## output is 10
a.X = 2
print(a.X) ## output is 2
I am confused by two points.
I assume getattr() is a syntactic sugar for __getattr__, but they behave differently.
Why does __setattr__ need to call super(), while __getattr__ doesn't?
I assume getattr() is a syntactic sugar for __getattr__, but they behave differently.
That's because the assumption is incorrect. getattr() goes through the entire attribute lookup process, of which __getattr__ is only a part.
Attribute lookup first invokes a different hook, namely the __getattribute__ method, which by default performs the familiar search through the instance dict and class hierarchy. __getattr__ will be called only if the attribute hasn't been found by __getattribute__. From the __getattr__ documentation:
Called when the default attribute access fails with an AttributeError (either __getattribute__() raises an AttributeError because name is not an instance attribute or an attribute in the class tree for self; or __get__() of a name property raises AttributeError).
In other words, __getattr__ is an extra hook to access attributes that don't exist, and would otherwise raise AttributeError.
Also, functions like getattr() or len() are not syntactic sugar for a dunder method. They almost always do more work, with the dunder method a hook for that process to call. Sometimes there are multiple hooks involved, such as here, or when creating an instance of a class by calling the class. Sometimes the connection is fairly direct, such as in len(), but even in the simple cases there are additional checks being made that the hook itself is not responsible for.
Why does __setattr__ need to call super(), while __getattr__ doesn't?
__getattr__ is an optional hook. There is no default implementation, which is why super().__getattr__() doesn't work. __setattr__ is not optional, so object provides you with a default implementation.
Note that by using getattr() you created an infinite loop! instance.non_existing will call __getattribute__('non_existing') and then __getattr__('non_existing'), at which point you use getattr(..., 'non_existing') which calls __getattribute__() and then __getattr__, etc.
In this case, you should override __getattribute__ instead:
class A(object):
x = 10
def __getattribute__(self, attribute):
return super().__getattribute__(attribute.lower())
def __setattr__(self, attribute, value):
return super().__setattr__(attribute.lower(), value)
Related
When using __new__ to customize the creation of a metaclass, we can pass attributes to the type().__new__ method which will be set on the object before it is returned, e.g.
class Foo(type):
def __new__(cls, name, bases, attrs):
attrs['customAttr'] = 'someVal'
return type.__new__(cls, name, bases, attrs)
So that:
>> Foo.__dict__
{'customeAttr': 'someVal', ...}
However I don't know how to do the same for a normal (non-meta) class, which causes a problem when using __setattr__:
class Bar(object):
def __new__(cls, someVal):
obj = object().__new__(cls) # cant pass custom attrs
obj.customAttr = someVal # obj is already a Bar and invokes __setattr__
return obj
def __setattr__(*args): raise Exception('read-only class')
So that unfortunately:
>>> Bar(42)
...
Exception: read-only class
In the __new__ of Bar I get back a fully fledged class instance from object() and any attribute access goes through normal lookup rules, in this case invoking __setattr__. Metaclass Foo avoids this as type() will set attributes before returning the instance during low-level creation whereas object() will not.
Is there a way of passing attributes to object() or is another another type I can use as the instance returned from __new__ that does allow attributes to be set before it becomes a full class instance? I am not interesting in solutions like setting __class__ after instance creation.
You have to explictly bypass your own class's __setattr__ by calling the super or root object __setattr__. So you'd change:
obj.customAttr = someVal
to:
object.__setattr__(obj, 'customAttr', someVal)
A less general approach (doesn't apply to __slots__ based classes) is to directly assign to __dict__ using dict operations:
obj.__dict__['customAttr'] = someVal # Equivalently: vars(obj)['customAttr'] = someVal
The first approach is what the newly __slots__-ed uuid.UUID now uses; before it became __slots__-ed, it used the second approach. In both cases this was needed because they used the same __setattr__ trick to make the type as immutable as possible (without going to the trouble of subclassing tuple, a la typing.NamedTuple).
How can I override the metaclass of a Python class, with a unittest.mock.MagicMock instance instead?
I have a function whose job involves working with the metaclass of an argument:
# lorem.py
class Foo(object):
pass
def quux(existing_class):
…
metaclass = type(existing_class)
new_class = metaclass(…)
The unit tests for this function will need to assert that the calls to
the metaclass go as expected, without actually calling a real class
object.
Note: The test case does not care about the metaclass's behaviour; it cares that quux retrieves that metaclass (using type(existing_class)) and calls the metaclass with the correct arguments.
So to write a unit test for this function, I want to pass a class object whose metaclass is a mock object instead. This will allow, for example, making assertions about how the metaclass was called, and ensuring no unwanted side effects.
# test_lorem.py
import unittest
import unittest.mock
import lorem
class stub_metaclass(type):
def __new__(metaclass, name, bases, namespace):
return super().__new__(metaclass, name, bases, namespace)
class quux_TestCase(unittest.TestCase):
#unittest.mock.patch.object(
lorem.Foo, '__class__', side_effect=stub_metaclass)
def test_calls_expected_metaclass_with_class_name(
self,
mock_foo_metaclass,
):
expected_name = 'Foo'
expected_bases = …
expected_namespace = …
lorem.quux(lorem.Foo)
mock_foo_metaclass.assert_called_with(
expected_name, expected_bases, expected_namespace)
When I try to mock the __class__ attribute of an existing class, though, I get this error:
File "/usr/lib/python3/dist-packages/mock/mock.py", line 1500, in start
result = self.__enter__()
File "/usr/lib/python3/dist-packages/mock/mock.py", line 1460, in __enter__
setattr(self.target, self.attribute, new_attr)
TypeError: __class__ must be set to a class, not 'MagicMock' object
This is telling me that unittest.mock.patch is attempting to set the __class__ attribute temporarily to a MagicMock instance, as I want; but Python is refusing that with a TypeError.
But placing a mock object as the metaclass is exactly what I'm trying to do: put a unittest.mock.MagicMock instance in the __class__ attribute in order that the mock object will do all that it does: record calls, pretend valid behaviour, etc.
How can I set a mock object in place of the Foo class's __class__ attribute, in order to instrument Foo and test that my code uses Foo's metaclass correctly?
You can't do exactly what you want. As you can see an object's __class__ attribute is very special in Python, and even for ordinary instances there are checks in runtime to verify it is assigned to a proper type.
When you get down to a class's __class__, that is even more strict.
Possible approach:
One thing to do in there is not pass a class to your test - but an object that is an instance from a crafted ordinary class, which will have an artificial __class__ attribute. Even them, you will have to change your code from calling type(existing_class) to do existing_class.__class__ directly. For an instance object to "falsify" its __class__ anyway, you have to implement __class__ as a property on its class (or override __getattribute__; (the class itself will report its true metaclass, but an instance can return whatever is coded on the __class__ property.
class Foo:
#property
def __class__(self):
return stub_metaclass
Actual suggestion:
But then, since you are at it, maybe the simplest thing is to mock type instead on the target module where quux is defined.
class MockType:
def __init__(self):
self.mock = mock.Mock()
def __call__(self, *args):
return self.mock
...
class ...:
...
def test_calls_expected_metaclass_with_class_name(
self,
):
try:
new_type = MockType()
# This creates "type" on the module "lorem" namespace
# as a global variable. It will then override the built-in "type"
lorem.type = new_type
lorem.quux(lorem.Foo)
finally:
del lorem.type # un-shadows the built-in type on the module
new_type.mock.assert_called_with(
'Foo', unittest.mock.ANY, unittest.mock.ANY)
Still another approach
Another thing that can be done is to craft a full "MockMetaclass" in the "old fashion": without unittest.magicmock at all, instead, with intrumented __new__ and other relevant methods that will record the called parameters, and function as a true metaclass for a class you pass in as parameter.
Considerations on what is being done
People reaching here, please note that one should not test the class creation (and metaclass) mechanisms themselves. One can just assume the Python runtime have these working and tested already.
I have one object wrapped inside another.
The "Wrapper" accesses the attributes from the "Wrapped" object by overriding __getattr__.
This works well until I need to override an atribute on a sub class, and then access the attribute from the base class using super().
I can still access the attribute directly from __getattr__ but why does super() not work?
class Wrapped(object):
def __init__(self, value):
self.value = value
def hello_world(self):
print 'hello world', self.value
class Wrapper(object):
def __init__(self, obj):
self.wrapped_obj = obj
def __getattr__(self, name):
if name in self.__dict__:
return getattr(self, name)
else:
return getattr(self.wrapped_obj, name)
class Subclass(Wrapper):
def __init__(self, obj):
super(Subclass, self).__init__(obj)
def hello_world(self):
# this works
func = super(Subclass, self).__getattr__('hello_world')()
# this doesn't
super(Subclass, self).hello_world()
a = Wrapped(2)
b = Subclass(a)
b.hello_world()
According to this, super does not allow implicit calls of "hook" functions such as __getattr__. I'm not sure why it is implemented this way (there's probably a good reason and things are already confusing enough since the super object has custom __getattribute__ and __get__ methods as it is), but it seems like it's just the way things are.
Edit: This post appears to clear things up a little. It looks like the problem is the extra layer of indirection caused by __getattribute__ is ignored when calling functions implicitly. Doing foo.x is equivalent to
foo.__getattr__(x)
(Assuming no __getattribute__ method is defined and x is not in foo.__dict__)
However, it is NOT equivalent to
foo.__getattribute__('__getattr__')(x)
Since super returns a proxy object, it has an extra layer of indirection which causes things to fail.
P.S. The self.__dict__ check in your __getattr__ function is completely unnecessary. __getattr__ is only called if the attribute doesn't already exist in your dict. (Use __getattribute__ if you want it to always be called, but then you have to be very careful, because even something simple like if name in self.__dict__ will cause infinite recursion.
Can someone explain me this behavior as to why 1) doesn't work while 2) and 3) works
1)
class bm(object):
def __init__(self,val):
self.a=val
def get(self):
return self.a
def set(self,val):
self.a=val
a=property(get,set)
In [43]: ob1=bm('vin')
gives me Recursive error
,while the below code works fine
2)
class bm(object):
def __init__(self,val):
self._a=val
def get(self):
return self._a
def set(self,val):
self._a=val
a=property(get,set)
In [43]: ob1=bm('vin')
Works fine.I can access ob.a and do ob.a=''
Even this works fine
3)
class bm(object):
def __init__(self,val):
self.a=val
def get(self):
return self._a
def set(self,val):
self._a=val
a=property(get,set)
In [43]: ob1=bm('vin')
Works fine.I can access ob.a and do ob.a=''
In the first example, you're creating the property a, which lets you do things like:
self.a # Get the value
self.a = some_obj # Set the value
But within the a property, you're referring to the a property again, via self.a! This will create a problem with recursion.
In the next examples, the property a is backed by the variable self._a, avoiding this recursion issue.
The point here is that in Python, everything (including functions, methods, "properties" - or any other descriptor - classes and even modules) is an object, and that there's no distinct namespaces for "data" and "functions or methods". IOW, in Python, an object has attributes, period - no "member data" nor "member functions". Even the base classes are attributes (and are themselves objects so they have attributes too).
The attribute lookup rules are (quite simplified - I won't mention some special cases like slots etc):
for reading:
first look for an attribute by that name in the parent classes. If
found AND this attribute implements the "get" part of the descriptor protocol, call
that attribute's __get__ method.
then look for an instance attribute in the instance's __dict__
then if the class (or one of the parent classes) has a __getattr__ method, call it
then raise an AttributeError exception
for setting:
first look for an attribute by that name in the parent classes. If
found AND this attribute implements the "set" part of the descriptor protocol, call
that attribute's __set__ method.
then store the attribute in the instance's __dict__
I mentionned but did not explain the descriptor protocol. This protocol (if you come from Java, a protocol is kind of an "implied" interface - you don't have to declare it, just to implement it) says that if an object has a __get__ method and eventually a __set__ method (here again I overly simplify by not mentionning the __del__ part) AND is a class attribute then when looked up on an instance of the class it's __get__ method will be called (with the instance and class as arguments) on "read" lookup and it's __set__ method will be called (with the instance and value) on "write".
IOW, the descriptor protocol is the basis for computed attributes in Python.
Now about the property type (yes,it is a class, not a function): it does implement the descriptor protocol, in a very simple way. Here's a simplified version of how the property type would be implemented in pure Python (not taking the __del__ part into account):
class property(object):
def __init__(self, fget, fset=None, doc=None):
self.fget = fget
self.fset = fset
self.doc = doc
def __get__(self, instance, cls=None):
if not instance:
return self
return self.fget(instance)
def __set__(self, instance, value):
if not self.fset:
raise AttributeError("attribute is read-only")
self.fset(instance, value)
Back to the question - given the class statement :
class Something(object):
def __init__(self,val):
self.a=val
def get(self):
return self.a
def set(self,val):
self.a=val
a=property(get,set)
We have a class object Something with a (class) attribute a which is a property with fget=Something.get and fset=Something.set. Now what happens when we instanciate Something ? The initializer is called with a val argument, and try to bind self.a to that argument. The attribute lookup rule (for "writing" - we should really say 'binding') kicks in and notice that the Something class object has an attribute a which - as an instance of the property type - implements the __set__ part of the protocol descriptor. So the lookup rule calls Something.a.__set__(theinstance, val), which resolves to Something.a.fset(theinstance, val), which is implemented as self.a = val. New attribute lookup, finds a class attribute a implementing the binding part of the descriptor protocol, invoke it, etc..., bang, infinite recursion.
To make a long story short: an attribute is an attribute is an attribute ;)
Note that in your third example, the set method try to set self._a and not self.a. Since the class has no descriptor named _a, this just create an instance attribute by that name, so no recursion here.
For more on the descriptor protocol, cf
- http://docs.python.org/reference/datamodel.html#implementing-descriptors
- http://wiki.python.org/moin/ComputedAttributesUsingPropertyObjects
and if you want to understand what Python "methods" really are (hint: the function type implement the descriptor protocol) and why the 'self' argument is required, you can read this too:
- http://wiki.python.org/moin/FromFunctionToMethod
the next is my code:
class foo:
def __init__(self):
self.a = "a"
def __getattr__(self,x,defalut):
if x in self:
return x
else:return defalut
a=foo()
print getattr(a,'b','sss')
i know the __getattr__ must be 2 argument,but i want to get a default attribute if the attribute is no being.
how can i get it, thanks
and
i found if defined __setattr__,my next code is also can't run
class foo:
def __init__(self):
self.a={}
def __setattr__(self,name,value):
self.a[name]=value
a=foo()#error ,why
hi alex,
i changed your example:
class foo(object):
def __init__(self):
self.a = {'a': 'boh'}
def __getattr__(self, x):
if x in self.a:
return self.a[x]
raise AttributeError
a=foo()
print getattr(a,'a','sss')
it print {'a': 'boh'},not 'boh'
i think it will print self.a not self.a['a'], This is obviously not want to see
why ,and Is there any way to avoid it
Your problem number one: you're defining an old-style class (we know you're on Python 2.something, even though you don't tell us, because you're using print as a keyword;-). In Python 2:
class foo:
means you're defining an old-style, aka legacy, class, whose behavior can be rather quirky at times. Never do that -- there's no good reason! The old-style classes exist only for compatibility with old legacy code that relies on their quirks (and were finally abolished in Python 3). Use new style classes instead:
class foo(object):
and then the check if x in self: will not cause a recursive __getattr__ call. It will however cause a failure anyway, because your class does not define a __contains__ method and therefore you cannot check if x is contained in an instance of that class.
If what you're trying to do is whether x is defined in the instance dict of self, don't bother: __getattr__ doesn't even get called in that case -- it's only called when the attribute is not otherwise found in self.
To support three-arguments calls to the getattr built-in, just raise AttributeError in your __getattr__ method if necessary (just as would happen if you had no __getattr__ method at all), and the built-in will do its job (it's the built-in's job to intercept such cases and return the default if provided). That's the reason one never ever calls special methods such as __getattr__ directly but rather uses built-ins and operators which internally call them -- the built-ins and operators provide substantial added value.
So to give an example which makes somewhat sense:
class foo(object):
def __init__(self):
self.blah = {'a': 'boh'}
def __getattr__(self, x):
if x in self.blah:
return self.blah[x]
raise AttributeError
a=foo()
print getattr(a,'b','sss')
This prints sss, as desired.
If you add a __setattr__ method, that one intercepts every attempt to set attributes on self -- including self.blah = whatever. So -- when you need to bypass the very __setattr__ you're defining -- you must use a different approach. For example:
class foo(object):
def __init__(self):
self.__dict__['blah'] = {}
def __setattr__(self, name, value):
self.blah[name] = value
def __getattr__(self, x):
if x in self.blah:
return self.blah[x]
raise AttributeError
a=foo()
print getattr(a,'b','sss')
This also prints sss. Instead of
self.__dict__['blah'] = {}
you could also use
object.__setattr__(self, 'blah', {})
Such "upcalls to the superclass's implementation" (which you could also obtain via the super built-in) are one of the rare exceptions to the rules "don't call special methods directly, call the built-in or use the operator instead" -- here, you want to specifically bypass the normal behavior, so the explicit special-method call is a possibility.
You are confusing the getattr built-in function, which retrieves some attribute binding of an object dynamically (by name), at runtime, and the __getattr__ method, which is invoked when you access some missing attribute of an object.
You can't ask
if x in self:
from within __getattr__, because the in operator will cause __getattr__ to be invoked, leading to infinite recursion.
If you simply want to have undefined attributes all be defined as some value, then
def __getattr__(self, ignored):
return "Bob Dobbs"