Trying to understand super and __new__
Here goes my code:
class Base(object):
def __new__(cls,foo):
if cls is Base:
if foo == 1:
# return Base.__new__(Child) complains not enough arguments
return Base.__new__(Child,foo)
if foo == 2:
# how does this work without giving foo?
return super(Base,cls).__new__(Child)
else:
return super(Base,cls).__new__(cls,foo)
def __init__(self,foo):
pass
class Child(Base):
def __init__(self,foo):
Base.__init__(self,foo)
a = Base(1) # returns instance of class Child
b = Base(2) # returns instance of class Child
c = Base(3) # returns instance of class Base
d = Child(1) # returns instance of class Child
Why doesn't super.__new__ need an argument while __new__ needs it?
Python: 2.7.11
super().__new__ is not the same function as Base.__new__. super().__new__ is object.__new__. object.__new__ doesn't require a foo argument, but Base.__new__ does.
>>> Base.__new__
<function Base.__new__ at 0x000002243340A730>
>>> super(Base, Base).__new__
<built-in method __new__ of type object at 0x00007FF87AD89EC0>
>>> object.__new__
<built-in method __new__ of type object at 0x00007FF87AD89EC0>
What may be confusing you is this line:
return super(Base,cls).__new__(cls, foo)
This calls object.__new__(cls, foo). That's right, it passes a foo argument to object.__new__ even though object.__new__ doesn't need it. This is allowed in python 2, but would crash in python 3. It would be best to remove the foo argument from there.
Related
When a class is created deriving from a MagicMock() object it has an unwanted spec='str'. Does anyone know why this happens? Does anyone know any operations that could be done to the MagicMock() object in this case such that it doesn't have the spec='str' or can use methods of the class?
from unittest.mock import MagicMock
a = MagicMock()
class b():
#staticmethod
def x():
return 1
class c(a):
#staticmethod
def x():
return 1
print(a)
print(b)
print(c)
print(a.x())
print(b.x())
print(c.x())
which returns
MagicMock id='140670188364408'>
<class '__main__.b'>
<MagicMock spec='str' id='140670220499320'>
<MagicMock name='mock.x()' id='140670220574848'>
1
Traceback (most recent call last):
File "/xyz/test.py", line 19, in <module>
print(c.x())
File "/xyz/lib/python3.7/unittest/mock.py", line 580, in _getattr_
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'x'
Basically I need the AttributeError to not be here. Is there something I can do to 'a' such that c.x() is valid?
edit - the issue seems to be with _mock_add_spec in mock.py still not sure how to fix this.
In Python, classes are actually instances of the type class. A class statement like this:
class c(a):
#staticmethod
def x():
return 1
is really syntactic sugar of calling type with the name of the class, the base classes and the class members:
c = type('c', (a,), {'x': staticmethod(lambda: 1)})
The above statement would go through the given base classes and call the __new__ method of the type of the first base class with the __new__ method defined, which in this case is a. The return value gets assigned to c to become a new class.
Normally, a would be an actual class--an instance of type or a subclass of type. But in this case, a is not an instance of type, but rather an instance of MagicMock, so MagicMock.__new__, instead of type.__new__, is called with these 3 arguments.
And here lies the problem: MagicMock is not a subclass of type, so its __new__ method is not meant to take the same arguments as type.__new__. And yet, when MagicMock.__new__ is called with these 3 arguments, it takes them without complaint anyway because according to the signature of MagicMock's constructor (which is the same as Mock's):
class unittest.mock.Mock(spec=None, side_effect=None,
return_value=DEFAULT, wraps=None, name=None, spec_set=None,
unsafe=False, **kwargs)
MagicMock.__new__ would assign the 3 positional arguments as spec, side_effect and return_value, respectively. As you now see, the first argument, the class name ('c' in this case), an instance of str, becomes spec, which is why your class c becomes an instance of MagicMock with a spec of str.
The solution
Luckily, a magic method named __mro_entries__ was introduced since Python 3.7 that can solve this problem by providing a non-class base class with a substitute base class:
from unittest.mock import MagicMock
class SubclassableMagicMock(MagicMock):
def __mro_entries__(self, bases):
return self.__class__,
so that:
a = SubclassableMagicMock()
class b():
#staticmethod
def x():
return 1
class c(a):
#staticmethod
def x():
return 1
print(a)
print(b)
print(c)
print(a.x())
print(b.x())
print(c.x())
outputs:
<SubclassableMagicMock id='140127365021408'>
<class '__main__.b'>
<class '__main__.c'>
<SubclassableMagicMock name='mock.x()' id='140127351680080'>
1
1
Demo: https://replit.com/#blhsing/HotAcademicCases
Interesting...
The issue you're facing is because the MagicMock object has a "spec" attribute, which only allows methods that are part of the specified object. In this case, the "spec" is set to "str", so only methods of the "str" class are allowed.
To resolve this, you can either create the MagicMock object without a "spec" attribute, or set the "spec" attribute to "None", which will allow all methods:
from unittest.mock import MagicMock
a = MagicMock(spec=None) # or a = MagicMock()
class b():
#staticmethod
def x():
return 1
class c(a):
#staticmethod
def x():
return 1
print(a)
print(b)
print(c)
print(a.x())
print(b.x())
print(c.x())
This should allow the "c.x()" method to be called without raising an "AttributeError". Also be careful of misconfigured mocks!
I am in the process of learning Python 3 and just ran into the getattr function. From what I can tell, it is invoked when the attribute call is not found in the class definition as a function or a variable.
In order to understand the behaviour, I wrote the following test class (based on what I've read):
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm is 'test':
return lambda x: "%s%s" % (x.foo, x.bar)
raise AttributeError(itm)
And I then initate my object and call the non-existent function test which, expectedly, returns the reference to the function:
t = Test("Foo", "Bar")
print(t.test)
<function Test.__getattr__.<locals>.<lambda> at 0x01A138E8>
However, if I call the function, the result is not the expected "FooBar", but an error:
print(t.test())
TypeError: <lambda>() missing 1 required positional argument: 'x'
In order to get my expected results, I need to call the function with the same object as the first parameter, like this:
print(t.test(t))
FooBar
I find this behaviour rather strange, as when calling p.some_function(), is said to add p as the first argument.
I would be grateful if someone could shine some light over this headache of mine. I am using PyDev in Eclipse.
__getattr__ return values are "raw", they don't behave like class attributes, invoking the descriptor protocol that plain methods involve that causes the creation of bound methods (where self is passed implicitly). To bind the function as a method, you need to perform the binding manually:
import types
...
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# Explicitly bind function to self
return types.MethodType(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
types.MethodType is poorly documented (the interactive help is more helpful), but basically, you pass it a user-defined function and an instance of a class and it returns a bound method that, when called, implicitly passes that instance as the first positional argument (the self argument).
Note that in your specific case, you could just rely on closure scope to make a zero-argument function continue to work:
def __getattr__(self, itm):
if itm is 'test': # Note: This should really be == 'test', not is 'test'
# No binding, but referring to self captures it in closure scope
return lambda: "%s%s" % (self.foo, self.bar)
raise AttributeError(itm)
Now it's not a bound method at all, just a function that happens to have captured self from the scope in which it was defined (the __getattr__ call). Which solution is best depends on your needs; creating a bound method is slightly slower, but gets a true bound method, while relying on closure scope is (trivially, ~10ns out of >400ns) faster, but returns a plain function (which may be a problem if, for example, it's passed as a callback to code that assumes it's a bound method and can have __self__ and __func__ extracted separately for instance).
To get what you want, you need a lambda that doesn't take arguments:
return lambda: "%s%s" % (self.foo, self.bar)
But you should really use a property for this, instead.
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test)
# FooBar
Note the lack of parentheses.
If you're absolutely determined that it must be a function, do this:
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
#property
def test(self):
return lambda: "{}{}".format(self.foo, self.bar)
t = Test("Foo", "Bar")
print(t.test())
# FooBar
You need to create something that behaves like a bound method, you could simply use functools.partial to bind the instance to the function:
from functools import partial
class Test(object):
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def __getattr__(self, itm):
if itm == 'test': # you shouldn't use "is" for comparisons!
return partial(lambda x: "%s%s" % (x.foo, x.bar), self)
raise AttributeError(itm)
The test:
t = Test("Foo", "Bar")
print(t.test)
# functools.partial(<function Test.__getattr__.<locals>.<lambda> at 0x0000020C70CA6510>, <__main__.Test object at 0x0000020C7217F8D0>)
print(t.test())
# FooBar
"I find this behaviour rather strange, as when calling
p.some_function(), is said to add p as the first argument."
some_function is actually a method, which is why it gets passed an instance implicitly when the method is "bound to an object." But plain functions don't work that way, only functions defined in the class body have this magic applied to them automagically. And actually, unbound methods (accessed via the class directly) function the same as normal functions! The terminology "bound and unbound" methods no longer applies, because in Python 3 we only have methods and functions (getting rid of the distinction between unbound methods and plain functions). When an instance is instantiated, accessing the attribute returns a method which implicitly calls the instance on invocation.
>>> class A:
... def method(self, x):
... return x
...
>>> a.method
<bound method A.method of <__main__.A object at 0x101a5b3c8>>
>>> type(a.method)
<class 'method'>
However, if you access the attribute of the class you'll see it's just a function:
>>> A.method
<function A.method at 0x101a64950>
>>> type(A.method)
<class 'function'>
>>> a = A()
Now, observe:
>>> bound = a.method
>>> bound(42)
42
>>> unbound = A.method
>>> unbound(42)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() missing 1 required positional argument: 'x'
But this is the magic of classes. Note, you can even add functions to classes dynamically, and they get magically turned into methods when you invoke them on an instance:
>>> A.method2 = lambda self, x: x*2
>>> a2 = A()
>>> a2.method2(4)
8
And, as one would hope, the behavior still applies to objects already created!
>>> a.method2(2)
4
Note, this doesn't work if you dynamically add to an instance:
>>> a.method3 = lambda self, x: x*3
>>> a.method3(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: <lambda>() missing 1 required positional argument: 'x'
You have to do the magic yourself:
>>> from types import MethodType
>>> a.method4 = MethodType((lambda self, x: x*4), a)
>>> a.method4(4)
16
>>>
Notice that if you do print(t.__getattr__) you get something like <bound method Test.__getattr__ of <__main__.Test object at 0x00000123FBAE4DA0>>. The key point is that methods defined on an object are said to be 'bound' and so always take the object as the first parameter. Your lambda function is just an anonymous function not 'bound' to anything, so for it to access the object it needs to be explicitly passed in.
I presume you are only doing this to experiment with using `__getattr__', as what you are doing could be much more easily achieved by making your lambda a method on the object.
Part A
I want to do some checking on arguments to a class instantiation and possibly return None if it doesn't make sense to even create the object.
I've read the docs but I don't understand what to return in this case.
class MyClass:
def __new__(cls, Param):
if Param == 5:
return None
else:
# What should 'X' be?
return X
What should X be in return X?
It cannot be self because the object doesn't exist yet so self is not a valid keyword in this context.
Part B
Tied to my question, I don't understand the need to have the cls parameter.
If you call the constructor of MyClass - var = MyClass(1) - won't cls always be MyClass?
How could it be anything else?
According to the docs, cls in object.__new__(cls[, ...]) is:
. . .the class of which an instance was requested as its first
argument.
(I'm assuming you are using Python 3 because you provided a link to Python 3 docs)
X could be super().__new__(cls).
super() returns the parent class (in this case it is simply object). Most of the times when you are overriding methods you will need to call the parent class's method at some point.
See this example:
class MyClass:
def __new__(cls, param):
if param == 5:
return None
else:
return super().__new__(cls)
def __init__(self, param):
self.param = param
And then:
a = MyClass(1)
print(a)
print(a.param)
>> <__main__.MyClass object at 0x00000000038964A8>
1
b = MyClass(5)
print(b)
print(b.param)
>> None
Traceback (most recent call last):
File "main.py", line 37, in <module>
print(b.param)
AttributeError: 'NoneType' object has no attribute 'param'
You could just return the instance of cls like this return object.__ new__(cls). Because every class is subclass of object, you can use that as a object creator for your class. The returnes object is passed as a first argument to the __init__() with the any number of positional argument or any number of keyword argument you passed to new. There you will create instance variable assigning those values.
I just realized that all user defined functions have a __get__ that allows the function to operate as descriptors when the functions are used in a class. This __get__ returns a <bound method object> when invoked in the context of an instance (myinstance.method) and returns the original function when invoked in the context of class (MyClass.method).
I was trying to get methods to behave like attributes (in the same way as with #property but without the side effect of data descriptor, i.e non-overridable from instances). I succeeded by creating a non-data descriptor that just invokes the original method on __get__ but when I discover that functions are already descriptors I tried to change the function's __get__ to do the invocation directly instead of returning a <bound method ...> but without luck.
Here is my try:
class A(object):
def m(self):
self.m = 20 # to show that .m is overridable at instance level
return 10
def alternative__get(self, instance, owner):
if instance is None:
return self
else:
return self.__func__(instance)
print(A.__dict__['m'].__get__) # => <method-wrapper '__get__' of function object ...>
A.__dict__['m'].__get__ = alternative__get
print(A.__dict__['m']) # <function m ..>
print(A.__dict__['m'].__get__) # <function alternative__get ...>
print(A.m) # <unbound method A.m>
a = A()
print(a.m) # <bound method A.m of ...>
print(a.m) # <bound method A.m of ...>
This doesn have the desired effect a.m still resolves to <bound method...> instead of returning directly the result of invoking A.m(a).
My current approach is to define a descriptor:
class attribute(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, instance, owner):
if instance is None:
return self
else:
return self.fget(instance)
class A(object):
def __init__(self):
pass
#attribute
def m(self):
self.m = 20;
return 10
a = A()
print(a.m) # => 10
# a.m = 30
print(a.m) # => 20 because it's overriden at instance level
This approach works but I would like to know if it's possible to change A.m.__get__ to achieve the same effect, or why it can't work.
You can't do this by setting A.m.__get__, because the Python language internals skip the instance dict when looking up special methods like __get__. (This is so, for example, a class Foo that defines a __repr__ method uses type.__repr__ instead of Foo.__repr__ when you do repr(Foo).)
I currently have this code:
class Generator(object):
def __getattr__(self, name):
def f(self):
return ("Result of"+name, self)
f.__name__ = name
return f
def foo(self):
pass
g = Generator()
print g.foo
print Generator.foo
print g.bar
print Generator.bar
Which gives:
<bound method Generator.foo of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.foo>
<function bar at 0x00A9DE70>
AttributeError: type object 'Generator' has no attribute 'bar'
What do I have to do to make it give:
<bound method Generator.foo of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.foo>
<bound method Generator.bar of <__main__.Generator object at 0x00B62D30>>
<unbound method Generator.bar>
Here's a metaclass that adds the __getattr__ function from the class definition back to the metaclass itself. This avoids having to define the function in multiple places, or as a separate global function defined beforehand and added individually to the metaclass and class.
class Meta(type):
def __new__(mcls, name, bases, dikt):
fgetattr = dikt.get('__getattr__')
if fgetattr is not None:
setattr(mcls, '__getattr__', fgetattr)
return super(Meta, mcls).__new__(mcls, name, bases, dikt)
class Generator(object):
__metaclass__ = Meta
def __getattr__(obj, name):
def f(self):
return "Result of %s for %r" % (name, self)
f.__name__ = name
if isinstance(obj, type):
setattr(obj, name, f)
else:
setattr(type(obj), name, f)
return getattr(obj, name)
Rather than directly create the method via the dynamic function's __get__ descriptor method, I think it's better to store the function in the class dict and rely on getattr to return the proper bound/unbound method. Subsequent attribute access will use the function from the class. Since the same __getattr__ function is used for both the class and the instance, an isinstance check is required to ensure the dynamic function gets stored to the class and not the instance.
In Python 3, getting the function as an attribute of the class merely returns the function since unbound methods were removed from the language. Also, the metaclass syntax has changed to a keyword argument in the class definition line.
Test:
>>> g = Generator()
>>> g.foo
<bound method Generator.foo of <__main__.Generator object at 0xb7248c2c>>
>>> Generator.foo
<unbound method Generator.foo>
>>> g.bar
<bound method Generator.bar of <__main__.Generator object at 0xb7248c2c>>
>>> Generator.bar
<unbound method Generator.bar>
>>> g.foo()
'Result of foo for <__main__.Generator object at 0xb7248c2c>'
>>> Generator.foo(g)
'Result of foo for <__main__.Generator object at 0xb7248c2c>'
>>> 'foo' in vars(Generator), 'bar' in vars(Generator)
(True, True)
__getattr__() only works on instances, you will need to make the function on a metaclass on Generator to get the behavior your want.
You have two problems.
You only set the dynamic function on the instance g (not the class Generator), so Generator.bar is not defined.
You don't wrap it in a MethodType, so you get a function instead of a method.
For 1, if you always call g.foo before you call Generator.foo, you can just add the line
setattr(self.__class__, name, f)
inside __getattr__, which will bind the name as a method on the class. Otherwise, you will need a custom __getattr__ on the type object, which means that you have to make it an instance of a custom class i.e. write your own metaclass.
For 2, see #thg435's answer. Note that this is icky because of backwards compatibility in Python 2, and has been neatened considerably in Py3k -- now what you are doing would basically work. It's because of the automatic injection of self.
For the first case (g.bar), replace return f with return MethodType(f, self).