With the file super5.py:
class A:
def m(self):
print("m of A called")
class B(A):
def m(self):
print("m of B called")
super().m()
class C(A):
def m(self):
print("m of C called")
super().m()
class D(B,C):
def m(self):
print("m of D called")
super().m()
we can do the following:
>>> from super5 import D
>>> x = D()
>>> x.m()
m of D called
m of B called
m of C called
m of A called
To me, this doesn't make sense, because when I execute x.m(), I expect the following to happen:
The first line of m of D is executed and thus "m of D called" is output.
The second line, super().m() is executed, which first takes us to m of B.
In m of B, "m of B called" is first output, and then, m of A is executed due to the super.m() call in m of B, and "m of A called" is output.
m of C is executed in a fashion analogous to 3.
As you can see, what I expect to see is:
m of D called
m of B called
m of A called
m of C called
m of A called
Why am I wrong? Is python somehow keeping track of the number of super() calls to a particular superclass and limiting the execution to 1?
No, Python keep a track of all super classes in a special __mro__ attribute (Method Resolution Order in new-style classes):
print(D.__mro__)
You get:
(<class 'D'>, <class 'B'>, <class 'C'>, <class 'A'>, <class 'object'>)
So, when you call super, it follow this list in order.
See this question: What does mro() do?.
Everything is explained in the official document in the chapter "Multiple Inheritance".
For most purposes, in the simplest cases, you can think of the search for attributes inherited from a parent class as depth-first, left-to-right, not searching twice in the same class where there is an overlap in the hierarchy. Thus, if an attribute is not found in DerivedClassName, it is searched for in Base1, then (recursively) in the base classes of Base1, and if it was not found there, it was searched for in Base2, and so on.
In fact, it is slightly more complex than that; the method resolution order changes dynamically to support cooperative calls to super(). This approach is known in some other multiple-inheritance languages as call-next-method and is more powerful than the super call found in single-inheritance languages.
Dynamic ordering is necessary because all cases of multiple inheritance exhibit one or more diamond relationships (where at least one of the parent classes can be accessed through multiple paths from the bottommost class). For example, all classes inherit from object, so any case of multiple inheritance provides more than one path to reach object. To keep the base classes from being accessed more than once, the dynamic algorithm linearizes the search order in a way that preserves the left-to-right ordering specified in each class, that calls each parent only once, and that is monotonic (meaning that a class can be subclassed without affecting the precedence order of its parents). Taken together, these properties make it possible to design reliable and extensible classes with multiple inheritance.
Related
In a python program, I implemented some class A, and a subclass B which extends some of A's methods and implements new ones. So it's something that looks like this.
class A:
def method(self):
print("This is A's method.")
class B(A):
def method(self):
super().method()
print("This is B's method.")
def another_method(self):
pass
Later, I wanted to use an object which would have access to all of B's methods, except for a small change in method: this object would have to first call A.method, and then do other things, different from what I added in B.method. I thought it was natural to introduce a class C which would inherit from B but modify method, so I defined C like this.
class C(B):
def method(self):
super(B, self).method()
print("This is C's method.")
Everything seems to work as I expect. However, I stumbled across this question and this one, which both address similar problems as what I described here. In both posts, someone quickly added a comment to say that calling a grandparent method in a child when the parent has overridden this method is a sign that there is something wrong with the inheritance design.
How should I have coded this? What would be a better design? Of course this is a toy example and I guess the answer might depend on the actual classes I defined. To give a more complete picture, let's say the method method from the example represents a single method in my actual program, but the method another_method represents many different methods.
There is a famous Python example
class A(object):
def go(self):
print("go A go!")
class B(A):
def go(self):
super(B, self).go()
print("go B go!")
class C(A):
def go(self):
super(C, self).go()
print("go C go!")
class D(B,C):
def go(self):
super(D, self).go()
print("go D go!")
d = D()
d.go()
#go A go!
#go C go!
#go B go!
#go D go!
I have several questions. The first one is B calls A and C calls A so I expect A to appear twice. The second question is about the order.
Since Python 2.3, method resolution has used an algorithm called C3 Linearization (borrowed from Dylan). Wikipedia has a nice article on it.
As the name implies, the idea is to force the method resolution graph to be a straight line, even if the inheritance graph isn't. Which means A is not going to appear twice, by design.
Why? Well, for one thing, it completely avoids the "diamond problem" that plagues multiple inheritance in many other languages. (Or maybe it's more accurate to say that many other languages either ban MI or restrict it to pure "interfaces" because they didn't have a solution to the problem as it exists in C++.)
The original Python explanation—including the motivation behind it—is available in The Python 2.3 Method Resolution Order. This is a bit technical, but worth reading if you're interested.
You might also want to read the original Dylan paper, which goes into more detail about what's wrong with non-linear MRO graphs, and the challenges of coming up with a linearization that's monotonic (i.e., it goes in the order you expect, or at least the order you expect once you get over the fact that it's linear), and so on.
And if you want a deeper understanding of how type() works under the covers, or just want to see what's changed between 2.3 and 3.7 (e.g., the way __mro__ gets created and updated—although magic 3.x super is elsewhere), there's really no better place than the CPython source.
The class super does not just recover the superclass. It instantiate an object which recovers methods in the context of a given method resolution order. Every class has a mro that you can access through the __mro__ attribute.
D.__mro__ # (D, B, C, A, object)
So when given a class and an instance, super first recovers the mro from that instance. When you try to recover an attribute from the super object, it returns it from the first class following the provided class that has such an attribute.
If you were to implement the behaviour of super in Python, it would look something like this.
class super:
def __init__(self, cls, instance):
if not isinstance(cls, type):
raise TypeError('super() argument 1 must be type')
if isinstance(instance, cls):
self.mro = type(instance).__mro__
elif isinstance(instance, type) and issubclass(instance, cls):
self.mro = instance.__mro__
else:
raise TypeError('super(type, obj): obj must be an instance or subtype of type')
self.cls = cls
self.instance = instance
def __getattr__(self, attr):
cls_index = self.mro.index(self.cls)
for supercls in self.mro[cls_index + 1:]:
if hasattr(supercls, attr): break
# The actual implementation binds instances to methods before returning
return getattr(supercls, attr)
So back to your example, when you call super(B, self).go, it recovers the __mro__ of self, which is of type D. It then picks go from the first class following B in the mro that has such an attribute.
So in this case since self.__mro__ is (D, B, C, A, object), the first class following B that has the attribute go is C and not A.
If you want details on how Python determines the mro, then I suggest abarnert's answer.
There are a lot of great resources on super(), including this great blog post that pops up a lot, as well as many questions on Stack Overflow. However I feel like they all stop short of explaining how it works in the most general case (with arbitrary inheritance graphs), as well as what is going on under the hood.
Consider this basic example of diamond inheritance:
class A(object):
def foo(self):
print 'A foo'
class B(A):
def foo(self):
print 'B foo before'
super(B, self).foo()
print 'B foo after'
class C(A):
def foo(self):
print 'C foo before'
super(C, self).foo()
print 'C foo after'
class D(B, C):
def foo(self):
print 'D foo before'
super(D, self).foo()
print 'D foo after'
If you read up on Python's rules for method resolution order from sources like this or look up the wikipedia page for C3 linearization, you will see that the MRO must be (D, B, C, A, object). This is of course confirmed by D.__mro__:
(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <type 'object'>)
And
d = D()
d.foo()
prints
D foo before
B foo before
C foo before
A foo
C foo after
B foo after
D foo after
which matches the MRO. However, consider that above super(B, self).foo() in B actually calls C.foo, whereas in b = B(); b.foo() it would simply go straight to A.foo. Clearly using super(B, self).foo() is not simply a shortcut for A.foo(self) as is sometimes taught.
super() is then obviously aware of the previous calls before it and the overall MRO the chain is trying to follow. I can see two ways this might be accomplished. The first is to do something like passing the super object itself as the self argument to the next method in the chain, which would act like the original object but also contain this information. However this also seems like it would break a lot of things (super(D, d) is d is false) and by doing a little experimenting I can see this isn't the case.
The other option is to have some sort of global context that stores the MRO and the current position in it. I imagine the algorithm for super goes something like:
Is there currently a context we are working in? If not, create one which contains a queue. Get the MRO for the class argument, push all elements except for the first into the queue.
Pop the next element from the current context's MRO queue, use it as the current class when constructing the super instance.
When a method is accessed from the super instance, look it up in the current class and call it using the same context.
However, this doesn't account for weird things like using a different base class as the first argument to a call to super, or even calling a different method on it. I would like to know the general algorithm for this. Also, if this context exists somewhere, can I inspect it? Can I muck with it? Terrible idea of course, but Python typically expects you to be a mature adult even if you're not.
This also introduces a lot of design considerations. If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it! If I want my class to be easily extensible I will need to account for this, but I am not sure if it is more complicated than simply making sure all versions of foo have identical signatures. There is also the question of when to put code before or after the call to super, even if it does not make any difference considering B's base classes only.
super() is then obviously aware of the previous calls before it
It's not. When you do super(B, self).foo, super knows the MRO because that's just type(self).__mro__, and it knows that it should start looking for foo at the point in the MRO immediately after B. A rough pure-Python equivalent would be
class super(object):
def __init__(self, klass, obj):
self.klass = klass
self.obj = obj
def __getattr__(self, attrname):
classes = iter(type(self.obj).__mro__)
# search the MRO to find self.klass
for klass in classes:
if klass is self.klass:
break
# start searching for attrname at the next class after self.klass
for klass in classes:
if attrname in klass.__dict__:
attr = klass.__dict__[attrname]
break
else:
raise AttributeError
# handle methods and other descriptors
try:
return attr.__get__(self.obj, type(self.obj))
except AttributeError:
return attr
If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it!
There's no expectation that you should be able to multiple-inherit from arbitrary classes. Unless foo is specifically designed to be overloaded by sibling classes in a multiple-inheritance situation, D should not exist.
I am little confused about python multiple inheritance.
For example if you had:
class A(object):
def __init__(self):
print "init A"
super(A, self).__init__()
class B(A):
def __init__(self):
print "init B"
super(B, self).__init__()
class C(A):
def __init__(self):
print "init C"
super(C, self).__init__()
class D(C, B):
def __init__(self):
print "init D"
super(D, self).__init__()
if __name__ == '__main__':
D()
The method resolution order (MRO) would be D-C-B-A.
Why the order is not D-C-A-B-A?
Python docs:
With new-style classes, dynamic ordering is necessary because all cases of multiple inheritance exhibit one or more diamond relationships (where at least one of the parent classes can be accessed through multiple paths from the bottommost class). For example, all new-style classes inherit from object, so any case of multiple inheritance provides more than one path to reach object. To keep the base classes from being accessed more than once, the dynamic algorithm linearizes the search order in a way that preserves the left-to-right ordering specified in each class, that calls each parent only once, and that is monotonic (meaning that a class can be subclassed without affecting the precedence order of its parents). Taken together, these properties make it possible to design reliable and extensible classes with multiple inheritance. For more detail, see https://www.python.org/download/releases/2.3/mro/.
Roughly speaking, the MRO is given by taking all of the bases going depth-first, left to right, and then removing any duplicates, simply leaving the last.
The algorithm is such that for any of the base classes, the order of its base classes in the mro will always be the same (if this is not possible I believe you get an error).
If duplicates were not removed, the order would actually be D-C-A-object-B-A-object. This is confusing for a number of reasons. Firstly, if methods call super, as in your case, then the same method can be called twice. Secondly, it is reasonable for the implementation of B to expect to be ahead of A and object in the method resolution order. If it overrides properties of A, it is not expected that this overriding will be reversed.
The reason the MRO is D-C-B-A is that, it being D-C-A-B-A (or D-C-A-B) would have strange effects. In this particular example, the A class constructor takes no arguments and explicitly does the super() call that only matters in diamond inheritance trees. If it didn't do the super call, then the B constructor wouldn't be called from the A constructor. If the constructors took arguments then the A constructor would be stuck between a rock and a hard place. If it didn't call super then it wouldn't work in diamond inheritances. If it did call super then, since object.__init__ takes no arguments, the constructor would fail if it wasn't used in a diamond inheritance pattern.
You instantiate only one object. Python allows each ancestor's __init__ to execute only once in the MRO. Thus, we don't get both the A->C and A->B relationships executing A.__init__
With only that much explained, you might think we'd get D-C-A-B ... but since B also requires A.__init__, the MRO is patient enough to resolve the entire graph before determining the execution order.
Below is two versions of my code:
Non-working one
class A:
def __init__(self):
print "I am A "
class B:
def __init__(self):
print "I am B "
class C(A, B):
def __init__(self):
super(C, self).__init__()
c = C()
This raises exception:
super(C, self).__init__()
TypeError: must be type, not classobj
Working version
class A(object):
def __init__(self):
print "I am A "
class B:
def __init__(self):
print "I am B "
class C(A, B):
def __init__(self):
super(C, self).__init__()
c = C()
If one of the parent class has inherited object explicitly , there is no exception and things works as desired. Any explanation, why?
With the above working one , it prints "I am A" but not "I am B" which means it initializes only Class A and not Class B. HOw to initialize multiple parent classes in children class?
That's because super only works with new-style classes (that inherit from object). In Python 2, classes where object is nowhere in the inheritance hierarchy (like your first example) are called old-style classes, and should never be used anywhere.
It's an old historical artifact from when Python first got OO added to it, and the developers got it wrong. Among other things, new-style classes allow for the use of super, the descriptor protocol (properties), and makes several fixes to the way multiple inheritance is handled. super is one of them, but the most important is the way method resolution order is computed for diamond inheritance cases (A inherits from B and C which both inherit from D). Read more about it here: http://python-history.blogspot.fr/2010/06/method-resolution-order.html
Note that Python 3 ditched old-style classes entirely, and thus in Python 3 all classes are new-style classes and inherit from object, regardless of whether or not they do it explicitly.
super() only works for new-style classes(Any class which inherits from object).So You couldn't pass class object to super.
There are two typical use cases for super. In a class hierarchy with single inheritance, super can be used to refer to parent classes without naming them explicitly, thus making the code more maintainable. This use closely parallels the use of super in other programming languages.
The second use case is to support cooperative multiple inheritance in a dynamic execution environment. This use case is unique to Python and is not found in statically compiled languages or languages that only support single inheritance. This makes it possible to implement “diamond diagrams” where multiple base classes implement the same method. Good design dictates that this method have the same calling signature in every case (because the order of calls is determined at runtime, because that order adapts to changes in the class hierarchy, and because that order can include sibling classes that are unknown prior to runtime).
a typical superclass call looks like this:
class C(B):
def method(self, arg):
super(C, self).method(arg)