I have two classes that are supposed to implement the same test cases for two independent libraries (let's call them LibA and LibB). So far I define the test methods to be implemented in an abstract base class which ensures that both test classes implement all desired tests:
from abc import ABC, abstractmethod
class MyTests(ABC):
#abstractmethod
def test_foo(self):
pass
class TestsA(MyTests):
def test_foo(self):
pass
class TestsB(MyTests):
def test_foo(self):
pass
This works as expected, but what may still happen is that someone working on LibB accidentally adds a test_bar() method to TestB instead of the base class. The missing test_bar() in the TestA class would go unnoticed in that case.
Is there a way to prohibit the addition of new methods to an (abstract) base class? The objective is to force the addition of new methods to happen in the base class and thus force the implementation of new methods in all derived classes.
Yes. It can be done through a metaclass, or from Python 3.6 onwards, with a check in __init_subclass__ of the baseclass.
__init_sublass__ is a special method called by the language each time a subclass is instantiated. So it can check if the new class have any method that is not present in any of the superclasses and raise a TypeError when the subclass is declared. (__init_subclass__ is converted to a classmethod automatically)
class Base(ABC):
...
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
# By inspecting `cls.__dict__` we pick all methods declared directly on the class
for name, attr in cls.__dict__.items():
attr = getattr(cls, name)
if not callable(attr):
continue
for superclass in cls.__mro__[1:]:
if name in dir(superclass):
break
else:
# method not found in superclasses:
raise TypeError(f"Method {name} defined in {cls.__name__} does not exist in superclasses")
Note that unlike the TypeError raised by non-implemented abstractmethods, this error is raised at class declaration time, not class instantiation time. If the later is desired, you have to use a metaclass and move the check to its __call__ method - however that complicates things, as if one method is created in an intermediate class, that was never instantiated, it won't raise when the method is available in the leaf subclass. I guess what you need is more along the code above.
I have two classes that are supposed to implement the same test cases for two independent libraries (let's call them LibA and LibB). So far I define the test methods to be implemented in an abstract base class which ensures that both test classes implement all desired tests:
from abc import ABC, abstractmethod
class MyTests(ABC):
#abstractmethod
def test_foo(self):
pass
class TestsA(MyTests):
def test_foo(self):
pass
class TestsB(MyTests):
def test_foo(self):
pass
This works as expected, but what may still happen is that someone working on LibB accidentally adds a test_bar() method to TestB instead of the base class. The missing test_bar() in the TestA class would go unnoticed in that case.
Is there a way to prohibit the addition of new methods to an (abstract) base class? The objective is to force the addition of new methods to happen in the base class and thus force the implementation of new methods in all derived classes.
Yes. It can be done through a metaclass, or from Python 3.6 onwards, with a check in __init_subclass__ of the baseclass.
__init_sublass__ is a special method called by the language each time a subclass is instantiated. So it can check if the new class have any method that is not present in any of the superclasses and raise a TypeError when the subclass is declared. (__init_subclass__ is converted to a classmethod automatically)
class Base(ABC):
...
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
# By inspecting `cls.__dict__` we pick all methods declared directly on the class
for name, attr in cls.__dict__.items():
attr = getattr(cls, name)
if not callable(attr):
continue
for superclass in cls.__mro__[1:]:
if name in dir(superclass):
break
else:
# method not found in superclasses:
raise TypeError(f"Method {name} defined in {cls.__name__} does not exist in superclasses")
Note that unlike the TypeError raised by non-implemented abstractmethods, this error is raised at class declaration time, not class instantiation time. If the later is desired, you have to use a metaclass and move the check to its __call__ method - however that complicates things, as if one method is created in an intermediate class, that was never instantiated, it won't raise when the method is available in the leaf subclass. I guess what you need is more along the code above.
In Python, I can create a class method using the #classmethod decorator:
>>> class C:
... #classmethod
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> C.f()
f called with cls=<class '__main__.C'>
Alternatively, I can use a normal (instance) method on a metaclass:
>>> class M(type):
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> class C(metaclass=M):
... pass
...
>>> C.f()
f called with cls=<class '__main__.C'>
As shown by the output of C.f(), these two approaches provide similar functionality.
What are the differences between using #classmethod and using a normal method on a metaclass?
As classes are instances of a metaclass, it is not unexpected that an "instance method" on the metaclass will behave like a classmethod.
However, yes, there are differences - and some of them are more than semantic:
The most important difference is that a method in the metaclass is not "visible" from a class instance. That happens because the attribute lookup in Python (in a simplified way - descriptors may take precedence) search for an attribute in the instance - if it is not present in the instance, Python then looks in that instance's class, and then the search continues on the superclasses of the class, but not on the classes of the class. The Python stdlib make use of this feature in the abc.ABCMeta.register method.
That feature can be used for good, as methods related with the class themselves are free to be re-used as instance attributes without any conflict (but a method would still conflict).
Another difference, though obvious, is that a method declared in the metaclass can be available in several classes, not otherwise related - if you have different class hierarchies, not related at all in what they deal with, but want some common functionality for all classes, you'd have to come up with a mixin class, that would have to be included as base in both hierarchies (say for including all classes in an application registry). (NB. the mixin may sometimes be a better call than a metaclass)
A classmethod is a specialized "classmethod" object, while a method in the metaclass is an ordinary function.
So, it happens that the mechanism that classmethods use is the "descriptor protocol". While normal functions feature a __get__ method that will insert the self argument when they are retrieved from an instance, and leave that argument empty when retrieved from a class, a classmethod object have a different __get__, that will insert the class itself (the "owner") as the first parameter in both situations.
This makes no practical differences most of the time, but if you want access to the method as a function, for purposes of adding dynamically adding decorator to it, or any other, for a method in the metaclass meta.method retrieves the function, ready to be used, while you have to use cls.my_classmethod.__func__ to retrieve it from a classmethod (and then you have to create another classmethod object and assign it back, if you do some wrapping).
Basically, these are the 2 examples:
class M1(type):
def clsmethod1(cls):
pass
class CLS1(metaclass=M1):
pass
def runtime_wrap(cls, method_name, wrapper):
mcls = type(cls)
setattr(mcls, method_name, wrapper(getatttr(mcls, method_name)))
def wrapper(classmethod):
def new_method(cls):
print("wrapper called")
return classmethod(cls)
return new_method
runtime_wrap(cls1, "clsmethod1", wrapper)
class CLS2:
#classmethod
def classmethod2(cls):
pass
def runtime_wrap2(cls, method_name, wrapper):
setattr(cls, method_name, classmethod(
wrapper(getatttr(cls, method_name).__func__)
)
)
runtime_wrap2(cls1, "clsmethod1", wrapper)
In other words: apart from the important difference that a method defined in the metaclass is visible from the instance and a classmethod object do not, the other differences, at runtime will seem obscure and meaningless - but that happens because the language does not need to go out of its way with special rules for classmethods: Both ways of declaring a classmethod are possible, as a consequence from the language design - one, for the fact that a class is itself an object, and another, as a possibility among many, of the use of the descriptor protocol which allows one to specialize attribute access in an instance and in a class:
The classmethod builtin is defined in native code, but it could just be coded in pure python and would work in the exact same way. The 5 line class bellow can be used as a classmethod decorator with no runtime differences to the built-in #classmethod" at all (though distinguishable through introspection such as calls toisinstance, and evenrepr` of course):
class myclassmethod:
def __init__(self, func):
self.__func__ = func
def __get__(self, instance, owner):
return lambda *args, **kw: self.__func__(owner, *args, **kw)
And, beyond methods, it is interesting to keep in mind that specialized attributes such as a #property on the metaclass will work as specialized class attributes, just the same, with no surprising behavior at all.
When you phrase it like you did in the question, the #classmethod and metaclasses may look similar but they have rather different purposes. The class that is injected in the #classmethod's argument is usually used for constructing an instance (i.e. an alternative constructor). On the other hand, the metaclasses are usually used to modify the class itself (e.g. like what Django does with its models DSL).
That is not to say that you can't modify the class inside a classmethod. But then the question becomes why didn't you define the class in the way you want to modify it in the first place? If not, it might suggest a refactor to use multiple classes.
Let's expand the first example a bit.
class C:
#classmethod
def f(cls):
print(f'f called with cls={cls}')
Borrowing from the Python docs, the above will expand to something like the following:
class ClassMethod(object):
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
class C:
def f(cls):
print(f'f called with cls={cls}')
f = ClassMethod(f)
Note how __get__ can take either an instance or the class (or both), and thus you can do both C.f and C().f. This is unlike the metaclass example you give which will throw an AttributeError for C().f.
Moreover, in the metaclass example, f does not exist in C.__dict__. When looking up the attribute f with C.f, the interpreter looks at C.__dict__ and then after failing to find, looks at type(C).__dict__ (which is M.__dict__). This may matter if you want the flexibility to override f in C, although I doubt this will ever be of practical use.
In your example, the difference would be in some other classes that will have M set as their metaclass.
class M(type):
def f(cls):
pass
class C(metaclass=M):
pass
class C2(metaclass=M):
pass
C.f()
C2.f()
class M(type):
pass
class C(metaclass=M):
#classmethod
def f(cls):
pass
class C2(metaclass=M):
pass
C.f()
# C2 does not have 'f'
Here is more on metaclasses
What are some (concrete) use-cases for metaclasses?
Both #classmethod and Metaclass are different.
Everything in python is an object. Every thing means every thing.
What is Metaclass ?
As said every thing is an object. Classes are also objects in fact classes are instances of other mysterious objects formally called as meta-classes. Default metaclass in python is "type" if not specified
By default all classes defined are instances of type.
Classes are instances of Meta-Classes
Few important points are to understand metioned behaviour
As classes are instances of meta classes.
Like every instantiated object, like objects(instances) get their attributes from class. Class will get it's attributes from Meta-Class
Consider Following Code
class Meta(type):
def foo(self):
print(f'foo is called self={self}')
print('{} is instance of {}: {}'.format(self, Meta, isinstance(self, Meta)))
class C(metaclass=Meta):
pass
C.foo()
Where,
class C is instance of class Meta
"class C" is class object which is instance of "class Meta"
Like any other object(instance) "class C" has access it's attributes/methods defined in it's class "class Meta"
So, decoding "C.foo()" . "C" is instance of "Meta" and "foo" is method calling through instance of "Meta" which is "C".
First argument of method "foo" is reference to instance not class unlike "classmethod"
We can verify as if "class C" is instance of "Class Meta
isinstance(C, Meta)
What is classmethod?
Python methods are said to be bound. As python imposes the restriction that method has to be invoked with instance only.
Sometimes we might want to invoke methods directly through class without any instance (much like static members in java) with out having to create any instance.By default instance is required to call method. As a workaround python provides built-in function classmethod to bind given method to class instead of instance.
As class methods are bound to class. It takes at least one argument which is reference to class itself instead of instance (self)
if built-in function/decorator classmethod is used. First argument
will be reference to class instead of instance
class ClassMethodDemo:
#classmethod
def foo(cls):
print(f'cls is ClassMethodDemo: {cls is ClassMethodDemo}')
As we have used "classmethod" we call method "foo" without creating any instance as follows
ClassMethodDemo.foo()
Above method call will return True. Since first argument cls is indeed reference to "ClassMethodDemo"
Summary:
Classmethod's receive first argument which is "a reference to class(traditionally referred as cls) itself"
Methods of meta-classes are not classmethods. Methods of Meta-classes receive first argument which is "a reference to instance(traditionally referred as self) not class"
Why did the Python designers decide that subclasses' __init__() methods don't automatically call the __init__() methods of their superclasses, as in some other languages? Is the Pythonic and recommended idiom really like the following?
class Superclass(object):
def __init__(self):
print 'Do something'
class Subclass(Superclass):
def __init__(self):
super(Subclass, self).__init__()
print 'Do something else'
The crucial distinction between Python's __init__ and those other languages constructors is that __init__ is not a constructor: it's an initializer (the actual constructor (if any, but, see later;-) is __new__ and works completely differently again). While constructing all superclasses (and, no doubt, doing so "before" you continue constructing downwards) is obviously part of saying you're constructing a subclass's instance, that is clearly not the case for initializing, since there are many use cases in which superclasses' initialization needs to be skipped, altered, controlled -- happening, if at all, "in the middle" of the subclass initialization, and so forth.
Basically, super-class delegation of the initializer is not automatic in Python for exactly the same reasons such delegation is also not automatic for any other methods -- and note that those "other languages" don't do automatic super-class delegation for any other method either... just for the constructor (and if applicable, destructor), which, as I mentioned, is not what Python's __init__ is. (Behavior of __new__ is also quite peculiar, though really not directly related to your question, since __new__ is such a peculiar constructor that it doesn't actually necessarily need to construct anything -- could perfectly well return an existing instance, or even a non-instance... clearly Python offers you a lot more control of the mechanics than the "other languages" you have in mind, which also includes having no automatic delegation in __new__ itself!-).
I'm somewhat embarrassed when people parrot the "Zen of Python", as if it's a justification for anything. It's a design philosophy; particular design decisions can always be explained in more specific terms--and they must be, or else the "Zen of Python" becomes an excuse for doing anything.
The reason is simple: you don't necessarily construct a derived class in a way similar at all to how you construct the base class. You may have more parameters, fewer, they may be in a different order or not related at all.
class myFile(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
class readFile(myFile):
def __init__(self, filename):
super(readFile, self).__init__(filename, "r")
class tempFile(myFile):
def __init__(self, mode):
super(tempFile, self).__init__("/tmp/file", mode)
class wordsFile(myFile):
def __init__(self, language):
super(wordsFile, self).__init__("/usr/share/dict/%s" % language, "r")
This applies to all derived methods, not just __init__.
Java and C++ require that a base class constructor is called because of memory layout.
If you have a class BaseClass with a member field1, and you create a new class SubClass that adds a member field2, then an instance of SubClass contains space for field1 and field2. You need a constructor of BaseClass to fill in field1, unless you require all inheriting classes to repeat BaseClass's initialization in their own constructors. And if field1 is private, then inheriting classes can't initialise field1.
Python is not Java or C++. All instances of all user-defined classes have the same 'shape'. They're basically just dictionaries in which attributes can be inserted. Before any initialisation has been done, all instances of all user-defined classes are almost exactly the same; they're just places to store attributes that aren't storing any yet.
So it makes perfect sense for a Python subclass not to call its base class constructor. It could just add the attributes itself if it wanted to. There's no space reserved for a given number of fields for each class in the hierarchy, and there's no difference between an attribute added by code from a BaseClass method and an attribute added by code from a SubClass method.
If, as is common, SubClass actually does want to have all of BaseClass's invariants set up before it goes on to do its own customisation, then yes you can just call BaseClass.__init__() (or use super, but that's complicated and has its own problems sometimes). But you don't have to. And you can do it before, or after, or with different arguments. Hell, if you wanted you could call the BaseClass.__init__ from another method entirely than __init__; maybe you have some bizarre lazy initialization thing going.
Python achieves this flexibility by keeping things simple. You initialise objects by writing an __init__ method that sets attributes on self. That's it. It behaves exactly like a method, because it is exactly a method. There are no other strange and unintuitive rules about things having to be done first, or things that will automatically happen if you don't do other things. The only purpose it needs to serve is to be a hook to execute during object initialisation to set initial attribute values, and it does just that. If you want it to do something else, you explicitly write that in your code.
To avoid confusion it is useful to know that you can invoke the base_class __init__() method if the child_class does not have an __init__() class.
Example:
class parent:
def __init__(self, a=1, b=0):
self.a = a
self.b = b
class child(parent):
def me(self):
pass
p = child(5, 4)
q = child(7)
z= child()
print p.a # prints 5
print q.b # prints 0
print z.a # prints 1
In fact the MRO in python will look for __init__() in the parent class when can not find it in the children class. You need to invoke the parent class constructor directly if you have already an __init__() method in the children class.
For example the following code will return an error:
class parent:
def init(self, a=1, b=0):
self.a = a
self.b = b
class child(parent):
def __init__(self):
pass
def me(self):
pass
p = child(5, 4) # Error: constructor gets one argument 3 is provided.
q = child(7) # Error: constructor gets one argument 2 is provided.
z= child()
print z.a # Error: No attribute named as a can be found.
"Explicit is better than implicit." It's the same reasoning that indicates we should explicitly write 'self'.
I think in in the end it is a benefit-- can you recite all of the rules Java has regarding calling superclasses' constructors?
Right now, we have a rather long page describing the method resolution order in case of multiple inheritance: http://www.python.org/download/releases/2.3/mro/
If constructors were called automatically, you'd need another page of at least the same length explaining the order of that happening. That would be hell...
Often the subclass has extra parameters which can't be passed to the superclass.
Maybe __init__ is the method that the subclass needs to override. Sometimes subclasses need the parent's function to run before they add class-specific code, and other times they need to set up instance variables before calling the parent's function. Since there's no way Python could possibly know when it would be most appropriate to call those functions, it shouldn't guess.
If those don't sway you, consider that __init__ is Just Another Function. If the function in question were dostuff instead, would you still want Python to automatically call the corresponding function in the parent class?
i believe the one very important consideration here is that with an automatic call to super.__init__(), you proscribe, by design, when that initialization method is called, and with what arguments. eschewing automatically calling it, and requiring the programmer to explicitly do that call, entails a lot of flexibility.
after all, just because class B is derived from class A does not mean A.__init__() can or should be called with the same arguments as B.__init__(). making the call explicit means a programmer can have e.g. define B.__init__() with completely different parameters, do some computation with that data, call A.__init__() with arguments as appropriate for that method, and then do some postprocessing. this kind of flexibility would be awkward to attain if A.__init__() would be called from B.__init__() implicitly, either before B.__init__() executes or right after it.
As Sergey Orshanskiy pointed out in the comments, it is also convenient to write a decorator to inherit the __init__ method.
You can write a decorator to inherit the __init__ method, and even perhaps automatically search for subclasses and decorate them. – Sergey Orshanskiy Jun 9 '15 at 23:17
Part 1/3: The implementation
Note: actually this is only useful if you want to call both the base and the derived class's __init__ since __init__ is inherited automatically. See the previous answers for this question.
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
super(type(self), self).__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
child(42)
Outputs:
Base: 42
Part 2/3: A warning
Warning: this doesn't work if base itself called super(type(self), self).
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
'''Warning: recursive calls.'''
super(type(self), self).__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
class child2(child):
#default_init
def __init__(self, n: int) -> None:
pass
child2(42)
RecursionError: maximum recursion depth exceeded while calling a Python object.
Part 3/3: Why not just use plain super()?
But why not just use the safe plain super()? Because it doesn't work since the new rebinded __init__ is from outside the class, and super(type(self), self) is required.
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
child(42)
Errors:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-9-6f580b3839cd> in <module>
13 pass
14
---> 15 child(42)
<ipython-input-9-6f580b3839cd> in wrapper(self, *args, **kwargs)
1 def default_init(func):
2 def wrapper(self, *args, **kwargs) -> None:
----> 3 super().__init__(*args, **kwargs)
4 return wrapper
5
RuntimeError: super(): __class__ cell not found
Background - We CAN AUTO init a parent AND child class!
A lot of answers here and say "This is not the python way, use super().__init__() from the subclass". The question is not asking for the pythonic way, it's comparing to the expected behavior from other languages to python's obviously different one.
The MRO document is pretty and colorful but it's really a TLDR situation and still doesn't quite answer the question, as is often the case in these types of comparisons - "Do it the Python way, because.".
Inherited objects can be overloaded by later declarations in subclasses, a pattern building on #keyvanrm's (https://stackoverflow.com/a/46943772/1112676) answer solves the case where I want to AUTOMATICALLY init a parent class as part of calling a class without explicitly calling super().__init__() in every child class.
In my case where a new team member might be asked to use a boilerplate module template (for making extensions to our application without touching the core application source) which we want to make as bare and easy to adopt without them needing to know or understand the underlying machinery - to only need to know of and use what is provided by the application's base interface which is well documented.
For those who will say "Explicit is better than implicit." I generally agree, however, when coming from many other popular languages inherited automatic initialization is the expected behavior and it is very useful if it can be leveraged for projects where some work on a core application and others work on extending it.
This technique can even pass args/keyword args for init which means pretty much any object can be pushed to the parent and used by the parent class or its relatives.
Example:
class Parent:
def __init__(self, *args, **kwargs):
self.somevar = "test"
self.anothervar = "anothertest"
#important part, call the init surrogate pass through args:
self._init(*args, **kwargs)
#important part, a placeholder init surrogate:
def _init(self, *args, **kwargs):
print("Parent class _init; ", self, args, kwargs)
def some_base_method(self):
print("some base method in Parent")
self.a_new_dict={}
class Child1(Parent):
# when omitted, the parent class's __init__() is run
#def __init__(self):
# pass
#overloading the parent class's _init() surrogate
def _init(self, *args, **kwargs):
print(f"Child1 class _init() overload; ",self, args, kwargs)
self.a_var_set_from_child = "This is a new var!"
class Child2(Parent):
def __init__(self, onevar, twovar, akeyword):
print(f"Child2 class __init__() overload; ", self)
#call some_base_method from parent
self.some_base_method()
#the parent's base method set a_new_dict
print(self.a_new_dict)
class Child3(Parent):
pass
print("\nRunning Parent()")
Parent()
Parent("a string", "something else", akeyword="a kwarg")
print("\nRunning Child1(), keep Parent.__init__(), overload surrogate Parent._init()")
Child1()
Child1("a string", "something else", akeyword="a kwarg")
print("\nRunning Child2(), overload Parent.__init__()")
#Child2() # __init__() requires arguments
Child2("a string", "something else", akeyword="a kwarg")
print("\nRunning Child3(), empty class, inherits everything")
Child3().some_base_method()
Output:
Running Parent()
Parent class _init; <__main__.Parent object at 0x7f84a721fdc0> () {}
Parent class _init; <__main__.Parent object at 0x7f84a721fdc0> ('a string', 'something else') {'akeyword': 'a kwarg'}
Running Child1(), keep Parent.__init__(), overload surrogate Parent._init()
Child1 class _init() overload; <__main__.Child1 object at 0x7f84a721fdc0> () {}
Child1 class _init() overload; <__main__.Child1 object at 0x7f84a721fdc0> ('a string', 'something else') {'akeyword': 'a kwarg'}
Running Child2(), overload Parent.__init__()
Child2 class __init__() overload; <__main__.Child2 object at 0x7f84a721fdc0>
some base method in Parent
{}
Running Child3(), empty class, inherits everything, access things set by other children
Parent class _init; <__main__.Child3 object at 0x7f84a721fdc0> () {}
some base method in Parent
As one can see, the overloaded definition(s) take the place of those declared in Parent class but can still be called BY the Parent class thereby allowing one to emulate the classical implicit inheritance initialization behavior Parent and Child classes both initialize without needing to explicitly invoke the Parent's init() from the Child class.
Personally, I call the surrogate _init() method main() because it makes sense to me when switching between C++ and Python for example since it is a function that will be automatically run for any subclass of Parent (the last declared definition of main(), that is).