Ok, obviously __new__ in a metaclass runs when an instance of the metaclass i.e. a class object is instantiated, so __new__ in a metaclass provides a hook to intercept events (/code that runs) at class definition time (e.g. validating/enforcing rules for class attributes such as methods etc.).
Many online examples of __new__ in a metaclass return an instance of the type constructor from __new__, which seems a bit problematic since this blocks __init__ (docs: "If __new__() does not return an instance of cls, then the new instance’s __init__() method will not be invoked").
While tinkering with return values of __new__ in a metaclass I came across some somewhat strange cases which I do not fully understand, e.g.:
class Meta(type):
def __new__(self, name, bases, attrs):
print("Meta __new__ running!")
# return type(name, bases, attrs) # 1.
# return super().__new__(self, name, bases, attrs) # 2.
# return super().__new__(name, bases, attrs) # 3.
# return super().__new__(type, name, bases, attrs) # 4.
# return self(name, bases, attrs) # 5.
def __init__(self, *args, **kwargs):
print("Meta __init__ running!")
return super().__init__(*args, **kwargs)
class Cls(metaclass=Meta):
pass
This is often seen in examples and generally works, but blocks __init__
This works and __init__ also fires; but why pass self to a super() call? Shouldn't self/cls get passed automatically with super()?
This throws a somewhat strange error I can't really make sense of: TypeError: type.__new__(X): X is not a type object (str); what is X? shouldn't self be auto-passed?
The error of 3. inspired me to play with the first arg of the super() call, so I tried to pass type directly; this also blocks __init__. What is happening here?
tried just for fun; this leads to a RecursionError
Also especially cases 1. and 2. appear to have quite profound implications for inheriting from classes bound to metaclasses:
class LowerCaseEnforcer(type):
""" Allows only lower case names as class attributes! """
def __new__(self, name, bases, attrs):
for name in attrs:
if name.lower() != name:
raise TypeError(f"Inappropriate method name: {name}")
# return type(name, bases, attrs) # 1.
# return super().__new__(self, name, bases, attrs) # 2.
class Super(metaclass=LowerCaseEnforcer):
pass
class Sub(Super):
def some_method(self):
pass
## this will error in case 2 but not case 1
def Another_method(self):
pass
expected behavior: metaclass is bound to superclass, but not to subclass
binds the superclass /and/ subclasses to the metaclass; ?
I would much appreciate if someone could slowly and kindly explain what exactly is going on in the above examples! :D
It is simpler than what you got too.
As you have noted, the correct thing to do is your 2 above:
return super().__new__(self, name, bases, attrs) # 2.
Here it goes: __new__ is a special method - although in certain documentations, even part of the official documentation, it is described as being a classmethod, it is not quite so: it, as an object, behaves more like a static method - in a sense that Python does not automatically fill the first parameter when one calls MyClass.__new__() - i.e., you'd have to call MyClass.__new__(MyClass) for it to work. (I am a step back here - this info applies to all classes: metaclasses and ordinary classes).
When you call MyClass() to create a new instance, then Python will call MyClass.__new__ and insert the cls parameter as first parameter.
With metaclasses, the call to create a new instance of the metaclass is triggered by the execution of the class statement and its class body. Likewise, Python fills in the first parameter to Metaclass.__new__, passing the metaclass itself.
When you call super().__new__ from within your metaclass' __new__ you are in the same case of one calling __new__ manually: the parameter specifying which class' that __new__ should apply have to be explicitly filled.
Now, what is confusing you is that you are writting the first parameter to __new__ as self - which would be correct if it were an instance of the metaclass (i.e. an ordinary class). As it is, that parameter is a reference to the metaclass itself.
The docs does not inform an official, or recomended name for the first parameter of a metaclass __new__, but usually it is something along mcls, mcs, metaclass, metacls - to make it different from cls which is the usuall name for the first parameter of a non-metaclass __new__ method. In a metaclass, the "class" - cls is what is created by the ultimate call to type.__new__ (either hardcoded, or using super()) the return of it is the new-born class (it can be further modified in the __new__ method after the call to the superclass) - and when returned, the call to __init__ takes place normally.
So, I will just comment further the use of trying to call type(name, bases, namespace) instead of type.__new__(mcls, name, bases, namespace): the first form will just create a plain class, as if the metaclass had not been used at all - (lines in the metaclass __new__ that modify the namespace or bases, of course, have their effect. But the resulting class will have type as its metaclass, and subclasses of it won't call the metaclass at all. (For the record, it works as a "pre-class decorator" - which can act on class parameters before it is created, and it could even be an ordinary function, instead of a class with a __new__ method - the call to type is what will create the new class after all)
A simple way to check if the metaclass is "bound" to your class is to check its type with type(MyClass) or MyClass.__class__ .
I think I finally figured this out (somewhat), my initial confusion can mainly be ascribed to my failure to realize that
there is a difference between object.__new__ and type.__new__
there is a difference between returning type() and returning super().__new__ from a metaclass
Discussion of these two points should clear up my initial example as well as the seemingly enigmatic inheritance behavior.
1. The difference between object.__new__ and type.__new__
First a few words concerning __new__.
The documenation is imo pretty clear on this, but I'd still like to add and/or emphasize some things:
__new__ can be understood as a special cased static method that takes cls as first parameter and passes the remaining parameters (most often *args, **kwargs) to __init__.
_ __new__ and __init__ are invoked successively (actually by the metaclass's __call__!), whereby __init__ is only invoked if __new__ returns an instance of cls.
__new__ takes a single argument (this is about calling __new__, not defining/overloading it) i.e. cls and returns an instance of that class.
An important thing that eluded me at first is that there is a difference between object.__new__ and type.__new__.
I discovered this while I was playing with __new__'s parameters/arguments; take a look at these 'instructive errors':
class ObjNewExample:
def __new__(cls, *args, **kwargs):
# return super().__new__(cls) # correct
return super().__new__(cls, *args, **kwargs) # instructive error
def __init__(self, some_attr):
self._some_attr = some_attr
one = ObjNewExample(42)
class TypeNewExample(type):
def __new__(mcls, name, bases, attrs):
# return super().__new__(mcls, name, bases, attrs) # correct
return super().__new__(mcls) # instructive error
# class Cls(metaclass=TypeNewExample):
# pass
ObjNewexample with return super().__new__(cls, *args, **kwargs) throws something like
TypeError: object.__new__() takes exactly one argument (the type to instantiate),
while TypeNewexample with return super().__new__(mcls) throws
TypeError: type.__new__() takes exactly 3 arguments,
which shows that object.__new__ and type.__new__ are quite different methods!
Also: consider the difference between parameters and arguments with __new__:
object.__new__ takes cls, *args, **kwargs as parameters, but requires only cls as argument (*args, **kwargs get passed to __init__)
type.__new__ takes mcls, name, bases, attrs as parameters and arguments
The difference between returning type() and returning super().__new__ from a metaclass
The main problem with the example I initially posted however is the difference between returning type() and returning super().__new__ from a metaclass's __new__ (which is embarrassingly obvious now..). (See also this discussion)
returning type(name, bases, attrs) from mcls: creates an instance of type
returning super().__new__(mcls, name, bases, attrs) from mcls: creates an instance of the actual metaclass (which is derived from type),
which also explains why __init__ is inhibited in case 1 but not case 2 of the initial example! (Remember: __init__ does not get invoked if __new__ returns anything but an instance if __new__'s first parameter i.e. (m)cls)
This should be instructive:
class Meta(type):
def __new__(mcls, name, bases, attrs):
# this creates an intance of type (the default metaclass)
# This should be eql to super().__new__(type, name, base, attrs)!
obj1 = type(name, bases, attrs)
# this creates an instance of the actual custom metaclass (which is derived from type)
# i.e. it creates an instance of type.__new__'s first arg
obj2 = super().__new__(mcls, name, bases, attrs)
print(isinstance(obj1, mcls))
print(obj1.__class__)
print(isinstance(obj2, mcls))
print(obj2.__class__)
class Fun(metaclass=Meta):
pass
So quickly walking through cases 1-5 from my initial post:
1: returns a new type object, that is an instance of type, not the actual custom metaclass (derived from type), thus __init__ of the custom metaclass is inhibited; this appears to be actually equivalent to case 4!
2: as #jsbueno pointed out, this is the most likely intended ('correct') behavior: this creates an instance of the actual custom metaclass.
3: this barfs itself because type.__new__ expects an object of type type (the object to be instantiated) as first argument
4: see case 1
5: self (probably better named 'cls' or 'mcls') is Meta; calling a class in its own constructor is obviously recursive.
The above also provides an explanation for the seemingly weird inheritance behavior of the second snippet from my initial posts!
So why does Sub's definition of Another_method error in case 2 of LowerCaseEnforcer, but not case 1?
Because in case 1 Lowercaseenforcer returns an instance of type (not of LowerCaseEnforcer!), so Super is of type type (its metaclass is type, not LowerCaseEnforcer)! So while LowerCaseEnforcer.__new__ fires and enforces the lowercase restriction for Super, Super is just a vanilla class of type type and Sub is derived from it (with no special effect).
Whereas in case 2 Super's metaclass is of type LowerCaseEnforcer and so is Sub's, so LowerCaseEnforcer.__new__ is involved in the definition of Sub.
One thing that is still a bit unclear however is the behavior of static methods in super calls (see also this discussion).
E.g. why does super().__new__(cls) work? Shouldn't this be super(cls, cls).__new__(cls) (or something like that)?
But I guess this is another (interesting) topic! :D
Related
Calling super().__new__() when overriding __new__ in a metaclass contains the following function signature:
class MyMeta(type):
def __new__(cls, name, bases, classdict):
clsobj = super().__new__(cls, name, bases, classdict)
However, when overriding __new__ in a normal class, we have the following:
class A:
def __new__(cls, *a, **kw):
clsobj = super().__new__(cls)
Passing any other arguments to super() in A.__new__ will lead to the following error:
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
I understand that in the second case, we are dealing with object.__new__ while in the first case we are dealing with type.__new__.
My question is, why are these function signatures different? Why does object.__new__ only accept cls?
object and type have an interesting relationship. object is an instance of type, but type is a subclass of object.
__new__, however, focuses on the creation of an instance of a class, and so object.__new__ is the lowest common denominator: it does virtually nothing on its own, because there is virtually nothing that instances of every possible class have in common. As such, object.__new__ needs no more information than the type its return value will have.
type.__new__ does quite a bit more than object.__new__: it has to create an object that itself is capable of creating new objects. This requires quite a bit more information, so type.__new__ defines several additional parameters when it overrides object.__new__.
Note, however, that type itself does not use super; if you define two classes
class A:
def __new__(cls, *args, **kwargs):
print("In A.__new__")
return super().__new__(cls)
class B(type, A):
def __new__(cls, name, bases, dct, *args, **kwargs):
print("In B.__new__")
return super().__new__(cls, name, bases, dct)
you'll see that B("B", (), {}) outputs In B.__new__ but not In A.__new__. type.__new__ is the end of the line, so to speak, for creating metaclasses.
Typically, though, you wouldn't mix classes like this. Just like you rarely include object in the list of base classes for a class (I'm almost willing to say you never would), you don't often inherit from type and another (meta)class, and I can't think of any reason why you would try to inherit from both type and a non-type subclass of object.
I am a bit surprised by the method call order and the different arguments when overriding new and init in a metaclass. Consider the following:
class AT(type):
def __new__(mcs, name, bases, dct):
print(f"Name as received in new: {name}")
return super().__new__(mcs, name + 'HELLO', bases + (list,), dct)
def __init__(cls, name, bases, dct):
print(f"Name as received in init: {name}")
pass
class A(metaclass=AT):
pass
A.__name__
The output is:
Name as received in new: A
Name as received in init: A
'AHELLO'
In short I would have expected init to receive AHELLO with the argument name.
I imagined that __init__ was called by super().__new__: if the call is not done in the overridden __new__ then my __init__ is not called.
Could someone clarify how __init__ is called in this case?
For information my use case for this is that I wanted to make creation of classes, in a special case, easier at runtime by providing only a single "base" class (and not a tuple), I then added this code in __new__:
if not isinstance(bases, tuple):
bases = (bases, )
however, I found out that I also need to add it in __init__.
Your __init__ method is obviously called and the reason for that is because your __new__ method is returning an instance of your class.
From https://docs.python.org/3/reference/datamodel.html#object.new:
If __new__() returns an instance of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to __new__().
As you can see the arguments passed to __init__ are those passed to __new__ method's caller not when you call it using super. It's a little bit vague but that's what it means if you read it closely.
And regarding the rest it works just as expected:
In [10]: A.__bases__
Out[10]: (list,)
In [11]: a = A()
In [12]: a.__class__.__bases__
Out[12]: (list,)
The fact is that what orchestrates the call of __new__ and __init__ of an ordinary class is the __call__ method on its metaclass. The code in the __call__ method of type, the default metatype, is in C, but the equivalent of it in Python would be:
class type:
...
def __call__(cls, *args, **kw):
instance = cls.__new__(cls, *args, **kw) # __new__ is actually a static method - cls has to be passed explicitly
if isinstance(instance, cls):
instance.__init__(*args, **kw)
return instance
That takes place for most object instantiation in Python, including when instantiating classes themselves - the metaclass is implicitly called as part of a class statement. In this case, the __new__ and __init__ called from type.__call__ are the methods on the metaclass itself. And in this case, type is acting as the "metametaclass" - a concept seldom needed, but it is what creates the behavior you are exploring.
When creating classes, type.__new__ will be responsible for calling the class (not the metaclass) __init_subclass__, and its descriptors' __set_name__ methods - so, the "metametaclass" __call__ method can't control that.
So, if you want the args passed to the metaclass __init__ to be programmatically modified, the "normal" way will be to have a "metametaclass", inheriting from type and distinct from your metaclass itself, and override its __call__ method:
class MM(type):
def __call__(metacls, name, bases, namespace, **kw):
name = modify(name)
cls = metacls.__new__(metacls, name, bases, namespace, **kw)
metacls.__init__(cls, name, bases, namespace, **kw)
return cls
# or you could delegate to type.__call__, replacing the above with just
# return super().__call__(modify(name), bases, namespace, **kw)
Of course that is a way to get to go closer to "turtles all way to the bottom" than anyone would ever like in production code.
An alternative is to keep the modified name as an attribute on the metaclass, so that its __init__ method can take the needed information from there, and ignore the name passed in from its own metaclass' __call__ invocation. The information channel can be an ordinary attribute on the metaclass instance. Well - it happens that the "metaclass instance" is the class being created itself - and oh, see - that the name passed to type.__new__ already gets recorded in it - on the __name__ atribute.
In other words, all you have to do to use a class name modified in a metaclass __new__ method in its own __init__ method, is to ignore the passed in name argument, and use cls.__name__ instead:
class Meta(type):
def __new__(mcls, name, bases, namespace, **kw):
name = modified(name)
return super().__new__(mcls, name, bases, namespace, **kw)
def __init__(cls, name, bases, namespace, **kw):
name = cls.__name__ # noQA (otherwise linting tools would warn on the overriden parameter name)
...
I want to implement a singleton pattern in python, and I liked the pattern described in the http://www.python-course.eu/python3_metaclasses.php.
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class SingletonClass(metaclass=Singleton):
pass
class RegularClass():
pass
x = SingletonClass()
y = SingletonClass()
print(x == y)
x = RegularClass()
y = RegularClass()
print(x == y)
And the code works perfect. But, the __call__() does not have the self, and it also does not have #classmethod or #staticmethod declaration.
But, in the Python data model https://docs.python.org/3/reference/datamodel.html#object.__call__ the __call__() method has a self in the arguments.
The code does not work if I pass self, or declare as #staticmethod or #classmethod.
Can someone please explain the logic of the syntax behind the __call__() method.
Naming the first argument of a method cls or self are just a convention. The __call__ method does have a self argument, only it is named cls here. That's because for a metaclass, the method is bound to a class object, and the name reflects this.
The same convention is applied to #classmethod methods; the first argument is a class, always, due to the nature of how a classmethod object is bound, so it makes sense to name that first argument cls.
But you are free to name that first argument anything else. It is not the name that makes a classmethod or a regular method or a method on a metatype work. All that using self or cls does is document what type of object this is, making it easier for other developers to mentally track what is going on.
So no, this is not an implicit class method. That first argument is not bound to the Singleton metaclass object, it is bound to the class that was called. That makes sense, because that class object is an instance of the Singleton metatype.
If you want to dive into how binding works (the process that causes that first argument to be passed in, whatever the name), you can read up on the Descriptor HOWTO. TLDR: functions, property, classmethod and staticmethod objects are all descriptors, and whenever you access them as an attribute on a supporting object such as an instance or a class, they are bound, often causing a different object to be returned as a result, which when called passes in the bound object to the actual function.
Martin's answer says it all. I am adding this so maybe a different wording can throw in more light for different people:
Python call() is this an implicit classmethod?
No. But all "metaclass" methods are implicit "class methods" for the classes that use that metaclass.
That is implicit when we take in account the fact that classes are simply instances of the metaclass. From the language point of view, a class behave almost exactly like any other instance - and any interactions with a class that would trigger dunder (__magic__) methods in an object - like using the "+, -, *, /" operators, or index retrieval with [ ], or calling them with ( ) trigger the corresponding methods on its class. That is ordinarily type.
And, as put in the other answer, Python does not care what name you ut on the first argument of a method. For metaclasses it makes sense to use cls there, since the "instances" the methods are dealing with are metaclasses. As it makes sense that the first argument to a metaclass' __new__ method be named metacls (like in the example bellow). Because the new "cls" is the object we get after calling type.__new__ - the only call possible in pure Python that will actually create a class object.
class Meta(type):
def __new__(metacls, name, bases, namespace):
...
cls = super().__new__(metacls, name, bases, namespace)
...
return cls
(Now, still on the topic of the question: __new__ is a special case
of an implicit static method (even for ordinary classes that are not intended to be metaclasses) - to which Python specially add the first argument, using a different mechanism than what is done to regular classmethods. Thats is why the super().__new__ call above needs to include the metacls as the first parameter)
I'm reading through http://blog.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/#.VwPSjDG1XGD. In it, they have:
class Singleton(type):
instance = None
def __call__(cls, *args, **kw):
if not cls.instance:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
return cls.instance
The explanation is:
We are defining a new type, which inherits from type to provide all bells and whistles of Python classes. We override the call method, that is a special method invoked when we call the class, i.e. when we instance it. The new method wraps the original method of type by calling it only when the instance attribute is not set, i.e. the first time the class is instanced, otherwise it just returns the recorded instance. As you can see this is a very basic cache class, the only trick is that it is applied to the creation of instances.
I'm not sure I understand the line:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
Can someone explain what is happening here in another way?
By default, __call__ing on a class object produces an instance of such class (remember that classes are callables like functions, in contrast to other languages like PHP where they are completely separate monsters in their interface). This instance will be stored in cls.instance.
cls.instance = super(Singleton, cls).__call__(*args, **kw)
By wrapping in the previous condition, if the instance is already an... instance, it is returned. This means: stuff like __new__ is not called on the class again, and stuff like __init__ is not called on the instance again, but just returned the old -already existent- instance.
Notes: When you call __call__ in the default implementation, the __new__ method is called for the class (it is always a class method; you never use the #classmethod decorator there) to create the instance. After that, the __init__ message is sent to the instance to initialize it.
Notes on Metaclasses: Remember that Singleton is not an usual class, but a Metaclass. It is not a class you will inherit from, but a Metaclass you will instantiate your class with.
Objects are instances of Classes, and Classes are instances of Metaclasses (which may confuse you since they are also classes). This means that there's a closed loop in which the reference type is an instance of itself:
assert isinstance(type, type)
There, your code will come like this:
class MySingletonClass(object):
__metaclass__ = Singleton
And the magic will begin:
The __call__ method you are overriding with your code will be executed on the class since it is defined on the metaclass (Singleton). This means:
i = MySingletonClass() # Executes what is defined in Singleton as __call__
You should not confuse with this one:
i() # Will crash if MySingletonClass does not define __call__ for its instances.
The call you are making in your code has another equivalent:
super(Singleton, cls).__call__(*a, **kwa)
Is the same as:
(a type isntance)(*a, **kwa)
Or
(a type isntance).__call__(*a, **kwa)
Which is the same behavior every class, not using a custom metaclass like yours, uses to create their instances.
I have two questions converning metaclasses and multiple inheritance. The first is: Why do I get a TypeError for the class Derived but not for Derived2?
class Metaclass(type): pass
class Klass(object):
__metaclass__ = Metaclass
#class Derived(object, Klass): pass # if I uncomment this, I get a TypeError
class OtherClass(object): pass
class Derived2(OtherClass, Klass): pass # I do not get a TypeError for this
The exact error message is:
TypeError: Error when calling the metaclass bases
Cannot create a consistent method resolution order (MRO) for bases object, Klass
The second question is: Why does super not work in this case(if I use __init__ instead of __new__, super works again):
class Metaclass(type):
def __new__(self, name, bases, dict_):
return super(Metaclass, self).__new__(name, bases, dict_)
class Klass(object):
__metaclass__ = Metaclass
There I get:
TypeError: Error when calling the metaclass bases type.__new__(X):
X is not a type object (str)
I'm using Python 2.6.
The second question has already been well answered twice, though __new__ is actually a staticmethod, not a classmethod as erroneously claimed in a comment...:
>>> class sic(object):
... def __new__(cls, *x): return object.__new__(cls, *x)
...
>>> type(sic.__dict__['__new__'])
<type 'staticmethod'>
The first question (as somebody noted) has nothing to do with metaclasses: you simply can't multiply inherit from any two classes A and B in this order where B is a subclass of A. E.g.:
>>> class cis(sic): pass
...
>>> class oops(sic, cis): pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
Cannot create a consistent method resolution
order (MRO) for bases sic, cis
The MRO guarantees that leftmost bases are visited before rightmost ones - but it also guarantees that among ancestors if x is a subclass of y then x is visited before y. It's impossible to satisfy both of these guarantees in this case. There's a good reason for these guarantees of course: without them (e.g. in old style classes, which only guarantee the left-right order in method resolution, not the subclass constraint) all overrides in x would be ignored in favor of the definitions in y, and that can't make much sense. Think about it: what does it mean to inherit from object first, and from some other class second? That object's (essentially nonexistent;-) definition of its several special methods must take precedence over the other class's, causing the other class's overrides to be ignored?
For the first question, have a look at the description of MRO in python - specifically, the "bad Method Resolution order" section. Essentially, it's to do with the fact that python doesn't know whether to use object or Klass's methods. (It's nothing to do with the usage of metaclasses.)
For the second question, it looks like you're misunderstanding how the __new__ function works. It doesn't take a reference to itself as the first argument - it takes a reference to the type of the class being instantiated. So your code should look like this:
class Metaclass(type):
def __new__(cls, name, bases, dictn):
return type.__new__(cls, name, bases, dictn)
For the second question, you need to pass self to __new__ like this:
class Metaclass(type):
def __new__(self, name, bases, dict_):
return super(Metaclass, self).__new__(self, name, bases, dict_)
class Klass(object):
__metaclass__ = Metaclass
I can't recall off the top of my head why this is, but I think it's because type.__new__ isn't a bound method and thus doesn't magically get the self argument.
Why would you do?
class Derived(object, Klass):
Klass already derives from object.
class Derived(Klass):
Is the reasonable thing here.