Develop Reference

Singleton with __new__ returns "Was __classcell__ propagated to type.__new_?" using Python 3.8

Trying to change singleton using metaclass of Python 2 to Python 3, __new__ returns:
[ ERROR ] Error in file Importing test library 'C:\Users\TestTabs.py' failed: __class__ not set defining 'BrowserDriver' as <class 'BrowserDriver.BrowserDriver'>. Was __classcell__ propagated to type.__new__?
CODE:
class Singleton(type):
_instance = None
def __new__(cls, *args, **kwargs):
print('Newtest')
if cls._instance is None:
Singleton._instance = type.__new__(cls, *args, **kwargs)
return Singleton._instance
This one is called:
class BrowserDriver(metaclass=Singleton)

first: you should not be using a metaclass for having a singleton
Second: your "singleton" code is broken, even if it would work:
By luck it crossed the way of a new mechanism used in class creation, which requires type.__new__ to receive the "class cell" when creating a new class, and this was detected.
So, the misterious __class__ cell will exit if any method in your class uses a call to super(). Python will create a rathr magic __class__ variable that will receive a reference to the class that will be created, when the class body execution ends. At that point, the metaclass.__new__ is called. When the call to metaclass.__new__ returns, the Python runtime expects that the __class__ magic variable for that class is now "filled in" with a reference to the class itself.
This is for a working class creation - now we come to the bug in your code:
I don't know where you got this "singleton metaclass code" at all, but it is broken: (if it would work), it creates ONE SINGLE CLASS, for all classes using this metaclass - and not, as probably was desired, allow one single-instance of each class using this metaclass. (as the new class body do not have its __class__ attribute set, you get the error you described under Python 3.8)
In other words: any classes past the first one using this metaclass is simply ignored, and not used by the program at all.
The (overkill) idea of using a metaclass to create singleton-enforcing classes is, yes, to allow a single-instance of a class, but the cache for the single instance should be set in the class itself, not on the metaclass - or in an attribute in the metaclass that holds one instance for each class created, like a dictionary would. A simple class attribute of the metaclass as featured in this code just makes classes past the first be ignored.
So, to fix that using metaclasses, the cache logic should be in the metaclass __call__ method, not in its __new__ method -
This is the expressly not recommended, but working, metaclass to enforce singletons:
class SingletonEnforcingmeta(type):
def __call__(cls, *args, **kw):
# check "__dict__" entry insead of "hasattr" - allows inheritance
# and one instance per subclass
if "_instance" not in cls.__dict__:
cls._instance = super().__call__(*args, **kw)
return cls._instance
But, as I wrote above, it is overkill to have a metaclass if you just once a singleton - the instantiation mechanism in __new__ itself is enough for creating a single-instance cache.
But before doing that - on should think: is a "singleton enforcing class really necessary" ? This is Python - the flexible structure and "consenting adults" mindset of the language can have you simply create an instance of your class in the same namespace you created the class itself - and just use that single instance from that point on.
Actually, if your single-instance have the same name the class have, one can't even create a new instance by accident, as the class itself will be reachable only indirectly. That is:
nice thing to do: if you need a singleton, create a singleton, not a 'singleton-enforcing-class
class BrowserDriver(...):
# normal code for the class here
...
BrowserDriver = BrowserDriver()
That is all there is to it. All you have now is a single-instance of
the BrowserDriver class that can be used from any place in your code.
Now, if you really need a singleton-enforcing class, one that upon
trying to create any instance beyond the first will silently do not
raise this attempt as an error, and just return the first instance ever created,
then the code you need in then __new__ method of the class is like the code
you were trying to use as the metaclass´ __new__. It records the sinvgle instance in the class itself:
if really needed: singleton enforcing-class using __new__:
class SingletonBase:
def __new__(cls, *args, **kw):
if "_instance" not in cls.__dict__:
cls._instance = super().__new__(cls, *args, **kw)
return cls._instance
And then just inherit your "I must be a singleton" classes from this base.
Note however, that __init__ will be called on the single-instance at each instantiation attempt - so, these singletons should use __new__ (and call super() as appropriate, instead of having an __init__ method, or have an idempotent __init__ (i.e. it can be called more than once, but this extra call have no effects)

super().__new__() for object vs type in Python 3.7

Calling super().__new__() when overriding __new__ in a metaclass contains the following function signature:
class MyMeta(type):
def __new__(cls, name, bases, classdict):
clsobj = super().__new__(cls, name, bases, classdict)
However, when overriding __new__ in a normal class, we have the following:
class A:
def __new__(cls, *a, **kw):
clsobj = super().__new__(cls)
Passing any other arguments to super() in A.__new__ will lead to the following error:
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
I understand that in the second case, we are dealing with object.__new__ while in the first case we are dealing with type.__new__.
My question is, why are these function signatures different? Why does object.__new__ only accept cls?

object and type have an interesting relationship. object is an instance of type, but type is a subclass of object.
__new__, however, focuses on the creation of an instance of a class, and so object.__new__ is the lowest common denominator: it does virtually nothing on its own, because there is virtually nothing that instances of every possible class have in common. As such, object.__new__ needs no more information than the type its return value will have.
type.__new__ does quite a bit more than object.__new__: it has to create an object that itself is capable of creating new objects. This requires quite a bit more information, so type.__new__ defines several additional parameters when it overrides object.__new__.
Note, however, that type itself does not use super; if you define two classes
class A:
def __new__(cls, *args, **kwargs):
print("In A.__new__")
return super().__new__(cls)
class B(type, A):
def __new__(cls, name, bases, dct, *args, **kwargs):
print("In B.__new__")
return super().__new__(cls, name, bases, dct)
you'll see that B("B", (), {}) outputs In B.__new__ but not In A.__new__. type.__new__ is the end of the line, so to speak, for creating metaclasses.
Typically, though, you wouldn't mix classes like this. Just like you rarely include object in the list of base classes for a class (I'm almost willing to say you never would), you don't often inherit from type and another (meta)class, and I can't think of any reason why you would try to inherit from both type and a non-type subclass of object.

What are the differences between a `classmethod` and a metaclass method?

In Python, I can create a class method using the #classmethod decorator:
>>> class C:
... #classmethod
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> C.f()
f called with cls=<class '__main__.C'>
Alternatively, I can use a normal (instance) method on a metaclass:
>>> class M(type):
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> class C(metaclass=M):
... pass
...
>>> C.f()
f called with cls=<class '__main__.C'>
As shown by the output of C.f(), these two approaches provide similar functionality.
What are the differences between using #classmethod and using a normal method on a metaclass?

As classes are instances of a metaclass, it is not unexpected that an "instance method" on the metaclass will behave like a classmethod.
However, yes, there are differences - and some of them are more than semantic:
The most important difference is that a method in the metaclass is not "visible" from a class instance. That happens because the attribute lookup in Python (in a simplified way - descriptors may take precedence) search for an attribute in the instance - if it is not present in the instance, Python then looks in that instance's class, and then the search continues on the superclasses of the class, but not on the classes of the class. The Python stdlib make use of this feature in the abc.ABCMeta.register method.
That feature can be used for good, as methods related with the class themselves are free to be re-used as instance attributes without any conflict (but a method would still conflict).
Another difference, though obvious, is that a method declared in the metaclass can be available in several classes, not otherwise related - if you have different class hierarchies, not related at all in what they deal with, but want some common functionality for all classes, you'd have to come up with a mixin class, that would have to be included as base in both hierarchies (say for including all classes in an application registry). (NB. the mixin may sometimes be a better call than a metaclass)
A classmethod is a specialized "classmethod" object, while a method in the metaclass is an ordinary function.
So, it happens that the mechanism that classmethods use is the "descriptor protocol". While normal functions feature a __get__ method that will insert the self argument when they are retrieved from an instance, and leave that argument empty when retrieved from a class, a classmethod object have a different __get__, that will insert the class itself (the "owner") as the first parameter in both situations.
This makes no practical differences most of the time, but if you want access to the method as a function, for purposes of adding dynamically adding decorator to it, or any other, for a method in the metaclass meta.method retrieves the function, ready to be used, while you have to use cls.my_classmethod.__func__ to retrieve it from a classmethod (and then you have to create another classmethod object and assign it back, if you do some wrapping).
Basically, these are the 2 examples:
class M1(type):
def clsmethod1(cls):
pass
class CLS1(metaclass=M1):
pass
def runtime_wrap(cls, method_name, wrapper):
mcls = type(cls)
setattr(mcls, method_name, wrapper(getatttr(mcls, method_name)))
def wrapper(classmethod):
def new_method(cls):
print("wrapper called")
return classmethod(cls)
return new_method
runtime_wrap(cls1, "clsmethod1", wrapper)
class CLS2:
#classmethod
def classmethod2(cls):
pass
def runtime_wrap2(cls, method_name, wrapper):
setattr(cls, method_name, classmethod(
wrapper(getatttr(cls, method_name).__func__)
)
)
runtime_wrap2(cls1, "clsmethod1", wrapper)
In other words: apart from the important difference that a method defined in the metaclass is visible from the instance and a classmethod object do not, the other differences, at runtime will seem obscure and meaningless - but that happens because the language does not need to go out of its way with special rules for classmethods: Both ways of declaring a classmethod are possible, as a consequence from the language design - one, for the fact that a class is itself an object, and another, as a possibility among many, of the use of the descriptor protocol which allows one to specialize attribute access in an instance and in a class:
The classmethod builtin is defined in native code, but it could just be coded in pure python and would work in the exact same way. The 5 line class bellow can be used as a classmethod decorator with no runtime differences to the built-in #classmethod" at all (though distinguishable through introspection such as calls toisinstance, and evenrepr` of course):
class myclassmethod:
def __init__(self, func):
self.__func__ = func
def __get__(self, instance, owner):
return lambda *args, **kw: self.__func__(owner, *args, **kw)
And, beyond methods, it is interesting to keep in mind that specialized attributes such as a #property on the metaclass will work as specialized class attributes, just the same, with no surprising behavior at all.

When you phrase it like you did in the question, the #classmethod and metaclasses may look similar but they have rather different purposes. The class that is injected in the #classmethod's argument is usually used for constructing an instance (i.e. an alternative constructor). On the other hand, the metaclasses are usually used to modify the class itself (e.g. like what Django does with its models DSL).
That is not to say that you can't modify the class inside a classmethod. But then the question becomes why didn't you define the class in the way you want to modify it in the first place? If not, it might suggest a refactor to use multiple classes.
Let's expand the first example a bit.
class C:
#classmethod
def f(cls):
print(f'f called with cls={cls}')
Borrowing from the Python docs, the above will expand to something like the following:
class ClassMethod(object):
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
class C:
def f(cls):
print(f'f called with cls={cls}')
f = ClassMethod(f)
Note how __get__ can take either an instance or the class (or both), and thus you can do both C.f and C().f. This is unlike the metaclass example you give which will throw an AttributeError for C().f.
Moreover, in the metaclass example, f does not exist in C.__dict__. When looking up the attribute f with C.f, the interpreter looks at C.__dict__ and then after failing to find, looks at type(C).__dict__ (which is M.__dict__). This may matter if you want the flexibility to override f in C, although I doubt this will ever be of practical use.

In your example, the difference would be in some other classes that will have M set as their metaclass.
class M(type):
def f(cls):
pass
class C(metaclass=M):
pass
class C2(metaclass=M):
pass
C.f()
C2.f()
class M(type):
pass
class C(metaclass=M):
#classmethod
def f(cls):
pass
class C2(metaclass=M):
pass
C.f()
# C2 does not have 'f'
Here is more on metaclasses
What are some (concrete) use-cases for metaclasses?

Both #classmethod and Metaclass are different.
Everything in python is an object. Every thing means every thing.
What is Metaclass ?
As said every thing is an object. Classes are also objects in fact classes are instances of other mysterious objects formally called as meta-classes. Default metaclass in python is "type" if not specified
By default all classes defined are instances of type.
Classes are instances of Meta-Classes
Few important points are to understand metioned behaviour
As classes are instances of meta classes.
Like every instantiated object, like objects(instances) get their attributes from class. Class will get it's attributes from Meta-Class
Consider Following Code
class Meta(type):
def foo(self):
print(f'foo is called self={self}')
print('{} is instance of {}: {}'.format(self, Meta, isinstance(self, Meta)))
class C(metaclass=Meta):
pass
C.foo()
Where,
class C is instance of class Meta
"class C" is class object which is instance of "class Meta"
Like any other object(instance) "class C" has access it's attributes/methods defined in it's class "class Meta"
So, decoding "C.foo()" . "C" is instance of "Meta" and "foo" is method calling through instance of "Meta" which is "C".
First argument of method "foo" is reference to instance not class unlike "classmethod"
We can verify as if "class C" is instance of "Class Meta
isinstance(C, Meta)
What is classmethod?
Python methods are said to be bound. As python imposes the restriction that method has to be invoked with instance only.
Sometimes we might want to invoke methods directly through class without any instance (much like static members in java) with out having to create any instance.By default instance is required to call method. As a workaround python provides built-in function classmethod to bind given method to class instead of instance.
As class methods are bound to class. It takes at least one argument which is reference to class itself instead of instance (self)
if built-in function/decorator classmethod is used. First argument
will be reference to class instead of instance
class ClassMethodDemo:
#classmethod
def foo(cls):
print(f'cls is ClassMethodDemo: {cls is ClassMethodDemo}')
As we have used "classmethod" we call method "foo" without creating any instance as follows
ClassMethodDemo.foo()
Above method call will return True. Since first argument cls is indeed reference to "ClassMethodDemo"
Summary:
Classmethod's receive first argument which is "a reference to class(traditionally referred as cls) itself"
Methods of meta-classes are not classmethods. Methods of Meta-classes receive first argument which is "a reference to instance(traditionally referred as self) not class"

Dynamically add class variables to classes inheriting mixin class

I've got a mixin class, that adds some functionality to inheriting classes, but the mixin requires some class attributes to be present, for simplicity let's say only one property handlers. So this would be the usage of the mixin:
class Mixin:
pass
class Something(Mixin):
handlers = {}
The mixin can't function without this being defined, but I really don't want to specify the handlers in every class that I want to use the mixin with. So I solved this by writing a metaclass:
class MixinMeta:
def __new__(mcs, *args, **kwargs):
cls = super().__new__(mcs, *args, **kwargs)
cls.handlers = {}
return cls
class Mixin(metaclass=MixinMeta):
pass
And this works exactly how I want it to. But I'm thinking this can become a huge problem, since metaclasses don't work well together (I read various metaclass conflicts can only be solved by creating a new metaclass that resolves those conflicts).
Also, I don't want to make the handlers property a property of the Mixin class itself, since that would mean having to store handlers by their class names inside the Mixin class, complicating the code a bit. I like having each class having their handlers on their own class - it makes working with them simpler, but clearly this has drawbacks.
My question is, what would be a better way to implement this? I'm fairly new to metaclasses, but they seem to solve this problem well. But metaclass conflicts are clearly a huge issue when dealing with complex hierarchies without having to define various metaclasses just to resolve those conflicts.

Your problem is very real, and Python folks have thought of this for Python 3.6 (still unrealsed) on. For now (up to Python 3.5), if your attributes can wait to exist until your classes are first instantiated, you could put cod to create a (class) attribute on the __new__ method of your mixin class itself - thus avoiding the (extra) metaclass:
class Mixin:
def __new__(cls):
if not hasattr(cls, handlers):
cls.handlers = {}
return super().__new__(cls)
For Python 3.6 on, PEP 487 defines a __init_subclass__ special method to go on the mixin class body. This special method is not called for the mixin class itself, but will be called at the end of type.__new__ method (the "root" metaclass) for each class that inherits from your mixin.
class Mixin:
def __init_subclass__(cls, **kwargs):
cls.handlers = {}
return super().__init_subclass__(**kwargs)
As per the PEP's background text, the main motivation for this is exactly what led you to ask your question: avoid the need for meta-classes when simple customization of class creation is needed, in order to reduce the chances of needing different metaclasses in a project, and thus triggering a situation of metaclass conflict.

Python 2 and 3 metaclass compatibility when kwargs are used

I am making a metaclass where I customize the __new__ method to customize how the new class will be created according to the provided values in kwargs. This probably makes more sense in an example:
class FooMeta(type):
def __new__(cls, name, bases, kwargs):
# do something with the kwargs...
# for example:
if 'foo' in kwargs:
kwargs.update({
'fooattr': 'foovalue'
})
return super(FooMeta, cls).__new__(cls, name, bases, kwargs)
My problem is how can I make this compatible for both Python 2 and 3. Six is a great compatibility library however it does not solve my problem. You would use it as:
class FooClass(six.with_metaclass(FooMeta, FooBase)):
pass
This does not work because six creates a new base class by using the given metaclass. Below is the six's code (link) (as of 1.3):
def with_metaclass(meta, base=object):
return meta("NewBase", (base,), {})
As a result, my __new__ method will be called without any kwargs hence essentially "breaking" my function. The question is how can I accomplish the behavior I want without breaking the compatibility for both Python 2 and 3. Please note that I don't need Python 2.6 support so if something is possible for only Python 2.7.x and 3.x I am fine with that.
Background
I need this to work in Django. I want to create a model factory metaclass which depending on the model attributes will customize how the model is created.
class ModelMeta(ModelBase):
def __new__(cls, name, bases, kwargs):
# depending on kwargs a different model is constructed
# ...
class FooModel(six.with_metaclass(ModelMeta, models.Model):
# some attrs here to be passed to the metaclass

If I understand you right, there is no problem and it already works. Did you actually check whether your metaclass has the desired effect? Because I think it already does.
The class returned by with_metaclass is not meant to play the role of your class FooClass. It is a dummy class used as the base class of your class. Since this base class is of metaclass FooMeta, your derived class will also have metaclass FooMeta and so the metaclass will be called again with the appropriate arguments.
class FooMeta(type):
def __new__(cls, name, bases, attrs):
# do something with the kwargs...
# for example:
if 'foo' in attrs:
attrs['fooattr'] = 'foovalue'
return super(FooMeta, cls).__new__(cls, name, bases, attrs)
class FooBase(object):
pass
class FooClass(with_metaclass(FooMeta, FooBase)):
foo = "Yes"
>>> FooClass.fooattr
'foovalue'
Incidentally, it is confusing to call the third argument of your metaclass kwargs. It isn't really keyword arguments. It is the __dict__ of the class to be created --- that is, the class's attributes and their values. In my experience this attribute is conventionally called attrs or perhaps dct (see e.g., here and here).