Develop Reference

Using ABC, PolymorphicModel, django-models gives metaclass conflict

So far every other answer on SO answers in the exact same way: construct your metaclasses and then inherit the 'joined' version of those metaclasses, i.e.
class M_A(type): pass
class M_B(type): pass
class A(metaclass=M_A): pass
class B(metaclass=M_B): pass
class M_C(M_A, M_B): pass
class C:(A, B, metaclass=M_C): pass
But I don't know what world these people are living in, where they're constructing your own metaclasses! Obviously, one would be using classes from other libraries and unless you have a perfect handle on meta programming, how are you supposed to know whether you can just override a class's metaclass? (Clearly I do not have a handle on them yet).
My problem is:
class InterfaceToTransactions(ABC):
def account(self):
return None
...
class Category(PolymorphicModel, InterfaceToTransactions):
def account(self):
return self.source_account
...
class Income(TimeStampedModel, InterfaceToTransactions):
def account(self):
return self.destination_account
...
Which of course gives me the error: "metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases"
I've tried many variations of the solution given above, the following does not work, gives the same error.
class InterfaceToTransactionsIntermediaryMeta(type(PolymorphicModel), type(InterfaceToTransactions)):
pass
class Category(PolymorphicModel, InterfaceToTransactions):
__metaclass__ = InterfaceToTransactionsIntermediaryMeta
...
Nor does putting anything inside the class Meta function. I've read every single other SO question on this topic, please don't simply mark it as duplicate.
-------------------Edited 1/8/18 after accepting the solution-------
Oddly enough, if I try to makemigrations with this new configuration (the one I accepted), it starts giving the metaclass error again, but it still works during runtime. If I comment out the metaclass parts then makemigrations and migrate, it will do it successfully, but then I have to put it back in there after migrating every time.

If you are using Python 3, you are trying to use your derived metaclass incorrectly.
And since you get "the same error", and not other possible, more subtle, error, I'd say this is what is happening.
Try just changing to:
class IntermediaryMeta(type(InterfaceToTransactions), type(PolymorphicModel)):
pass
class Category(PolymorphicModel, InterfaceToTransactions, metaclass=IntermediaryMeta):
...
(At least the ABCMeta class is guaranteed to work collaboratively using super, that is enough motive to place the classe it first on the bases )
tuple)
If that yields you new and improved errors, this means that one or both of those classes can't really collaborate properly due to one of several motives. Then, the way to go is to force your inheritance tree that depends on ABCMeta not to do so, since its role is almost aesthetical in a language where everything else is for "consenting adults" like Python.
Unfortunatelly, the way to that is to use varying methods of brute-force, from safe "rewritting everything" to monkey patching ABCMeta and abstractmethod on the place were "InterfaceToTransactions" is defined to simply do nothing.
If you need to get there, and need some help, please post another question.
Sorry - this is actually the major drawbacks of using metaclasses.

Unless django-polymorphic decides to inherit from abc.ABC this is going to be very difficult to achieve. A good solution would be to "manually" create your interface. For instance:
class InterfaceToTransactions:
def account(self):
raise NotImplementedError("Account method must be implemented.")
...
class Category(PolymorphicModel, InterfaceToTransactions):
def account(self):
return self.source_account
...
class Income(TimeStampedModel, InterfaceToTransactions):
def account(self):
return self.destination_account
...

Subclass avoiding parent's metaclass

Say I have a third-party library where a metaclass requires me to implement something. But I want to have an intermediate "abstract" subclass that doesn't. How can I do this?
Consider this to be a very minimal example of what third-party library has:
class ServingMeta(type):
def __new__(cls, name, bases, classdict):
if any(isinstance(b, ServingMeta) for b in bases):
if "name" not in classdict:
# Actual code fails for a different reason,
# but the logic is the same.
raise TypeError(f"Class '{name}' has no 'name' attribute")
return super().__new__(cls, name, bases, classdict)
class Serving(object, metaclass=ServingMeta):
def shout_name(self):
return self.name.upper()
I cannot modify the code above. It's an external dependency (and I don't want to fork it).
The code is meant to be used this way:
class Spam(Serving):
name = "SPAM"
spam = Spam()
print(spam.shout_name())
However, I happen to have a lot of spam, and I want to introduce a base class with the common helper methods. Something like this:
class Spam(Serving):
def thrice(self):
return " ".join([self.shout_name()] * 3)
class LovelySpam(Spam):
name = "lovely spam"
class WonderfulSpam(Spam):
name = "wonderful spam"
Obviously, this doesn't work and fails with the well-expected TypeError: Class 'SpamBase' has no 'name' attribute declared. Would third-party library had a SpamBase class without a metaclass, I could've subclassed that - but no such luck this time (I've mentioned this inconvenience to the library authors).
I can make it a mixin:
class SpamMixin(object):
def thrice(self):
return " ".join([self.shout_name()] * 3)
class LovelySpam(SpamMixin, Serving):
name = "lovely spam"
class WonderfulSpam(SpamMixin, Serving):
name = "wonderful spam"
However, this makes me and my IDE cringe a little, as it quickly becomes cumbersome to repeat SpamMixin everywhere and also because object has no shout_name attribute (and I don't want to silence analysis tools). In short, I just don't like this approach.
What else can I do?
Is there a way to get a metaclass-less version of Serving? I think of something like this:
ServingBase = remove_metaclass(Serving)
class Spam(ServingBase, metaclass=ServingMeta):
...
But don't know how to actually implement remove_metaclass and whenever it's even reasonably possible (of course, it must be doable, with some introspection, but it could require more arcane magic than I can cast).
Any other suggestions are also welcomed. Basically, I want to have my code DRY (one base class to rule them all), and have my linter/code analysis icons all green.

The mixin approach is the correct way to go. If your IDE "cringe" that is a deffect on that tool - just disable a little of the "features" that are obviously incorrect tunning when coding for a dynamic language like Python.
And this is not even about creating things dynamically, it is merely multiple-inheritance, which is supported by the language since forever. And one of the main uses of multiple-inheritance is exactly being able to create mixins just as this one you need.
Another inheritance-based workaround is to make your hierarchy one level deeper, and just introduce the metaclass after you come up with your mixin methods:
class Mixin(object):
def mimixin(self): ...
class SpamBase(Mixin, metaclass=ServingMeta):
name = "stub"
Or just addd the mixin in an intermediate subclass:
class Base(metaclass=Serving Meta):
name = "stub"
class MixedBase(Mixin, Base):
name = "stub"
class MyLovingSpam(MixedBase):
name = "MyLovingSpam"
If you don't want to be repeating the mixin=-base name in every class, that is the way to go.
"Removing" a metaclass just for the sake of having a late mixin is way over the top. Really. Broken. The way to do it wol e re-create the class dynamically, just as #vaultah mentions in the other answer, but doing that in an intermediate class is a thing you should not do. Doing that to please the IDE is something you should not do twice: messing with metaclasses is hard enough already. Removing things on inheritance/class creation that the language puts there naturally is something nasty (cf. this answer: How to make a class attribute exclusive to the super class ) . On the other hand, mixins and multiple inheritance are just natural.
Are you still there? I told you not to do so:
Now, onto your question - instead of "supressing the metaclass" in an intermediate class, it would be more feasible to inherit the metaclass you have there and change its behavior - so that it does not check for the constraints in specially marked classes - create an attribute for your use, like _skip_checking
class MyMeta(ServingMeta):
def __new__(metacls, name, bases, namespace):
if namespace.get("_skip_checking", False):
# hardcode call to "type" metaclass:
del namespace["_skip_checking"]
cls = type.__new__(metacls, name, bases, namespace)
else:
cls = super().__new__(metacls, name, bases, namespace)
return cls
# repeat for __init__ if needed.
class Base(metaclass=MyMeta):
_skip_checking = True
# define mixin methods
class LoveSpam(Base):
name = "LoveSpam"

There's really no direct way to remove the metaclass from a Python class, because the metaclass created that class. What you can try is re-create the class using a different metaclass, which doesn't have unwanted behaviour. For example, you could use type (the default metaclass).
In [6]: class Serving(metaclass=ServingMeta):
...: def shout_name(self):
...: return self.name.upper()
...:
In [7]: ServingBase = type('ServingBase', Serving.__bases__, dict(vars(Serving)))
Basically this takes the __bases__ tuple and the namespace of the Serving class, and uses them to create a new class ServingBase. N.B. this means that ServingBase will receive all bases and methods/attributes from Serving, some of which may have been added by ServingMeta.

In Python, when should I use a meta class?

I have gone through this: What is a metaclass in Python?
But can any one explain more specifically when should I use the meta class concept and when it's very handy?
Suppose I have a class like below:
class Book(object):
CATEGORIES = ['programming','literature','physics']
def _get_book_name(self,book):
return book['title']
def _get_category(self, book):
for cat in self.CATEGORIES:
if book['title'].find(cat) > -1:
return cat
return "Other"
if __name__ == '__main__':
b = Book()
dummy_book = {'title':'Python Guide of Programming', 'status':'available'}
print b._get_category(dummy_book)
For this class.
In which situation should I use a meta class and why is it useful?
Thanks in advance.

You use metaclasses when you want to mutate the class as it is being created. Metaclasses are hardly ever needed, they're hard to debug, and they're difficult to understand -- but occasionally they can make frameworks easier to use. In our 600Kloc code base we've used metaclasses 7 times: ABCMeta once, 4x models.SubfieldBase from Django, and twice a metaclass that makes classes usable as views in Django. As #Ignacio writes, if you don't know that you need a metaclass (and have considered all other options), you don't need a metaclass.

Conceptually, a class exists to define what a set of objects (the instances of the class) have in common. That's all. It allows you to think about the instances of the class according to that shared pattern defined by the class. If every object was different, we wouldn't bother using classes, we'd just use dictionaries.
A metaclass is an ordinary class, and it exists for the same reason; to define what is common to its instances. The default metaclass type provides all the normal rules that make classes and instances work the way you're used to, such as:
Attribute lookup on an instance checks the instance followed by its class, followed by all superclasses in MRO order
Calling MyClass(*args, **kwargs) invokes i = MyClass.__new__(MyClass, *args, **kwargs) to get an instance, then invokes i.__init__(*args, **kwargs) to initialise it
A class is created from the definitions in a class block by making all the names bound in the class block into attributes of the class
Etc
If you want to have some classes that work differently to normal classes, you can define a metaclass and make your unusual classes instances of the metaclass rather than type. Your metaclass will almost certainly be a subclass of type, because you probably don't want to make your different kind of class completely different; just as you might want to have some sub-set of Books behave a bit differently (say, books that are compilations of other works) and use a subclass of Book rather than a completely different class.
If you're not trying to define a way of making some classes work differently to normal classes, then a metaclass is probably not the most appropriate solution. Note that the "classes define how their instances work" is already a very flexible and abstract paradigm; most of the time you do not need to change how classes work.
If you google around, you'll see a lot of examples of metaclasses that are really just being used to go do a bunch of stuff around class creation; often automatically processing the class attributes, or finding new ones automatically from somewhere. I wouldn't really call those great uses of metaclasses. They're not changing how classes work, they're just processing some classes. A factory function to create the classes, or a class method that you invoke immediately after class creation, or best of all a class decorator, would be a better way to implement this sort of thing, in my opinion.
But occasionally you find yourself writing complex code to get Python's default behaviour of classes to do something conceptually simple, and it actually helps to step "further out" and implement it at the metaclass level.
A fairly trivial example is the "singleton pattern", where you have a class of which there can only be one instance; calling the class will return an existing instance if one has already been created. Personally I am against singletons and would not advise their use (I think they're just global variables, cunningly disguised to look like newly created instances in order to be even more likely to cause subtle bugs). But people use them, and there are huge numbers of recipes for making singleton classes using __new__ and __init__. Doing it this way can be a little irritating, mainly because Python wants to call __new__ and then call __init__ on the result of that, so you have to find a way of not having your initialisation code re-run every time someone requests access to the singleton. But wouldn't be easier if we could just tell Python directly what we want to happen when we call the class, rather than trying to set up the things that Python wants to do so that they happen to do what we want in the end?
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
Under 10 lines, and it turns normal classes into singletons simply by adding __metaclass__ = Singleton, i.e. nothing more than a declaration that they are a singleton. It's just easier to implement this sort of thing at this level, than to hack something out at the class level directly.
But for your specific Book class, it doesn't look like you have any need to do anything that would be helped by a metaclass. You really don't need to reach for metaclasses unless you find the normal rules of how classes work are preventing you from doing something that should be simple in a simple way (which is different from "man, I wish I didn't have to type so much for all these classes, I wonder if I could auto-generate the common bits?"). In fact, I have never actually used a metaclass for something real, despite using Python every day at work; all my metaclasses have been toy examples like the above Singleton or else just silly exploration.

A metaclass is used whenever you need to override the default behavior for classes, including their creation.
A class gets created from the name, a tuple of bases, and a class dict. You can intercept the creation process to make changes to any of those inputs.
You can also override any of the services provided by classes:
__call__ which is used to create instances
__getattribute__ which is used to lookup attributes and methods on a class
__setattr__ which controls setting attributes
__repr__ which controls how the class is diplayed
In summary, metaclasses are used when you need to control how classes are created or when you need to alter any of the services provided by classes.

If you for whatever reason want to do stuff like Class[x], x in Class etc., you have to use metaclasses:
class Meta(type):
def __getitem__(cls, x):
return x ** 2
def __contains__(cls, x):
return int(x ** (0.5)) == x ** 0.5
# Python 2.x
class Class(object):
__metaclass__ = Meta
# Python 3.x
class Class(metaclass=Meta):
pass
print Class[2]
print 4 in Class

check the link Meta Class Made Easy to know how and when to use meta class.

Dynamic sub-classing in Python

I have a number of atomic classes (Components/Mixins, not really sure what to call them) in a library I'm developing, which are meant to be subclassed by applications. This atomicity was created so that applications can only use the features that they need, and combine the components through multiple inheritance.
However, sometimes this atomicity cannot be ensured because some component may depend on another one. For example, imagine I have a component that gives a graphical representation to an object, and another component which uses this graphical representation to perform some collision checking. The first is purely atomic, however the latter requires that the current object already subclassed this graphical representation component, so that its methods are available to it. This is a problem, because we have to somehow tell the users of this library, that in order to use a certain Component, they also have to subclass this other one. We could make this collision component sub class the visual component, but if the user also subclasses this visual component, it wouldn't work because the class is not on the same level (unlike a simple diamond relationship, which is desired), and would give the cryptic meta class errors which are hard to understand for the programmer.
Therefore, I would like to know if there is any cool way, through maybe metaclass redefinition or using class decorators, to mark these unatomic components, and when they are subclassed, the additional dependency would be injected into the current object, if its not yet available. Example:
class AtomicComponent(object):
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent(object):
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
Can someone give me an hint on how I can do this? or if it is even possible...
edit:
Since it is debatable that the meta class solution is the best one, I'll leave this unaccepted for 2 days.
Other solutions might be to improve error messages, for example, doing something like UserClass2 would give an error saying that UnAtomicComponent already extends this component. This however creates the problem that it is impossible to use two UnAtomicComponents, given that they would subclass object on different levels.

"Metaclasses"
This is what they are for! At time of class creation, the class parameters run through the
metaclass code, where you can check the bases and change then, for example.
This runs without error - though it does not preserve the order of needed classes
marked with the "depends" decorator:
class AutoSubclass(type):
def __new__(metacls, name, bases, dct):
new_bases = set()
for base in bases:
if hasattr(base, "_depends"):
for dependence in base._depends:
if not dependence in bases:
new_bases.add(dependence)
bases = bases + tuple(new_bases)
return type.__new__(metacls, name, bases, dct)
__metaclass__ = AutoSubclass
def depends(*args):
def decorator(cls):
cls._depends = args
return cls
return decorator
class AtomicComponent:
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent:
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
(I removed inheritance from "object", as I declared a global __metaclass__ variable. All classs will still be new style class and have this metaclass. Inheriting from object or another class does override the global __metaclass__variable, and a class level __metclass__ will have to be declared)
-- edit --
Without metaclasses, the way to go is to have your classes to properly inherit from their dependencies. Tehy will no longer be that "atomic", but, since they could not work being that atomic, it may be no matter.
In the example bellow, classes C and D would be your User classes:
>>> class A(object): pass
...
>>> class B(A, object): pass
...
>>>
>>> class C(B): pass
...
>>> class D(B,A): pass
...
>>>

A django model that subclasses an abc, gives a metaclass conflict

I have a following model and abstract base class
import abc
from django.db import models
class AbstractBase():
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def my_method(self):
return
class MyModel(models.Model, AbstractBase):
#abc.abstractmethod
def my_method(self):
return 1
But I am getting the following error.
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases
I think the problem here is (As it is described here http://code.activestate.com/recipes/204197-solving-the-metaclass-conflict/) that two base class has two different metaclasses so python cannot decide which metaclass to use for child object.
In order to solve this I removed multiple inheritence and use following register method to register child class
abc.register(Child)
But I did not really like this approach since it looks like monkey patching.
Is there another way to solve this problem?
I try to assign Model metaclass to Child explicitly but it did not work.
I am not looking for a way to solve it by writing code. I think this must be solved by changing my class structure.

Apart from creating a new metaclass that inherits from both ABCMeta and ModelBase, or making ABCMeta inherit from ModelBase, there isn't much you can do.
However, possibly a different registration pattern might be appropriate? Maybe something like contrib.admin.autodiscover? Or a class decorator? Or a loop at the bottom of the .py file which calls register on the appropriate classes (ex, for var in globals().values(): if isinstance(var, type) and issubclass(var, AbastractBase): register(var))?
Edit: D'oh. I'd assumed that ABCMeta was an example, not ABCMeta. That's what I get for browsing StackOverflow on too little sleep.