Develop Reference

Subclass avoiding parent's metaclass

Say I have a third-party library where a metaclass requires me to implement something. But I want to have an intermediate "abstract" subclass that doesn't. How can I do this?
Consider this to be a very minimal example of what third-party library has:
class ServingMeta(type):
def __new__(cls, name, bases, classdict):
if any(isinstance(b, ServingMeta) for b in bases):
if "name" not in classdict:
# Actual code fails for a different reason,
# but the logic is the same.
raise TypeError(f"Class '{name}' has no 'name' attribute")
return super().__new__(cls, name, bases, classdict)
class Serving(object, metaclass=ServingMeta):
def shout_name(self):
return self.name.upper()
I cannot modify the code above. It's an external dependency (and I don't want to fork it).
The code is meant to be used this way:
class Spam(Serving):
name = "SPAM"
spam = Spam()
print(spam.shout_name())
However, I happen to have a lot of spam, and I want to introduce a base class with the common helper methods. Something like this:
class Spam(Serving):
def thrice(self):
return " ".join([self.shout_name()] * 3)
class LovelySpam(Spam):
name = "lovely spam"
class WonderfulSpam(Spam):
name = "wonderful spam"
Obviously, this doesn't work and fails with the well-expected TypeError: Class 'SpamBase' has no 'name' attribute declared. Would third-party library had a SpamBase class without a metaclass, I could've subclassed that - but no such luck this time (I've mentioned this inconvenience to the library authors).
I can make it a mixin:
class SpamMixin(object):
def thrice(self):
return " ".join([self.shout_name()] * 3)
class LovelySpam(SpamMixin, Serving):
name = "lovely spam"
class WonderfulSpam(SpamMixin, Serving):
name = "wonderful spam"
However, this makes me and my IDE cringe a little, as it quickly becomes cumbersome to repeat SpamMixin everywhere and also because object has no shout_name attribute (and I don't want to silence analysis tools). In short, I just don't like this approach.
What else can I do?
Is there a way to get a metaclass-less version of Serving? I think of something like this:
ServingBase = remove_metaclass(Serving)
class Spam(ServingBase, metaclass=ServingMeta):
...
But don't know how to actually implement remove_metaclass and whenever it's even reasonably possible (of course, it must be doable, with some introspection, but it could require more arcane magic than I can cast).
Any other suggestions are also welcomed. Basically, I want to have my code DRY (one base class to rule them all), and have my linter/code analysis icons all green.

The mixin approach is the correct way to go. If your IDE "cringe" that is a deffect on that tool - just disable a little of the "features" that are obviously incorrect tunning when coding for a dynamic language like Python.
And this is not even about creating things dynamically, it is merely multiple-inheritance, which is supported by the language since forever. And one of the main uses of multiple-inheritance is exactly being able to create mixins just as this one you need.
Another inheritance-based workaround is to make your hierarchy one level deeper, and just introduce the metaclass after you come up with your mixin methods:
class Mixin(object):
def mimixin(self): ...
class SpamBase(Mixin, metaclass=ServingMeta):
name = "stub"
Or just addd the mixin in an intermediate subclass:
class Base(metaclass=Serving Meta):
name = "stub"
class MixedBase(Mixin, Base):
name = "stub"
class MyLovingSpam(MixedBase):
name = "MyLovingSpam"
If you don't want to be repeating the mixin=-base name in every class, that is the way to go.
"Removing" a metaclass just for the sake of having a late mixin is way over the top. Really. Broken. The way to do it wol e re-create the class dynamically, just as #vaultah mentions in the other answer, but doing that in an intermediate class is a thing you should not do. Doing that to please the IDE is something you should not do twice: messing with metaclasses is hard enough already. Removing things on inheritance/class creation that the language puts there naturally is something nasty (cf. this answer: How to make a class attribute exclusive to the super class ) . On the other hand, mixins and multiple inheritance are just natural.
Are you still there? I told you not to do so:
Now, onto your question - instead of "supressing the metaclass" in an intermediate class, it would be more feasible to inherit the metaclass you have there and change its behavior - so that it does not check for the constraints in specially marked classes - create an attribute for your use, like _skip_checking
class MyMeta(ServingMeta):
def __new__(metacls, name, bases, namespace):
if namespace.get("_skip_checking", False):
# hardcode call to "type" metaclass:
del namespace["_skip_checking"]
cls = type.__new__(metacls, name, bases, namespace)
else:
cls = super().__new__(metacls, name, bases, namespace)
return cls
# repeat for __init__ if needed.
class Base(metaclass=MyMeta):
_skip_checking = True
# define mixin methods
class LoveSpam(Base):
name = "LoveSpam"

There's really no direct way to remove the metaclass from a Python class, because the metaclass created that class. What you can try is re-create the class using a different metaclass, which doesn't have unwanted behaviour. For example, you could use type (the default metaclass).
In [6]: class Serving(metaclass=ServingMeta):
...: def shout_name(self):
...: return self.name.upper()
...:
In [7]: ServingBase = type('ServingBase', Serving.__bases__, dict(vars(Serving)))
Basically this takes the __bases__ tuple and the namespace of the Serving class, and uses them to create a new class ServingBase. N.B. this means that ServingBase will receive all bases and methods/attributes from Serving, some of which may have been added by ServingMeta.

How to dynamically change base class of instances at runtime?

This article has a snippet showing usage of __bases__ to dynamically change the inheritance hierarchy of some Python code, by adding a class to an existing classes collection of classes from which it inherits. Ok, that's hard to read, code is probably clearer:
class Friendly:
def hello(self):
print 'Hello'
class Person: pass
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
That is, Person doesn't inherit from Friendly at the source level, but rather this inheritance relation is added dynamically at runtime by modification of the __bases__attribute of the Person class. However, if you change Friendly and Person to be new style classes (by inheriting from object), you get the following error:
TypeError: __bases__ assignment: 'Friendly' deallocator differs from 'object'
A bit of Googling on this seems to indicate some incompatibilities between new-style and old style classes in regards to changing the inheritance hierarchy at runtime. Specifically: "New-style class objects don't support assignment to their bases attribute".
My question, is it possible to make the above Friendly/Person example work using new-style classes in Python 2.7+, possibly by use of the __mro__ attribute?
Disclaimer: I fully realise that this is obscure code. I fully realize that in real production code tricks like this tend to border on unreadable, this is purely a thought experiment, and for funzies to learn something about how Python deals with issues related to multiple inheritance.

Ok, again, this is not something you should normally do, this is for informational purposes only.
Where Python looks for a method on an instance object is determined by the __mro__ attribute of the class which defines that object (the M ethod R esolution O rder attribute). Thus, if we could modify the __mro__ of Person, we'd get the desired behaviour. Something like:
setattr(Person, '__mro__', (Person, Friendly, object))
The problem is that __mro__ is a readonly attribute, and thus setattr won't work. Maybe if you're a Python guru there's a way around that, but clearly I fall short of guru status as I cannot think of one.
A possible workaround is to simply redefine the class:
def modify_Person_to_be_friendly():
# so that we're modifying the global identifier 'Person'
global Person
# now just redefine the class using type(), specifying that the new
# class should inherit from Friendly and have all attributes from
# our old Person class
Person = type('Person', (Friendly,), dict(Person.__dict__))
def main():
modify_Person_to_be_friendly()
p = Person()
p.hello() # works!
What this doesn't do is modify any previously created Person instances to have the hello() method. For example (just modifying main()):
def main():
oldperson = Person()
ModifyPersonToBeFriendly()
p = Person()
p.hello()
# works! But:
oldperson.hello()
# does not
If the details of the type call aren't clear, then read e-satis' excellent answer on 'What is a metaclass in Python?'.

I've been struggling with this too, and was intrigued by your solution, but Python 3 takes it away from us:
AttributeError: attribute '__dict__' of 'type' objects is not writable
I actually have a legitimate need for a decorator that replaces the (single) superclass of the decorated class. It would require too lengthy a description to include here (I tried, but couldn't get it to a reasonably length and limited complexity -- it came up in the context of the use by many Python applications of an Python-based enterprise server where different applications needed slightly different variations of some of the code.)
The discussion on this page and others like it provided hints that the problem of assigning to __bases__ only occurs for classes with no superclass defined (i.e., whose only superclass is object). I was able to solve this problem (for both Python 2.7 and 3.2) by defining the classes whose superclass I needed to replace as being subclasses of a trivial class:
## T is used so that the other classes are not direct subclasses of object,
## since classes whose base is object don't allow assignment to their __bases__ attribute.
class T: pass
class A(T):
def __init__(self):
print('Creating instance of {}'.format(self.__class__.__name__))
## ordinary inheritance
class B(A): pass
## dynamically specified inheritance
class C(T): pass
A() # -> Creating instance of A
B() # -> Creating instance of B
C.__bases__ = (A,)
C() # -> Creating instance of C
## attempt at dynamically specified inheritance starting with a direct subclass
## of object doesn't work
class D: pass
D.__bases__ = (A,)
D()
## Result is:
## TypeError: __bases__ assignment: 'A' deallocator differs from 'object'

I can not vouch for the consequences, but that this code does what you want at py2.7.2.
class Friendly(object):
def hello(self):
print 'Hello'
class Person(object): pass
# we can't change the original classes, so we replace them
class newFriendly: pass
newFriendly.__dict__ = dict(Friendly.__dict__)
Friendly = newFriendly
class newPerson: pass
newPerson.__dict__ = dict(Person.__dict__)
Person = newPerson
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
We know that this is possible. Cool. But we'll never use it!

Right of the bat, all the caveats of messing with class hierarchy dynamically are in effect.
But if it has to be done then, apparently, there is a hack that get's around the "deallocator differs from 'object" issue when modifying the __bases__ attribute for the new style classes.
You can define a class object
class Object(object): pass
Which derives a class from the built-in metaclass type.
That's it, now your new style classes can modify the __bases__ without any problem.
In my tests this actually worked very well as all existing (before changing the inheritance) instances of it and its derived classes felt the effect of the change including their mro getting updated.

I needed a solution for this which:
Works with both Python 2 (>= 2.7) and Python 3 (>= 3.2).
Lets the class bases be changed after dynamically importing a dependency.
Lets the class bases be changed from unit test code.
Works with types that have a custom metaclass.
Still allows unittest.mock.patch to function as expected.
Here's what I came up with:
def ensure_class_bases_begin_with(namespace, class_name, base_class):
""" Ensure the named class's bases start with the base class.
:param namespace: The namespace containing the class name.
:param class_name: The name of the class to alter.
:param base_class: The type to be the first base class for the
newly created type.
:return: ``None``.
Call this function after ensuring `base_class` is
available, before using the class named by `class_name`.
"""
existing_class = namespace[class_name]
assert isinstance(existing_class, type)
bases = list(existing_class.__bases__)
if base_class is bases[0]:
# Already bound to a type with the right bases.
return
bases.insert(0, base_class)
new_class_namespace = existing_class.__dict__.copy()
# Type creation will assign the correct ‘__dict__’ attribute.
del new_class_namespace['__dict__']
metaclass = existing_class.__metaclass__
new_class = metaclass(class_name, tuple(bases), new_class_namespace)
namespace[class_name] = new_class
Used like this within the application:
# foo.py
# Type `Bar` is not available at first, so can't inherit from it yet.
class Foo(object):
__metaclass__ = type
def __init__(self):
self.frob = "spam"
def __unicode__(self): return "Foo"
# … later …
import bar
ensure_class_bases_begin_with(
namespace=globals(),
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
Use like this from within unit test code:
# test_foo.py
""" Unit test for `foo` module. """
import unittest
import mock
import foo
import bar
ensure_class_bases_begin_with(
namespace=foo.__dict__,
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
class Foo_TestCase(unittest.TestCase):
""" Test cases for `Foo` class. """
def setUp(self):
patcher_unicode = mock.patch.object(
foo.Foo, '__unicode__')
patcher_unicode.start()
self.addCleanup(patcher_unicode.stop)
self.test_instance = foo.Foo()
patcher_frob = mock.patch.object(
self.test_instance, 'frob')
patcher_frob.start()
self.addCleanup(patcher_frob.stop)
def test_instantiate(self):
""" Should create an instance of `Foo`. """
instance = foo.Foo()

The above answers are good if you need to change an existing class at runtime. However, if you are just looking to create a new class that inherits by some other class, there is a much cleaner solution. I got this idea from https://stackoverflow.com/a/21060094/3533440, but I think the example below better illustrates a legitimate use case.
def make_default(Map, default_default=None):
"""Returns a class which behaves identically to the given
Map class, except it gives a default value for unknown keys."""
class DefaultMap(Map):
def __init__(self, default=default_default, **kwargs):
self._default = default
super().__init__(**kwargs)
def __missing__(self, key):
return self._default
return DefaultMap
DefaultDict = make_default(dict, default_default='wug')
d = DefaultDict(a=1, b=2)
assert d['a'] is 1
assert d['b'] is 2
assert d['c'] is 'wug'
Correct me if I'm wrong, but this strategy seems very readable to me, and I would use it in production code. This is very similar to functors in OCaml.

This method isn't technically inheriting during runtime, since __mro__ can't be changed. But what I'm doing here is using __getattr__ to be able to access any attributes or methods from a certain class. (Read comments in order of numbers placed before the comments, it makes more sense)
class Sub:
def __init__(self, f, cls):
self.f = f
self.cls = cls
# 6) this method will pass the self parameter
# (which is the original class object we passed)
# and then it will fill in the rest of the arguments
# using *args and **kwargs
def __call__(self, *args, **kwargs):
# 7) the multiple try / except statements
# are for making sure if an attribute was
# accessed instead of a function, the __call__
# method will just return the attribute
try:
return self.f(self.cls, *args, **kwargs)
except TypeError:
try:
return self.f(*args, **kwargs)
except TypeError:
return self.f
# 1) our base class
class S:
def __init__(self, func):
self.cls = func
def __getattr__(self, item):
# 5) we are wrapping the attribute we get in the Sub class
# so we can implement the __call__ method there
# to be able to pass the parameters in the correct order
return Sub(getattr(self.cls, item), self.cls)
# 2) class we want to inherit from
class L:
def run(self, s):
print("run" + s)
# 3) we create an instance of our base class
# and then pass an instance (or just the class object)
# as a parameter to this instance
s = S(L) # 4) in this case, I'm using the class object
s.run("1")
So this sort of substitution and redirection will simulate the inheritance of the class we wanted to inherit from. And it even works with attributes or methods that don't take any parameters.

Dynamic sub-classing in Python

I have a number of atomic classes (Components/Mixins, not really sure what to call them) in a library I'm developing, which are meant to be subclassed by applications. This atomicity was created so that applications can only use the features that they need, and combine the components through multiple inheritance.
However, sometimes this atomicity cannot be ensured because some component may depend on another one. For example, imagine I have a component that gives a graphical representation to an object, and another component which uses this graphical representation to perform some collision checking. The first is purely atomic, however the latter requires that the current object already subclassed this graphical representation component, so that its methods are available to it. This is a problem, because we have to somehow tell the users of this library, that in order to use a certain Component, they also have to subclass this other one. We could make this collision component sub class the visual component, but if the user also subclasses this visual component, it wouldn't work because the class is not on the same level (unlike a simple diamond relationship, which is desired), and would give the cryptic meta class errors which are hard to understand for the programmer.
Therefore, I would like to know if there is any cool way, through maybe metaclass redefinition or using class decorators, to mark these unatomic components, and when they are subclassed, the additional dependency would be injected into the current object, if its not yet available. Example:
class AtomicComponent(object):
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent(object):
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
Can someone give me an hint on how I can do this? or if it is even possible...
edit:
Since it is debatable that the meta class solution is the best one, I'll leave this unaccepted for 2 days.
Other solutions might be to improve error messages, for example, doing something like UserClass2 would give an error saying that UnAtomicComponent already extends this component. This however creates the problem that it is impossible to use two UnAtomicComponents, given that they would subclass object on different levels.

"Metaclasses"
This is what they are for! At time of class creation, the class parameters run through the
metaclass code, where you can check the bases and change then, for example.
This runs without error - though it does not preserve the order of needed classes
marked with the "depends" decorator:
class AutoSubclass(type):
def __new__(metacls, name, bases, dct):
new_bases = set()
for base in bases:
if hasattr(base, "_depends"):
for dependence in base._depends:
if not dependence in bases:
new_bases.add(dependence)
bases = bases + tuple(new_bases)
return type.__new__(metacls, name, bases, dct)
__metaclass__ = AutoSubclass
def depends(*args):
def decorator(cls):
cls._depends = args
return cls
return decorator
class AtomicComponent:
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent:
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
(I removed inheritance from "object", as I declared a global __metaclass__ variable. All classs will still be new style class and have this metaclass. Inheriting from object or another class does override the global __metaclass__variable, and a class level __metclass__ will have to be declared)
-- edit --
Without metaclasses, the way to go is to have your classes to properly inherit from their dependencies. Tehy will no longer be that "atomic", but, since they could not work being that atomic, it may be no matter.
In the example bellow, classes C and D would be your User classes:
>>> class A(object): pass
...
>>> class B(A, object): pass
...
>>>
>>> class C(B): pass
...
>>> class D(B,A): pass
...
>>>

Why can't I change the __metaclass__ attribute of a class?

I have a weird and unusual use case for metaclasses where I'd like to change the __metaclass__ of a base class after it's been defined so that its subclasses will automatically use the new __metaclass__. But that oddly doesn't work:
class MetaBase(type):
def __new__(cls, name, bases, attrs):
attrs["y"] = attrs["x"] + 1
return type.__new__(cls, name, bases, attrs)
class Foo(object):
__metaclass__ = MetaBase
x = 5
print (Foo.x, Foo.y) # prints (5, 6) as expected
class MetaSub(MetaBase):
def __new__(cls, name, bases, attrs):
attrs["x"] = 11
return MetaBase.__new__(cls, name, bases, attrs)
Foo.__metaclass__ = MetaSub
class Bar(Foo):
pass
print(Bar.x, Bar.y) # prints (5, 6) instead of (11, 12)
What I'm doing may very well be unwise/unsupported/undefined, but I can't for the life of me figure out how the old metaclass is being invoked, and I'd like to least understand how that's possible.
EDIT: Based on a suggestion made by jsbueno, I replaced the line Foo.__metaclass__ = MetaSub with the following line, which did exactly what I wanted:
Foo = type.__new__(MetaSub, "Foo", Foo.__bases__, dict(Foo.__dict__))

The problem is the __metaclass__ attribute is not used when inherited, contrary to what you might expect. The 'old' metaclass isn't called either for Bar. The docs say the following about how the metaclass is found:
The appropriate metaclass is determined by the following precedence
rules:
If dict['__metaclass__'] exists, it is used.
Otherwise, if there is at least one base class, its metaclass is used (this looks for a __class__ attribute first and if not found,
uses its type).
Otherwise, if a global variable named __metaclass__ exists, it is used.
Otherwise, the old-style, classic metaclass (types.ClassType) is used.
So what is actually used as metaclass in your Bar class is found in the parent's __class__ attribute and not in the parent's __metaclass__ attribute.
More information can be found on this StackOverflow answer.

The metaclass information for a class is used at the moment it is created (either parsed as a class block, or dynamically, with a explicit call to the metaclass). It can't be changed because the metaclass usually does make changes at class creation time - the created class type is the metaclasse. Its __metaclass__ attribute is irrelevant once it is created.
However, it is possible to create a copy of a given class, and have the copy bear a different metclass than the original class.
On your example, if instead of doing:
Foo.__metaclass__ = MetaSub
you do:
Foo = Metasub("Foo", Foo.__bases__, dict(Foo.__dict__))
You will achieve what you intended. The new Foo is for all effects equal its predecessor, but with a different metaclass.
However, previously existing instances of Foo won't be considered an instance of the new Foo - if you need that, you better create the Foo copy with a different name instead.

Subclasses use the __metaclass__ of their parent.
The solution to your use-case is to program the parent's __metaclass__ so that it will have different behaviors for the parent than for its subclasses. Perhaps have it inspect the class dictionary for a class variable and implement different behaviors depending on its value
(this is the technique type uses to control whether or not instances are given a dictionary depending on the whether or not __slots__ is defined).

How to change baseclass

I have a class which is derived from a base class, and have many many lines of code
e.g.
class AutoComplete(TextCtrl):
.....
What I want to do is change the baseclass so that it works like
class AutoComplete(PriceCtrl):
.....
I have use for both type of AutoCompletes and may be would like to add more base classes, so how can I do it dynamically?
Composition would have been a solution, but I do not want to modify code a lot.
any simple solutions?

You could have a factory for your classes:
def completefactory(baseclass):
class AutoComplete(baseclass):
pass
return AutoComplete
And then use:
TextAutoComplete = completefactory(TextCtrl)
PriceAutoComplete = completefactory(PriceCtrl)
On the other hand depending on what you want to achieve and how your classes look, maybe AutoComplete is meant to be a mixin, so that you would define TextAutoComplete with:
class TextAutocomplete(TextCtrl, AutoComplete):
pass

You could use multiple inheritance for this:
class AutoCompleteBase(object):
# code for your class
# remember to call base implementation with super:
# super(AutoCompleteBase, self).some_method()
class TextAutoComplete(AutoCompleteBase, TextCtrl):
pass
class PriceAutoComplete(AutoCompleteBase, PriceCtrl):
pass
Also, there's the option of a metaclass:
class BasesToSeparateClassesMeta(type):
"""Metaclass to create a separate childclass for each base.
NB: doesn't create a class but a list of classes."""
def __new__(self, name, bases, dct):
classes = []
for base in bases:
cls = type.__new__(self, name, (base,), dct)
# Need to init explicitly because not returning a class
type.__init__(cls, name, (base,), dct)
classes.append(cls)
return classes
class autocompletes(TextCtrl, PriceCtrl):
__metaclass__ = BasesToSeparateClassesMeta
# Rest of the code
TextAutoComplete, PriceAutoComplete = autocompletes
But I'd still suggest the class factory approach already suggested, one level of indentation really isn't that big of a deal.

You could modify the __bases__ tuple. For example you could add another baseclass:
AutoComplete.__bases__ += (PriceCtrl,)
But in general I would try to avoid such hacks, it quickly creates a terrible mess.