Develop Reference

Django: Custom Metaclass Inheriting From And Extending `ModelBase`

I am trying to do some metaclass hocus-pocus. I want my own Metaclass
to inherit from ModelBase and then I want to add additional logic by
extending its __new__ method. However I think there is something
strange happening with the MRO/inheritance order in the way I'm using it.
Here is the basic situation:
from django.db.models import Model, ModelBase
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
# As I am trying to extend `ModelBase`, I was expecting this
# call to `super` to give me the return value from here:
# https://github.com/django/django/blob/master/django/db/models/base.py#L300
# And that I would be able to access everyhing in `_meta` with
# `clsobj._meta`. But actually this object is
# `MyAbstractModel` and has no `_meta` property so I'm pretty
# sure `__new__` isn't being called on `ModelBase` at all at
# this point.
clsobj = super().__new__(cls, name, bases, attrs)
# Now, I want to have access to the `_meta` property setup by
# `ModelBase` so I can dispatch on the data in there. For
# example, let's do something with the field definitions.
for field in clsobj._meta.get_fields():
do_stuff_with_fields()
return clsobj
class MyAbstractModel(metaclass=CustomMetaclass):
"""This model is abstract because I only want the custom metaclass
logic to apply to those models of my choosing and I don't want to
be able to instantiate it directly. See the class definitions below.
"""
class Meta:
abstract = True
class MyModel(Model):
"""Regular model, will be derived from metaclass `ModelBase` as usual.
"""
pass
class MyCustomisedModel(MyAbstractModel):
"""This model should enjoy the logic defined by our extended `__new__` method.
"""
pass
Any ideas why __new__ on ModelBase isn't being called by
CustomMetaClass? How can I correctly extend ModelBase in this way? I'm pretty sure metaclass inheritance is possible
but seems like I'm missing something...

The way to get a clsobj with the _meta attribute is as simple as:
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
bases = (Model,)
clsobj = super().__new__(cls, name, bases, attrs)
for field in clsobj._meta.get_fields():
do_stuff_with_fields()
return clsobj
And we can do the same thing with MyAbstractModel(Model, metaclass=CustomMetaclass).
But, ultimate success here still depends on the kind of work we intend to do in the __new__ method. If we want to somehow introspect and work with the class's fields using metaprogramming, we need to be aware that we are trying to rewrite the class using __new__ at import time and thus (because this is Django) the app registry is not yet ready and this can cause exceptions to be raised if certain conditions arise (e.g. we are forbidden to access or work with reverse relations). This happens here even when Model is passed into __new__ as a base.
We can half-circumvent some of those problems by using the following non-public call to _get_fields (which Django does itself in certain places):
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
bases = (Model,)
clsobj = super().__new__(cls, name, bases, attrs)
for field in clsobj._meta._get_fields(reverse=False):
do_stuff_with_fields()
return clsobj
But depending on the scenario and what we are trying to achieve we might still hit problems; for example, we won't be able to access any reverse relations using our metaclass. So still no good.
To overcome this restriction we have to leverage signals in the app registry to make our classes as dynamic as we want them to be with full access to _meta.get_fields.
See this ticket: https://code.djangoproject.com/ticket/24231
The main takeaway being: "a Django model class is not something you are permitted to work with outside the context of a prepared app registry."

How do Django models work?

So I can create Django model like this:
from django.db import models
class Something(models.Model):
title = models.TextField(max_length=200)
and I can work with it like this:
thing = Something()
#set title
thing.title = "First thing"
#get title
thing.title
All works as it should but I'd like to understand HOW it works.
title = models.TextField(max_length=200)
in non-Django Python code above line defines class variable title of type models.TextField and I could access it also like this: thing.__class__.title(link)
But in Django when I create instance of Something I suddenly have a title attribute where I can get/set text. And cannot access it with thing.__class__.title So clearly when doing thing.title I'm not accessing class variable "title" but some generated attribute/property, or?
I know that fields ended up in thing._meta.fields but how? What's going on and how?
1, Does Django create property "title" behind the scenes?
2, What happened to class variable "title"?

I think its hard to beat what Django documentation has to say on this.
The Model class (see base.py) has a metaclass attribute that defines ModelBase (also in base.py) as the class to use for creating new classes. So ModelBase.new is called to create this new Example class. It is important to realise that we are creating the class object here, not an instance of it. In other words, Python is creating the thing that will eventually be bound to the Example name in our current namespace.
Basically a metaclass defines how a class itself will be created. During creation, additional attributes/methods/anything can be bound to that class. The example this stackoverflow answer gives, capitalizes all the attributes of a class
# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type):
# __new__ is the method called before __init__
# it's the method that creates the object and returns it
# while __init__ just initializes the object passed as parameter
# you rarely use __new__, except when you want to control how the object
# is created.
# here the created object is the class, and we want to customize it
# so we override __new__
# you can do some stuff in __init__ too if you wish
# some advanced use involves overriding __call__ as well, but we won't
# see this
def __new__(upperattr_metaclass, future_class_name,
future_class_parents, future_class_attr):
attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
uppercase_attr = dict((name.upper(), value) for name, value in attrs)
return type(future_class_name, future_class_parents, uppercase_attr)
In a similar way, Django's metaclass for Models can digest the attributes you've applied to the class and add various useful attributes for validation/etc, including even methods and what-not.

python is extremely powerfull and permit the developer to use intrespection.
django use a lot of metaclass. and it seem that models.Model use it too. see in
django\db\models\base.py
class Model(object):
__metaclass__ = ModelBase
i think the metaclass just take the classes attributes such a the Field and for all new instance for these Model subclass,create the apropriate variable.
1) yes, django create the instance variable of property "title" automaticaly
2) in the same way, the metaclass move the fields into the meta class...

Access class object builtin values from class (not class member) scope

How can you access class values from within the top level class scope? What I mean by that is, how do you do something like:
class FooClass(object):
zeroith_base = __bases__[0]
.
.
.
What I'm specifically trying to do in this case is derive the metaclasses of all base classes to dynamically generate a metaclass that subclasses all the base classes' metaclasses to get past metclass conflict problems. I found http://code.activestate.com/recipes/204197-solving-the-metaclass-conflict/, and while all the concepts make sense to me, the actual code of the recipe is just beyond my ability to follow it. I don't want to use code I can't understand though, so instead, I tried to implement my own, more rudimentary system, but I'm stuck at square one trying to inspect the class object during creation.

You cannot inspect a class prior to its creation, and it has not yet been created yet until the suite of statements, or class body, have finished executing. The first time you have access to this information would be in the MetaClass.__new__ method of the class creating the class in question, or the execution of the thing creating the class in question, which technically need not be a meta-class or a class at all (as in the example below).
Here is a very rough prototype that probably does not work in all cases, but works in the simple case, and is probably easier to follow than the recipe.
def meta_class_synthesize(name, bases, attrmap):
seen = set()
seen_add = seen.add
metas = [type(base) for base in bases]
metas = tuple([
meta for meta in metas
if meta is not type and meta not in seen and not seen_add(meta)])
if not metas:
return type(name, bases, attrmap)
elif len(metas) == 1:
return metas[0](name, bases, attrmap)
newmeta_name = "__".join(meta.__name__ for meta in metas)
newmeta = type(newmeta_name, metas, {})
return newmeta(name, bases, attrmap)
class M_A(type):
pass
class M_B(type):
pass
class A:
__metaclass__ = M_A
class B:
__metaclass__ = M_B
class C(A, B):
__metaclass__ = meta_class_synthesize
print type(C) # prints "<class '__main__.M_A__M_B'>"

You'll find that __bases__ is not part of the class namespace. The class namespace is passed to the metaclass as the third parameter; the bases are passed as the second parameter. They are totally separate until the class is created.
So what you'll need to do is write a metaclass that synthesizes the metaclass you want, then uses that to create the class. I have no idea if that'll actually work, but I can't see any reason why it wouldn't.

Python: Class factory using user input as class names

I want to add class atttributes to a superclass dynamically. Furthermore, I want to create classes that inherit from this superclass dynamically, and the name of those subclasses should depend on user input.
There is a superclass "Unit", to which I can add attributes at runtime. This already works.
def add_attr (cls, name, value):
setattr(cls, name, value)
class Unit(object):
pass
class Archer(Unit):
pass
myArcher = Archer()
add_attr(Unit, 'strength', 5)
print "Strenght ofmyarcher: " + str(myArcher.strength)
Unit.strength = 2
print "Strenght ofmyarcher: " + str(myArcher.strength)
This leads to the desired output:
Strenght ofmyarcher: 5
Strenght ofmyarcher: 2
But now I don't want to predefine the subclass Archer, but I'd rather let the user decide how to call this subclass. I've tried something like this:
class Meta(type, subclassname):
def __new__(cls, subclassname, bases, dct):
return type.__new__(cls, subclassname, Unit, dct)
factory = Meta()
factory.__new__("Soldier")
but no luck. I guess I haven't really understood what new does here.
What I want as a result here is
class Soldier(Unit):
pass
being created by the factory. And if I call the factory with the argument "Knight", I'd like a class Knight, subclass of Unit, to be created.
Any ideas? Many thanks in advance!
Bye
-Sano

To create a class from a name, use the class statement and assign the name. Observe:
def meta(name):
class cls(Unit):
pass
cls.__name__ = name
return cls
Now I suppose I should explain myself, and so on. When you create a class using the class statement, it is done dynamically-- it is equivalent of calling type().
For example, the following two snippets do the same thing:
class X(object): pass
X = type("X", (object,), {})
The name of a class-- the first argument to type-- is assigned to __name__, and that's basically the end of that (the only time __name__ is itself used is probably in the default __repr__() implementation). To create a class with a dynamic name, you can in fact call type like so, or you can just change the class name afterward. The class syntax exists for a reason, though-- it's convenient, and it's easy to add to and change things later. If you wanted to add methods, for example, it would be
class X(object):
def foo(self): print "foo"
def foo(self): print "foo"
X = type("X", (object,), {'foo':foo})
and so on. So I would advise using the class statement-- if you had known you could do so from the beginning, you likely would have done so. Dealing with type and so on is a mess.
(You should not, by the way, call type.__new__() by hand, only type())

Have a look at the type() builtin function.
knight_class = type('Knight', (Unit,), {})
First parameter: Name of new class
Second parameter: Tuple of parent classes
Third parameter: dictionary of class attributes.
But in your case, if the subclasses don't implement a different behaviour, maybe giving the Unit class a name attribute is sufficient.

How to change baseclass

I have a class which is derived from a base class, and have many many lines of code
e.g.
class AutoComplete(TextCtrl):
.....
What I want to do is change the baseclass so that it works like
class AutoComplete(PriceCtrl):
.....
I have use for both type of AutoCompletes and may be would like to add more base classes, so how can I do it dynamically?
Composition would have been a solution, but I do not want to modify code a lot.
any simple solutions?

You could have a factory for your classes:
def completefactory(baseclass):
class AutoComplete(baseclass):
pass
return AutoComplete
And then use:
TextAutoComplete = completefactory(TextCtrl)
PriceAutoComplete = completefactory(PriceCtrl)
On the other hand depending on what you want to achieve and how your classes look, maybe AutoComplete is meant to be a mixin, so that you would define TextAutoComplete with:
class TextAutocomplete(TextCtrl, AutoComplete):
pass

You could use multiple inheritance for this:
class AutoCompleteBase(object):
# code for your class
# remember to call base implementation with super:
# super(AutoCompleteBase, self).some_method()
class TextAutoComplete(AutoCompleteBase, TextCtrl):
pass
class PriceAutoComplete(AutoCompleteBase, PriceCtrl):
pass
Also, there's the option of a metaclass:
class BasesToSeparateClassesMeta(type):
"""Metaclass to create a separate childclass for each base.
NB: doesn't create a class but a list of classes."""
def __new__(self, name, bases, dct):
classes = []
for base in bases:
cls = type.__new__(self, name, (base,), dct)
# Need to init explicitly because not returning a class
type.__init__(cls, name, (base,), dct)
classes.append(cls)
return classes
class autocompletes(TextCtrl, PriceCtrl):
__metaclass__ = BasesToSeparateClassesMeta
# Rest of the code
TextAutoComplete, PriceAutoComplete = autocompletes
But I'd still suggest the class factory approach already suggested, one level of indentation really isn't that big of a deal.

You could modify the __bases__ tuple. For example you could add another baseclass:
AutoComplete.__bases__ += (PriceCtrl,)
But in general I would try to avoid such hacks, it quickly creates a terrible mess.