I am trying to do some metaclass hocus-pocus. I want my own Metaclass
to inherit from ModelBase and then I want to add additional logic by
extending its __new__ method. However I think there is something
strange happening with the MRO/inheritance order in the way I'm using it.
Here is the basic situation:
from django.db.models import Model, ModelBase
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
# As I am trying to extend `ModelBase`, I was expecting this
# call to `super` to give me the return value from here:
# https://github.com/django/django/blob/master/django/db/models/base.py#L300
# And that I would be able to access everyhing in `_meta` with
# `clsobj._meta`. But actually this object is
# `MyAbstractModel` and has no `_meta` property so I'm pretty
# sure `__new__` isn't being called on `ModelBase` at all at
# this point.
clsobj = super().__new__(cls, name, bases, attrs)
# Now, I want to have access to the `_meta` property setup by
# `ModelBase` so I can dispatch on the data in there. For
# example, let's do something with the field definitions.
for field in clsobj._meta.get_fields():
do_stuff_with_fields()
return clsobj
class MyAbstractModel(metaclass=CustomMetaclass):
"""This model is abstract because I only want the custom metaclass
logic to apply to those models of my choosing and I don't want to
be able to instantiate it directly. See the class definitions below.
"""
class Meta:
abstract = True
class MyModel(Model):
"""Regular model, will be derived from metaclass `ModelBase` as usual.
"""
pass
class MyCustomisedModel(MyAbstractModel):
"""This model should enjoy the logic defined by our extended `__new__` method.
"""
pass
Any ideas why __new__ on ModelBase isn't being called by
CustomMetaClass? How can I correctly extend ModelBase in this way? I'm pretty sure metaclass inheritance is possible
but seems like I'm missing something...
The way to get a clsobj with the _meta attribute is as simple as:
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
bases = (Model,)
clsobj = super().__new__(cls, name, bases, attrs)
for field in clsobj._meta.get_fields():
do_stuff_with_fields()
return clsobj
And we can do the same thing with MyAbstractModel(Model, metaclass=CustomMetaclass).
But, ultimate success here still depends on the kind of work we intend to do in the __new__ method. If we want to somehow introspect and work with the class's fields using metaprogramming, we need to be aware that we are trying to rewrite the class using __new__ at import time and thus (because this is Django) the app registry is not yet ready and this can cause exceptions to be raised if certain conditions arise (e.g. we are forbidden to access or work with reverse relations). This happens here even when Model is passed into __new__ as a base.
We can half-circumvent some of those problems by using the following non-public call to _get_fields (which Django does itself in certain places):
class CustomMetaclass(ModelBase):
def __new__(cls, name, bases, attrs):
bases = (Model,)
clsobj = super().__new__(cls, name, bases, attrs)
for field in clsobj._meta._get_fields(reverse=False):
do_stuff_with_fields()
return clsobj
But depending on the scenario and what we are trying to achieve we might still hit problems; for example, we won't be able to access any reverse relations using our metaclass. So still no good.
To overcome this restriction we have to leverage signals in the app registry to make our classes as dynamic as we want them to be with full access to _meta.get_fields.
See this ticket: https://code.djangoproject.com/ticket/24231
The main takeaway being: "a Django model class is not something you are permitted to work with outside the context of a prepared app registry."
Related
I saw a couple of classmethod that doesn't have a decorator #classmethod. What's the reason about it?
https://github.com/django/django/blob/3.0/django/db/models/base.py#L320
https://github.com/django/django/blob/3.0/django/db/models/manager.py#L20
The items over which you talk about are used as metaclasses [Python-doc]. One could say that a meta-class is the type of the type. If we for example take a look at the ModelBase soure code [GitHub], we see:
class ModelBase(type):
"""Metaclass for all models."""
def __new__(cls, name, bases, attrs, **kwargs):
super_new = super().__new__
# …
It thus inherits from type, which is the basic base class of meta-classes.
Here the "self" object is thus the the class itself that is analyzed, updated, etc. While one does not per se needs to use cls, it is common that in meta-classes what would be the self of an ordinary class, is named cls in a meta-class definition, to stress the fact that is the class object itself that we are manipulating.
I've got a mixin class, that adds some functionality to inheriting classes, but the mixin requires some class attributes to be present, for simplicity let's say only one property handlers. So this would be the usage of the mixin:
class Mixin:
pass
class Something(Mixin):
handlers = {}
The mixin can't function without this being defined, but I really don't want to specify the handlers in every class that I want to use the mixin with. So I solved this by writing a metaclass:
class MixinMeta:
def __new__(mcs, *args, **kwargs):
cls = super().__new__(mcs, *args, **kwargs)
cls.handlers = {}
return cls
class Mixin(metaclass=MixinMeta):
pass
And this works exactly how I want it to. But I'm thinking this can become a huge problem, since metaclasses don't work well together (I read various metaclass conflicts can only be solved by creating a new metaclass that resolves those conflicts).
Also, I don't want to make the handlers property a property of the Mixin class itself, since that would mean having to store handlers by their class names inside the Mixin class, complicating the code a bit. I like having each class having their handlers on their own class - it makes working with them simpler, but clearly this has drawbacks.
My question is, what would be a better way to implement this? I'm fairly new to metaclasses, but they seem to solve this problem well. But metaclass conflicts are clearly a huge issue when dealing with complex hierarchies without having to define various metaclasses just to resolve those conflicts.
Your problem is very real, and Python folks have thought of this for Python 3.6 (still unrealsed) on. For now (up to Python 3.5), if your attributes can wait to exist until your classes are first instantiated, you could put cod to create a (class) attribute on the __new__ method of your mixin class itself - thus avoiding the (extra) metaclass:
class Mixin:
def __new__(cls):
if not hasattr(cls, handlers):
cls.handlers = {}
return super().__new__(cls)
For Python 3.6 on, PEP 487 defines a __init_subclass__ special method to go on the mixin class body. This special method is not called for the mixin class itself, but will be called at the end of type.__new__ method (the "root" metaclass) for each class that inherits from your mixin.
class Mixin:
def __init_subclass__(cls, **kwargs):
cls.handlers = {}
return super().__init_subclass__(**kwargs)
As per the PEP's background text, the main motivation for this is exactly what led you to ask your question: avoid the need for meta-classes when simple customization of class creation is needed, in order to reduce the chances of needing different metaclasses in a project, and thus triggering a situation of metaclass conflict.
I am making a metaclass where I customize the __new__ method to customize how the new class will be created according to the provided values in kwargs. This probably makes more sense in an example:
class FooMeta(type):
def __new__(cls, name, bases, kwargs):
# do something with the kwargs...
# for example:
if 'foo' in kwargs:
kwargs.update({
'fooattr': 'foovalue'
})
return super(FooMeta, cls).__new__(cls, name, bases, kwargs)
My problem is how can I make this compatible for both Python 2 and 3. Six is a great compatibility library however it does not solve my problem. You would use it as:
class FooClass(six.with_metaclass(FooMeta, FooBase)):
pass
This does not work because six creates a new base class by using the given metaclass. Below is the six's code (link) (as of 1.3):
def with_metaclass(meta, base=object):
return meta("NewBase", (base,), {})
As a result, my __new__ method will be called without any kwargs hence essentially "breaking" my function. The question is how can I accomplish the behavior I want without breaking the compatibility for both Python 2 and 3. Please note that I don't need Python 2.6 support so if something is possible for only Python 2.7.x and 3.x I am fine with that.
Background
I need this to work in Django. I want to create a model factory metaclass which depending on the model attributes will customize how the model is created.
class ModelMeta(ModelBase):
def __new__(cls, name, bases, kwargs):
# depending on kwargs a different model is constructed
# ...
class FooModel(six.with_metaclass(ModelMeta, models.Model):
# some attrs here to be passed to the metaclass
If I understand you right, there is no problem and it already works. Did you actually check whether your metaclass has the desired effect? Because I think it already does.
The class returned by with_metaclass is not meant to play the role of your class FooClass. It is a dummy class used as the base class of your class. Since this base class is of metaclass FooMeta, your derived class will also have metaclass FooMeta and so the metaclass will be called again with the appropriate arguments.
class FooMeta(type):
def __new__(cls, name, bases, attrs):
# do something with the kwargs...
# for example:
if 'foo' in attrs:
attrs['fooattr'] = 'foovalue'
return super(FooMeta, cls).__new__(cls, name, bases, attrs)
class FooBase(object):
pass
class FooClass(with_metaclass(FooMeta, FooBase)):
foo = "Yes"
>>> FooClass.fooattr
'foovalue'
Incidentally, it is confusing to call the third argument of your metaclass kwargs. It isn't really keyword arguments. It is the __dict__ of the class to be created --- that is, the class's attributes and their values. In my experience this attribute is conventionally called attrs or perhaps dct (see e.g., here and here).
I'm having a load of confusion between the __metaclass__ property of a class and actual inheritance, and how __new__ is called in either of these scenarios. My issue comes from digging through some model code in the django framework.
Let's say I wanted to append an attribute to a class as it's defined in the child's Meta subclass:
class Parent(type):
def __new__(cls, name, base, attrs):
meta = attrs.pop('Meta', None)
new_class = super(Parent, cls).__new__(cls, name, base, attrs)
new_class.fun = getattr(meta, 'funtime', None)
return new_class
I don't understand why the actual __new__ method is called in django's code, but when I try to code something like that it doesn't work.
From what I've experienced, the following does not actually call the __new__ method of the parent:
class Child(Parent):
class Meta:
funtime = 'yaaay'
C = Child()
When I try to do this it complains with the TypeError:
TypeError: __new__() takes exactly 4 arguments (1 given)
However the source code I have been looking at appears to work in that way.
I understand that it could be done with a metaclass:
class Child(object):
__metaclass__ = Parent
But I don't understand why their way works for them and not for me, since the non __metaclass___ would be cleaner for making a distributable module.
Could somebody please point me in the right direction on what I'm missing?
Thanks!
In django, Model is not a metaclass. Actually the metaclass is ModelBase. That's why their way works and your way doesn't work.
Moreover, the latest django used a helper function, six.with_metaclass, to wrap 'ModelBase'.
If we want to follow django's style, Parent and Child class will look like
def with_metaclass(meta, base=object):
"""Create a base class with a metaclass."""
return meta("NewBase", (base,), {})
class ParentBase(type):
def __new__(cls, name, base, attrs):
meta = attrs.pop('Meta', None)
new_class = super(ParentBase, cls).__new__(cls, name, base, attrs)
new_class.fun = getattr(meta, 'funtime', None)
return new_class
class Parent(with_metaclass(ParentBase)):
pass
class Child(Parent):
class Meta:
funtime = 'yaaay'
c = Child()
>>> c.fun
'yaaay'
Let us focus on Parent. It is almost equivalent to
NewBase = ParentBase("NewBase", (object,), {})
class Parent(NewBase):
pass
The key is how to understand ParentBase("NewBase", (object,), {}).
Let us recall type().
type(name, bases, dict)
With three arguments, return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the name attribute; the bases tuple itemizes the base classes and becomes the bases attribute; and the dict dictionary is the namespace containing definitions for class body and becomes the dict attribute. For example, the following two statements create identical type objects:
Since ParentBase is a metaclass, a subclass of type. Therefore, ParentBase("NewBase", (object,), {}) is very similar to type("NewBase", (object,), {}). In this case, the only difference is the class created dynamically is not an instance of type, but ParentBase.
In other word, the metaclass of NewBase is ParentBase. Parent is equivalent to
class NewBase(object):
__metaclass__ = ParentBase
class Parent(NewBase):
pass
Finally, we got a __metaclass__.
in a metaclass that extends type, __new__ is used to create a class.
in a class, __new__ is used to create an instance.
metaclass is a class that creates a class. you're confused of class inheritance and metaclass.
your Child class inherits Parent and you want to create an instance of Child. however, Parent being a metaclass means Parent.__new__ shouldn't be used to create an instance of a class.
I think I finally figured out they need to use this DeclarativeFieldsMetaclass (to turn the class fields into instance variables and maintain their order with an ordered/sorted dict). However, I'm still not quite sure why they opted to use a BaseForm rather than implementing everything directly within the Form class?
They left a comment,
class Form(BaseForm):
"A collection of Fields, plus their associated data."
# This is a separate class from BaseForm in order to abstract the way
# self.fields is specified. This class (Form) is the one that does the
# fancy metaclass stuff purely for the semantic sugar -- it allows one
# to define a form using declarative syntax.
# BaseForm itself has no way of designating self.fields.
But I don't really understand it. "In order to abstract the way self.fields is specified" -- but Python calls DeclarativeFieldsMetaclass.__new__ before Form.__init__, so they could have taken full advantage of self.fields inside Form.__init__ as is; why do they need an extra layer of abstraction?
I think reason is simpl,e with BaseForm alone you can't define fields using a decalrative syntax i.e.
class MyForm(Form):
field_xxx = form.TextField(...)
field_nnn _ form.IntegerField(...)
For such thing to work for should have a metaclass DeclarativeFieldsMetaclass which is set in Form only, they did that because
This is a separate class from BaseForm
in order to abstract the way,
self.fields is specifie
so now you can write WierdForm class in which fields can be defined may be in some wierd way e.g. passing params to class object, point is all the API is in BaseForm and Form class just provides an easy to defined fields.
Summary: IMO django preferred to introduce another layer so that if needed different type of field declaration can be implemented, at-least it keeps the non core functionality of forms separate.
Source:
class MetaForm(type):
def __new__(cls, name, bases, attrs):
print "%s: %s" % (name, attrs)
return type.__new__(cls, name, bases, attrs)
class BaseForm(object):
my_attr = 1
def __init__(self):
print "BaseForm.__init__"
class Form(BaseForm):
__metaclass__ = MetaForm
def __init__(self):
print "Form.__init__"
class CustomForm(Form):
my_field = 2
def __init__(self):
print "CustomForm.__init__"
f = CustomForm()
Output:
Form: {'__module__': '__main__', '__metaclass__': <class '__main__.MetaForm'>, '__init__':<function __init__ at 0x0227E0F0>}
CustomForm: {'__module__': '__main__', 'my_field': 2, '__init__': <function __init__ at 0x0227E170>}
CustomForm.__init__
Looks like MetaForm.__new__ is called twice. Once for Form and once for CustomForm, but never for BaseForm. By having a clean (empty) Form class, there won't be any extraneous attributes to loop over. It also means that you can define Fields inside the BaseForm that could be used for internal use, but avoid rendering.