I'm in the process of migrating from 2.7 to 3.x and I'm trying to understand the __prepare__ method for metaclasses introduced in PEP3115.
In most of the examples I've seen, implementations of this method ignore the parameters (name, bases, and **kwargs) simply returns a custom dictionary that does something interesting to the namespace provided to the __new__ and __init__ methods for the metaclass. Even the example in PEP3115 does nothing with the parameters.
I don't doubt that there is some good reason for the signature of __prepare__ but I haven't seen the use case.
What are some good examples that demonstrate the rational for making the signature of __prepare__ take these parameters?
__prepare__ will create a namespace for the class like you said, so we can do some logic inside of it like this:
class MyMeta(type):
#classmethod
def __prepare__(metacls, klass_name, bases):
namespace = {'s': 'my string',
'description': 'N/A'}
if klass_name.endswith('Base'):
namespace.update({'description': 'base class'})
return namespace
class KlassBase(metaclass=MyMeta):
def __init__(self, value):
self.value = value
class SubKlass(KlassBase):
def __init__(self, value):
super().__init__(value)
print(KlassBase(5).s)
print(KlassBase(5).description)
print(SubKlass(5).s)
print(SubKlass(5).description)
And you got:
my string
base class
my string
N/A
The reason why we don't do it, because same things could be done in other part of the meta class like : __new__, __init__, or be overrided by the latter. So most of time, we won't do it in __prepare__
The following is an image of the class creation work-flow, which is much more clearer:
[sorry I cannot find the original source of this pic]
When you look at __prepare__, and the state of Python class creation mechanism at the time, it is pretty much clear that what was really needed was a mechanism to enable attribute order preservation.
This way, one would be able to create classes that would describe data records with ordered fields, which is pretty much what humans would expect when describing a record. (Try to imagine a form that each time it is rendered, it shuffles the field ordes, so half the time you will be filling in the country you live in before the country).
Instead of a fixed mechanism to just enable the class body namespace to be an collections.OrderedDict, they came up with __prepare__ which enables this easily, with a single line returning a new OrderedDict instance.
But __prepare__ can have so many uses and abuses, that I think no one really thought of all the possibilities. The parameters ou mentioned are avaliable at the time it is called, and since it exists, there is no reason whatsoever for they not to be passed to the function. Why to cripple one of knowing the class' name inside the __prepare__ function?
So, it is just a powerful and flexible mechanism put in place, and not necessarily all possible use cases were thought of when making it. The "Ordered Attributes" thing n the other hand is so important that it became the default for Python 3.6, with PEP 520, even without any custom metaclass declaration.
__prepare__ receivog bases, for example, would allow one to pre-populate the namespace with certain objects that would be found in the superclass namespaces, overriding the inheritance attribute access, for example.
Or it could simply check the class name against a pre-existing registry, and either raise or pre-populate stuff from there.
One super-easy usage would be to pre-populate __name__ for example, so that class attributes could make use of it:
import collections
class M(type):
#classmethod
def __prepare__(metacls, name, bases):
ns = collections.OrderedDict()
ns["__name__"] = name
class T(metaclass=M):
__table__ = f"_{__name__.lower()}"
trivia
Due to the way functions work as methods in Python 3, one interesting related thing that is not documented anywhere is that if __prepare__ is not explicitly decorated to be a classmethod, it works as a staticmethod, and the target class name is passed directly in the first parameter to it.
Related
I have read answers for this question: What are metaclasses in Python? and this question: In Python, when should I use a meta class? and skimmed through documentation: Data model.
It is very possible I missed something, and I would like to clarify: is there anything that metaclasses can do that cannot be properly or improperly (unpythonic, etc) done with the help of other tools (decorators, inheritance, etc)?
That is a bit tricky to answer -
However, it is a very nice question to ask at this point, and there are certainly a few things that are easier to do with metaclasses.
So, first, I think it is important to note the things for which one used to need a metaclass in the past, and no longer needs to: I'd say that with the release of Python 3.6 and the inclusion of __init_subclass__ and __set_name__ dunder methods, a lot, maybe the majority of the cases I had always written a metaclass for (most of them for answering questions or in toy code - no one creates that many production-code metaclasses even in a lifetime as a programmer) became outdated.
Specially __init_subclass__ adds the convenience of being able to transform any attribute or method like class-decorators, but is automatically applied on inheritance, which does not happen with decorators.
I guess reading about it was a fator motivating your question - since most metaclasses found out in the wild deal with transforming these attributes in __new__ and __init__ metaclass methods.
However, note that if one needs to transform any attribute prior to having it included in the class, the metaclass __new__ method is the only place it can be done. In most cases, however, one can simply transform it in the final new class namespace.
Then, one version forward, in 3.7, we had __class_getitem__ implemented - since using the [ ] (__getitem__) operator directly on classes became popular due to typing annotations. Before that, one would have to create a metaclass with a __getitem__ method for the sole purpose of being able to indicate to the type-checker toolchain some extra information like generic variables.
One interesting possibility that did not exist in Python 2, was introduced in Python 3, then outdated, and now can only serve very specific cases is the use of the __prepare__ method on the metaclass:
I don't know if this is written in any official docs, but the obvious primary motivation for metaclass __prepare__ which allows one custom namespace for the class body, was to return an ordered dict, so that one could have ordered attributes in classes that would work as data entities. It turns out that also, from Python 3.6 on, class body namespaces where always ordered (which later on Python 3.7 were formalized for all Python dictionaries). However, although not needed for returning an OrderedDict anymore, __prepare__ is still aunique thing in the language in which it allows a custom mapping class to be used as namespace in a piece of Python code (even if that is limited to class bodies). For example, one can trivialy create an "auto-enumeration" metaclass by returning a
class MD(dict):
def __init__(self, *args, **kw):
super().__init__(*args, **kw)
self.counter = 0
def __missing__(self, key):
counter = self[key] = self.counter
self.counter += 1
return counter
class MC(type):
#classmethod
def __prepare__(mcls, name, bases, **kwd):
return MD()
class Colors(metaclass=MC):
RED
GREEN
BLUE
(an example similar to this is included in Luciano Ramalho's 'Fluent Python' 2nd edition)
The __call__ method on the metaclass is also peculiar: it control the calls to __new__ and __init__ whenever an instance of the class is created. There are recipes around that use this to create a "singleton" - I find those terrible and overkill: if I need a singleton, I just create an instance of the singleton class at module level. However, overriding typing.__call__ offers a level of control on class instantiation that may be hard to achieve on the class __new__ and __init__ themselves. But this definitely can be done by correctly keeping the desired states in the class object itself.
__subclasscheck__ and __instancecheck__: these are metaclass only methods, and the only workaround would be to make a class decorator that would re-create a class object so that it would be a "real" subclass of the intended base class. (and that is not always possible).
"hidden" class attributes: now, this can be useful, and is less known, as it derives from the language behavior itself: any attribute or method besides the dunder methods included in a metaclass can be used from a class, but from instances of that class. An example for this is the .register method in classes using abc.ABCMeta. This contrasts with ordinary classmethods which can be used normally from an instance.
And finally, any behavior defined with the dunder methods for a Python object can be implemented to work on classes if they are defined in the metaclass. So if you have any use case for "add-able" classes, or want a special repr for your classes, just implement __add__ or __repr__ on the metaclass: this behavior obviously can't be obtained by other means.
I think I got all covered there.
I've been learning about metaclasses, and I was wondering if it's possible to add a new attribute to every class that's defined in python, not just those which inherit explicitly from a custom metaclass.
I can add a new attribute explicitly using a custom metaclass like this
class NewAttrMeta(type):
def __new__(cls, name, bases, attrs):
attrs['new_attr'] = 'new_thing'
return super().__new__(cls, name, bases, attrs)
class A(metaclass=NewAttrMeta):
...
print(A.new_attr)
$ 'new_thing'
But is it possible to force a change like this on every class that's defined, not just the ones which explicitly inherit from your custom metaclass?
I thought maybe as all classes are of type type, if I overwrote type itself with my custom metaclass, then all new classes might then inherit from it. And as metaclasses are subclasses of type, then all classes defined that way would still be valid...
class NewAttrMeta(type):
def __new__(cls, name, bases, attrs):
attrs['new_attr'] = 'new_thing'
return super().__new__(cls, name, bases, attrs)
type = NewMagicMeta
But this only works if type is passed in explicitly again:
class A(type):
...
print(A.new_attr)
$ 'new_thing'
class B():
...
print(B.new_attr)
$ AttributeError: type object 'A' has no attribute 'new_attr'
Why on earth am I trying to do this? I wanted to see if I could locally implement a version of the rejected PEP 472: "Support for indexing with keyword arguments" by overriding the __getitem__ method of every class which defined any version of __getitem__. I'm only doing this for fun, so I would be interested in any insights or alternative ways to do that (the hackier the better!).
Python does not allow modification of built-in declared types. That
means that dictionaries, lists, classes defineds in extensions like
Numpy.ndarrays are "frozen" from Python code.
Even if that where possible, changing the metaclass for all classes would not change classes already defined. So, list, etc... would not have affected. You could arrange your program so that it could "install" your class creation hooks before importing any other modules with class definition, though - so it could affect classes written in Python code. (Classes created in extensions are defined in C code and do not go through the metaclass-class creation process anyway)
type is referenced as the metaclass for object, so even if you change type in the builtins - which is possible, that won't automatically be used as a metaclass for anyone. It is used by default because it is what is returned by type(object)
All that said, it is possible to create something that would seek through all existing classes in a running Python program, and, whenever the class is defined in Python, to decorate a __getitem__ method if it exists, to accept keyword parameters.
But then:
The support with indexing arguments as proposed in PEP 472 requires changes to the parser and to the language specification - simply accepting keyword arguments in __getitem__ won´t make a[b=1] work, or not be a syntax error. One would still have to write a.__getitem__(b=1)
An index name in __getitem__ is something very specific for a kind of objects. There is no way it would make sense for any class designed without that in mind. If a is a list, what a[fish='golden'] would mean? And what if a is a dict?
All in all, you'd already have a very cool class if you would come up with something for which it makes sense to have name passed in the index - and then you could just have any method to retrieve it, and use the regular parentheses notation for that a.get(fish="gold"), or even, if you write the __call__ method: a(fish="gold")
I would like to create a class which defines a particular interface, and then require all subclasses to conform to this interface. For example, I would like to define a class
class Interface:
def __init__(self, arg1):
pass
def foo(self, bar):
pass
and then be assured that if I am holding any element a which has type A, a subclass of Interface, then I can call a.foo(2) it will work.
It looked like this question almost addressed the problem, but in that case it is up to the subclass to explicitly change it's metaclass.
Ideally what I'm looking for is something similar to Traits and Impls from Rust, where I can specify a particular Trait and a list of methods that trait needs to define, and then I can be assured that any object with that Trait has those methods defined.
Is there any way to do this in Python?
So, first, just to state the obvious - Python has a built-in mechanism to test for the existence of methods and attributes in derived classes - it just does not check their signature.
Second, a nice package to look at is zope.interface. Despte the zope namespace, it is a complete stand-alone package that allows really neat methods of having objects that can expose multiple interfaces, but just when needed - and then frees-up the namespaces. It sure involve some learning until one gets used to it, but it can be quite powerful and provide very nice patterns for large projects.
It was devised for Python 2, when Python had a lot less features than nowadays - and I think it does not perform automatic interface checking (one have to manually call a method to find-out if a class is compliant) - but automating this call would be easy, nonetheless.
Third, the linked accepted answer at How to enforce method signature for child classes? almost works, and could be good enough with just one change. The problem with that example is that it hardcodes a call to type to create the new class, and do not pass type.__new__ information about the metaclass itself. Replace the line:
return type(name, baseClasses, d)
for:
return super().__new__(cls, name, baseClasses, d)
And then, make the baseclass - the one defining your required methods use the metaclass - it will be inherited normally by any subclasses. (just use Python's 3 syntax for specifying metaclasses).
Sorry - that example is Python 2 - it requires change in another line as well, I better repost it:
from types import FunctionType
# from https://stackoverflow.com/a/23257774/108205
class SignatureCheckerMeta(type):
def __new__(mcls, name, baseClasses, d):
#For each method in d, check to see if any base class already
#defined a method with that name. If so, make sure the
#signatures are the same.
for methodName in d:
f = d[methodName]
for baseClass in baseClasses:
try:
fBase = getattr(baseClass, methodName)
if not inspect.getargspec(f) == inspect.getargspec(fBase):
raise BadSignatureException(str(methodName))
except AttributeError:
#This method was not defined in this base class,
#So just go to the next base class.
continue
return super().__new__(mcls, name, baseClasses, d)
On reviewing that, I see that there is no mechanism in it to enforce that a method is actually implemented. I.e. if a method with the same name exists in the derived class, its signature is enforced, but if it does not exist at all in the derived class, the code above won't find out about it (and the method on the superclass will be called - that might be a desired behavior).
The answer:
Fourth -
Although that will work, it can be a bit rough - since it does any method that override another method in any superclass will have to conform to its signature. And even compatible signatures would break. Maybe it would be nice to build upon the ABCMeta and #abstractmethod existind mechanisms, as those already work all corner cases. Note however that this example is based on the code above, and check signatures at class creation time, while the abstractclass mechanism in Python makes it check when the class is instantiated. Leaving it untouched will enable you to work with a large class hierarchy, which might keep some abstractmethods in intermediate classes, and just the final, concrete classes have to implement all methods.
Just use this instead of ABCMeta as the metaclass for your interface classes, and mark the methods you want to check the interface as #abstractmethod as usual.
class M(ABCMeta):
def __init__(cls, name, bases, attrs):
errors = []
for base_cls in bases:
for meth_name in getattr(base_cls, "__abstractmethods__", ()):
orig_argspec = inspect.getfullargspec(getattr(base_cls, meth_name))
target_argspec = inspect.getfullargspec(getattr(cls, meth_name))
if orig_argspec != target_argspec:
errors.append(f"Abstract method {meth_name!r} not implemented with correct signature in {cls.__name__!r}. Expected {orig_argspec}.")
if errors:
raise TypeError("\n".join(errors))
super().__init__(name, bases, attrs)
You could follow the pyspark pattern, where the method of the base class performs (optional) argument validity checking, and then calls a "non-public" method of the subclass, for example:
class Regressor():
def fit(self, X, y):
self._check_arguments(X, y)
self._fit(X, y)
def _check_arguments(self, X, y):
if True:
pass
else:
raise ValueError('Invalid arguments.')
class LinearRegressor(Regressor):
def _fit(self, X, y):
# code here
TL;DR -
I have a class that uses a metaclass.
I would like to access the parameters of the object's constructor from the metaclass, just before the initialization process, but I couldn't find a way to access those parameters.
How can I access the constructor's parameters from the metaclass function __new__?
In order to practice the use of metaclasses in python, I would like to create a class that would be used as the supercomputer "Deep Thought" from the book "The Hitchhiker's Guide to the Galaxy".
The purpose of my class would be to store the various queries the supercomputer gets from users.
At the bottom line, it would just get some arguments and store them.
If one of the given arguments is number 42 or the string "The answer to life, the universe, and everything", I don't want to create a new object but rather return a pointer to an existing object.
The idea behind this is that those objects would be the exact same so when using the is operator to compare those two, the result would be true.
In order to be able to use the is operator and get True as an answer, I would need to make sure those variables point to the same object. So, in order to return a pointer to an existing object, I need to intervene in the middle of the initialization process of the object. I cannot check the given arguments at the constructor itself and modify the object's inner-variables accordingly because it would be too late: If I check the given parameters only as part of the __init__ function, those two objects would be allocated on different portions of the memory (they might be equal but won't return True when using the is operator).
I thought of doing something like that:
class SuperComputer(type):
answer = 42
def __new__(meta, name, bases, attributes):
# Check if args contains the number "42"
# or has the string "The answer to life, the universe, and everything"
# If so, just return a pointer to an existing object:
return SuperComputer.answer
# Else, just create the object as it is:
return super(SuperComputer, meta).__new__(meta, name, bases, attributes)
class Query(object):
__metaclass__ = SuperComputer
def __init__(self, *args, **kwargs):
self.args = args
for key, value in kwargs.items():
setattr(self, key, value)
def main():
number = Query(42)
string = Query("The answer to life, the universe, and everything")
other = Query("Sunny", "Sunday", 123)
num2 = Query(45)
print number is string # Should print True
print other is string # Should print False
print number is num2 # Should print False
if __name__ == '__main__':
main()
But I'm stuck on getting the parameters from the constructor.
I saw that the __new__ method gets only four arguments:
The metaclass instance itself, the name of the class, its bases, and its attributes.
How can I send the parameters from the constructor to the metaclass?
What can I do in order to achieve my goal?
You don't need a metaclass for that.
The fact is __init__ is not the "constructor" of an object in Python, rather, it is commonly called an "initializator" . The __new__ is closer to the role of a "constructor" in other languages, and it is not available only for the metaclass - all classes have a __new__ method. If it is not explicitly implemented, the object.__new__ is called directly.
And actually, it is object.__new__ which creates a new object in Python. From pure Python code, there is no other possible way to create an object: it will always go through there. That means that if you implement the __new__ method on your own class, you have the option of not creating a new instance, and instead return another pre-existing instance of the same class (or any other object).
You only have to keep in mind that: if __new__ returns an instance of the same class, then the default behavior is that __init__ is called on the same instance. Otherwise, __init__ is not called.
It is also worth noting that in recent years some recipe for creating "singletons" in Python using metaclasses became popular - it is actually an overkill approach,a s overriding __new__ is also preferable for creating singletons.
In your case, you just need to have a dictionary with the parameters you want to track as your keys, and check if you create a new instance or "recycle" one whenever __new__ runs. The dictionary may be a class attribute, or a global variable at module level - that is your pick:
class Recycler:
_instances = {}
def __new__(cls, parameter1, ...):
if parameter1 in cls._instances:
return cls._instances[parameter1]
self = super().__new__(cls) # don't pass remaining parameters to object.__new__
_instances[parameter1] = self
return self
If you'd have any code in __init__ besides that, move it to __new__ as well.
You can have a baseclass with this behavior and have a class hierarchy without needing to re-implement __new__ for every class.
As for a metaclass, none of its methods are called when actually creating a new instance of the classes created with that metaclass. It would only be of use to automatically insert this behavior, by decorating or creating a fresh __new__ method, on classes created with that metaclass. Since this behavior is easier to track, maintain, and overall to combine with other classes just using ordinary inheritance, no need for a metaclass at all.
I have gone through this: What is a metaclass in Python?
But can any one explain more specifically when should I use the meta class concept and when it's very handy?
Suppose I have a class like below:
class Book(object):
CATEGORIES = ['programming','literature','physics']
def _get_book_name(self,book):
return book['title']
def _get_category(self, book):
for cat in self.CATEGORIES:
if book['title'].find(cat) > -1:
return cat
return "Other"
if __name__ == '__main__':
b = Book()
dummy_book = {'title':'Python Guide of Programming', 'status':'available'}
print b._get_category(dummy_book)
For this class.
In which situation should I use a meta class and why is it useful?
Thanks in advance.
You use metaclasses when you want to mutate the class as it is being created. Metaclasses are hardly ever needed, they're hard to debug, and they're difficult to understand -- but occasionally they can make frameworks easier to use. In our 600Kloc code base we've used metaclasses 7 times: ABCMeta once, 4x models.SubfieldBase from Django, and twice a metaclass that makes classes usable as views in Django. As #Ignacio writes, if you don't know that you need a metaclass (and have considered all other options), you don't need a metaclass.
Conceptually, a class exists to define what a set of objects (the instances of the class) have in common. That's all. It allows you to think about the instances of the class according to that shared pattern defined by the class. If every object was different, we wouldn't bother using classes, we'd just use dictionaries.
A metaclass is an ordinary class, and it exists for the same reason; to define what is common to its instances. The default metaclass type provides all the normal rules that make classes and instances work the way you're used to, such as:
Attribute lookup on an instance checks the instance followed by its class, followed by all superclasses in MRO order
Calling MyClass(*args, **kwargs) invokes i = MyClass.__new__(MyClass, *args, **kwargs) to get an instance, then invokes i.__init__(*args, **kwargs) to initialise it
A class is created from the definitions in a class block by making all the names bound in the class block into attributes of the class
Etc
If you want to have some classes that work differently to normal classes, you can define a metaclass and make your unusual classes instances of the metaclass rather than type. Your metaclass will almost certainly be a subclass of type, because you probably don't want to make your different kind of class completely different; just as you might want to have some sub-set of Books behave a bit differently (say, books that are compilations of other works) and use a subclass of Book rather than a completely different class.
If you're not trying to define a way of making some classes work differently to normal classes, then a metaclass is probably not the most appropriate solution. Note that the "classes define how their instances work" is already a very flexible and abstract paradigm; most of the time you do not need to change how classes work.
If you google around, you'll see a lot of examples of metaclasses that are really just being used to go do a bunch of stuff around class creation; often automatically processing the class attributes, or finding new ones automatically from somewhere. I wouldn't really call those great uses of metaclasses. They're not changing how classes work, they're just processing some classes. A factory function to create the classes, or a class method that you invoke immediately after class creation, or best of all a class decorator, would be a better way to implement this sort of thing, in my opinion.
But occasionally you find yourself writing complex code to get Python's default behaviour of classes to do something conceptually simple, and it actually helps to step "further out" and implement it at the metaclass level.
A fairly trivial example is the "singleton pattern", where you have a class of which there can only be one instance; calling the class will return an existing instance if one has already been created. Personally I am against singletons and would not advise their use (I think they're just global variables, cunningly disguised to look like newly created instances in order to be even more likely to cause subtle bugs). But people use them, and there are huge numbers of recipes for making singleton classes using __new__ and __init__. Doing it this way can be a little irritating, mainly because Python wants to call __new__ and then call __init__ on the result of that, so you have to find a way of not having your initialisation code re-run every time someone requests access to the singleton. But wouldn't be easier if we could just tell Python directly what we want to happen when we call the class, rather than trying to set up the things that Python wants to do so that they happen to do what we want in the end?
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
Under 10 lines, and it turns normal classes into singletons simply by adding __metaclass__ = Singleton, i.e. nothing more than a declaration that they are a singleton. It's just easier to implement this sort of thing at this level, than to hack something out at the class level directly.
But for your specific Book class, it doesn't look like you have any need to do anything that would be helped by a metaclass. You really don't need to reach for metaclasses unless you find the normal rules of how classes work are preventing you from doing something that should be simple in a simple way (which is different from "man, I wish I didn't have to type so much for all these classes, I wonder if I could auto-generate the common bits?"). In fact, I have never actually used a metaclass for something real, despite using Python every day at work; all my metaclasses have been toy examples like the above Singleton or else just silly exploration.
A metaclass is used whenever you need to override the default behavior for classes, including their creation.
A class gets created from the name, a tuple of bases, and a class dict. You can intercept the creation process to make changes to any of those inputs.
You can also override any of the services provided by classes:
__call__ which is used to create instances
__getattribute__ which is used to lookup attributes and methods on a class
__setattr__ which controls setting attributes
__repr__ which controls how the class is diplayed
In summary, metaclasses are used when you need to control how classes are created or when you need to alter any of the services provided by classes.
If you for whatever reason want to do stuff like Class[x], x in Class etc., you have to use metaclasses:
class Meta(type):
def __getitem__(cls, x):
return x ** 2
def __contains__(cls, x):
return int(x ** (0.5)) == x ** 0.5
# Python 2.x
class Class(object):
__metaclass__ = Meta
# Python 3.x
class Class(metaclass=Meta):
pass
print Class[2]
print 4 in Class
check the link Meta Class Made Easy to know how and when to use meta class.