I'm learning about metaclasses in Python. I think it is a very powerful technique, and I'm looking for good uses for them. I'd like some feedback of good useful real-world examples of using metaclasses. I'm not looking for example code on how to write a metaclass (there are plenty examples of useless metaclasses out there), but real examples where you have applied the technique and it was really the appropriate solution. The rule is: no theoretical possibilities, but metaclasses at work in a real application.
I'll start with the one example I know:
Django models, for declarative programming, where the base class Model uses a metaclass to fill the model objects of useful ORM functionality from the attribute definitions.
Looking forward to your contributions.
In Python 2.6 and 3.1, the Python standard library provides an abc.ABCMeta, a meta-class for Abstract Base Classes ("ABCs"). Classes that use the meta-class can use #abstractmethod and #abstractproperty to define abstract methods and properties. The meta-class will ensure that derived classes override the abstract methods and properties.
Also, classes that implement the ABC without actually inheriting from it can register as implementing the interface, so that issubclass and isinstance will work.
For example, the collections module defines the Sequence ABC. It also calls Sequence.register(tuple) to register the built-in tuple type as a Sequence, even though tuple does not actually inherit from Sequence.
The Python implementation of Protocol Buffers uses metaclasses to generate the Python bindings that represent your data format. From the tutorial:
The important line in each class is __metaclass__ = reflection.GeneratedProtocolMessageType. While the details of how Python metaclasses work is beyond the scope of this tutorial, you can think of them as like a template for creating classes. At load time, the GeneratedProtocolMessageType metaclass uses the specified descriptors to create all the Python methods you need to work with each message type and adds them to the relevant classes. You can then use the fully-populated classes in your code.
FormEncode validators and Turbogears / Tosca widgets.
You might also be interested in class decorators: they can be written with the latest releases, and cover many use cases that were previously handled with metaclasses.
SQLalchemy also uses them for declarative database models.
Sorry my answer isn't very different from your example, but if you're looking for example code, I found declarative to be pretty readable.
The only time I used a metaclass so far was to write a deprecation warning mechanism. It was something along the following lines - syntax may be very approximative, but code will illustrate my point more easily than a complicated sentence :
class New(object):
pass
class Old(object):
def __new__(self):
deprecation_warning("Old class is no more supported, use New class instead")
return New()
Related
Basically, I know that abstract base classes are used as skeleton classes just like regular classes, and there main advantage would be to enforce their implementation on the child classes.
But I was wondering if I have the next case:
I have a class which is having only static methods / no init -> it would make sense to make it abstract? It would be pythonic?
I was thinking the advantage would be that some one reading the code would know that that class should not be instantiated...
It seems that you're trying to emulate namespaces. It's better to use modules. The mechanism is built into Python, and functions as a namespace:
https://docs.python.org/3/tutorial/modules.html
An abstract class with only static-methods can work as a namespace, but it's confusing to people reading the source code
I have a class which is having only static methods / no init -> it
would make sense to make it abstract? It would be pythonic?
PEP 3119 gives following rationale for Abstract Base Class
This PEP proposes a particular strategy for organizing these tests
known as Abstract Base Classes, or ABC. ABCs are simply Python classes
that are added into an object’s inheritance tree to signal certain
features of that object to an external inspector. Tests are done using
isinstance(), and the presence of a particular ABC means that the test
has passed.
Taking this in account I would find it confusing to find Abstract Base Class which is then not inherited at all. If all methods are static why do not simply make all of them just functions?
I first learned polymorphism in c++, in c++ we had types for every variable. So we used polymorphism to get a single pointer which can point to different type objects, and we could use them very nice.
But I don't get polymorphism and abstract classes in python. Here every variable can be everything. It could be an iterator, a list, a singe variable or a function. Every thing. So what makes a programmer to use an abstract class or use polymorphism here?
In c++ we used inheritance in many ways. But in python, it is just used to use another classes method or attribute. Am I right? what's the matter?
You don't understand what polymorphism is (OO polymorphic dispatch I mean). Polymorphism is the ability to have objects of different types understanding the same message, so you can use those objects the same way without worrying about their concrete type.
C++ actually uses the same concept (class) to denote two slightly different semantics: the abstract type (interface) which is the set of messages an object of this type understand) and the concrete type (implementation) which defines how this type reacts to those messages.
Java clearly distinguishes between abstract type (interface) and concrete type (class).
Python, being dynamically typed, relies mostly on "duck typing" (if it walks like a duck and quack like duck, then it's a duck - or at least it's "kind-of-a-duck" enough). You'll often find terms like "file-like" or "dict-like" in Python docs, meaning "anything that has the same interface as a file (or dict)", and quite a few "interfaces" are (or at least have long been) more or less implicit.
The issue with those implicit interfaces is that they are seldom fully documented, and one sometimes have to get to a function source code to find out exactly what the object passed needs to support. That's one of the reasons why the abc module was introduced in python 2 and improved in python 3: as a way to better document those implicit interfaces by creating an abstract base type that clearly defines the interface.
Another reason for abstract base classes (whether using the abc module or not) is to provide a common base implementation for a set of concrete subclasses. This is specially useful for frameworks, ie Django's models.Model (ORM) or forms.Form (user input collection and validation) classes - in both cases, just defining the database or form fields is enough to have something working.
Inheritance in C++ suffers from the same issue as classes: it serves both as defining the interface and implementation. This adds to the confusion... Java had the good idea (IMHO) to have separate abstract type from implementation, but failed to go all the way and restrict typing to interfaces - you can use either classes or interfaces for type declaration, so it still doesn't make the distinction clear.
In Python, since we don't have static typing, inheritance is mostly about implementation reuse indeed. The abc module allows you to register totally unrelated classes (no inheritance relationship) as also being subtypes of a defined abstract base case, but the point here is mostly to document that your class implements the same interface (and that it's not an accident...).
Let's say I define a base class that provides some methods that will likely be overridden. It's not required though, as the base class provides naive default implementations. How can I highlight those methods best?
In C++, I would just use virtual but Python is a dynamic language where methods can always be overridden without marking them. I am rather looking for a hint here. Ideally, this is faster to see than a textual explanation in the docstring and easier to understand to others than a custom decorator.
All the Python built-ins are subclasses of object and I come across many user-defined classes which are too. Why? What is the purpose of the class object? It's just an empty class, right?
Note: new-style classes are the default in Python 3. Subclassing object is unnecessary there. Read below for more information on usage with Python 2.
In short, it sets free magical ponies.
In long, Python 2.2 and earlier used "old style classes". They were a particular implementation of classes, and they had a few limitations (for example, you couldn't subclass builtin types). The fix for this was to create a new style of class. But, doing this would involve some backwards-incompatible changes. So, to make sure that code which is written for old style classes will still work, the object class was created to act as a superclass for all new-style classes.
So, in Python 2.X, class Foo: pass will create an old-style class and class Foo(object): pass will create a new style class.
In longer, see Guido's Unifying types and classes in Python 2.2.
And, in general, it's a good idea to get into the habit of making all your classes new-style, because some things (the #property decorator is one that comes to mind) won't work with old-style classes.
Short answer: subclassing object effectively makes it a new-style class (note that this is unnecessary since automatic in Python 3.x)
For the difference between new style classes and old style classes: see this stackoverflow question. For the complete story: see this nice writeup on Python Types and Objects.
It has to do with the "new-style" of classes. You can read more about it here: http://docs.python.org/tutorial/classes.html#multiple-inheritance and also here: http://docs.python.org/reference/datamodel.html#new-style-and-classic-classes
Using new-style classes will allow you to use "Python's newer, versatile features like __slots__, descriptors, properties, and __getattribute__()."
Right, but it marks the class as a new-style class. Newly developed classes should use the object base because it costs little and future-proofs your code.
The short version is that classic classes, which didn't need a superclass, had limitations that couldn't be worked around without breaking a lot of old code. So they created the concept of new-style classes which subclass from object, and now you can do cool things like define properties, and subclassing dict is no longer an exercise in pain and strange bugs.
The details are in section 3.3 of the Python docs: New-style and classic classes.
Python 2.2 introduced "new style classes" which had a number of additional features relative to the old style classes which did not subclass object. Subclasses object was the chosen way to indicate that your class should be a new style class, not an old style one.
I recently discovered metaclasses in python.
Basically a metaclass in python is a class that creates a class. There are many useful reasons why you would want to do this - any kind of class initialisation for example. Registering classes on factories, complex validation of attributes, altering how inheritance works, etc. All of this becomes not only possible but simple.
But in python, metaclasses are also plain classes. So, I started wondering if the abstraction could usefully go higher, and it seems to me that it can and that:
a metaclass corresponds to or implements a role in a pattern (as in GOF pattern languages).
a meta-metaclass is the pattern itself (if we allow it to create tuples of classes representing abstract roles, rather than just a single class)
a meta-meta-metaclass is a pattern factory, which corresponds to the GOF pattern groupings, e.g. Creational, Structural, Behavioural. A factory where you could describe a case of a certain type of problem and it would give you a set of classes that solved it.
a meta-meta-meta-metaclass (as far as I could go), is a pattern factory factory, a factory to which you could perhaps describe the type of your problem and it would give you a pattern factory to ask.
I have found some stuff about this online, but mostly not very useful. One problem is that different languages define metaclasses slightly differently.
Has anyone else used metaclasses like this in python/elsewhere, or seen this used in the wild, or thought about it? What are the analogues in other languages? E.g. in C++ how deep can the template recursion go?
I'd very much like to research it further.
This reminds me of the eternal quest some people seem to be on to make a "generic implementation of a pattern." Like a factory that can create any object (including another factory), or a general-purpose dependency injection framework that is far more complex to manage than simply writing code that actually does something.
I had to deal with people intent on abstraction to the point of navel-gazing when I was managing the Zend Framework project. I turned down a bunch of proposals to create components that didn't do anything, they were just magical implementations of GoF patterns, as though the pattern were a goal in itself, instead of a means to a goal.
There's a point of diminishing returns for abstraction. Some abstraction is great, but eventually you need to write code that does something useful.
Otherwise it's just turtles all the way down.
To answer your question: no.
Feel free to research it further.
Note, however, that you've conflated design patterns (which are just ideas) with code (which is an implementation.)
Good code often reflects a number of interlocking design patterns. There's no easy way for formalize this. The best you can do is a nice picture, well-written docstrings, and method names that reflect the various design patterns.
Also note that a meta-class is a class. That's a loop. There's no higher level of abstractions. At that point, it's just intent. The idea of meta-meta-class doesn't mean much -- it's a meta-class for meta-classes, which is silly but technically possible. It's all just a class, however.
Edit
"Are classes that create metaclasses really so silly? How does their utility suddenly run out?"
A class that creates a class is fine. That's pretty much it. The fact that the target class is a meta class or an abstract superclass or a concrete class doesn't matter. Metaclasses make classes. They might make other metaclasses, which is weird, but they're still just metaclasses making classes.
The utility "suddenly" runs out because there's no actual thing you need (or can even write) in a metaclass that makes another metaclass. It isn't that it "suddenly" becomes silly. It's that there's nothing useful there.
As I seed, feel free to research it. For example, actually write a metaclass that builds another metaclass. Have fun. There might be something useful there.
The point of OO is to write class definitions that model real-world entities. As such, a metaclass is sometimes handy to define cross-cutting aspects of several related classes. (It's a way to do some Aspect-Oriented Programming.) That's all a metaclass can really do; it's a place to hold a few functions, like __new__(), that aren't proper parts of the class itself.
During the History of Programming Languages conference in 2007, Simon Peyton Jones commented that Haskell allows meta programming using Type Classes, but that its really turtles all the way down. You can meta-meta-meta-meta etc program in Haskell, but that he's never heard of anyone using more than 3 levels of indirection.
Guy Steele pointed out that its the same thing in Lisp and Scheme. You can do meta-programming using backticks and evals (you can think of a backtick as a Python lambda, kinda), but he's never seen more than 3 backticks used.
Presumably they have seen more code than you or I ever has, so its only a slight exaggeration to say that no-one has ever gone beyond 3 levels of meta.
If you think about it, most people don't ever use meta-programming, and two levels is pretty hard to wrap your head around. I would guess that three is nearly impossible, and the that last guy to try four ended up in an asylum.
Since when I first understood metaclasses in Python, I kept wondering "what could be done with a meta-meta class?". This is at least 10 years ago - and now, just a couple months ago, it became clear for me that there is one mechanism in Python class creation that actually involves a "meta-meta" class. And therefore, it is possible to try to imagine some use for that.
To recap object instantiation in Python: Whenever one instantiates an object in Python by "calling" its class with the same syntax used for calling an ordinary function, the class's __new__ and __init__. What "orchestrates" the calling of these methods on the class is exactly the class'metaclass' __call__ method. Usually when one writes a metaclass in Python, either the __new__ or __init__ method of the metaclass is customized.
So, it turns out that by writing a "meta-meta" class one can customize its __call__ method and thus control which parameters are passed and to the metaclass's __new__ and __init__ methods, and if some other code is to be called before of after those. What turns out in the end is that metcalsses themselves are usually hardcoded and one needs just a few, if any, even in very large projects. So any customization that might be done at the "meta meta" call is usually done directly on the metaclass itself.
And them, there are those other less frequent uses for Python metaclasses - one can customize an __add__ method in a metaclass so that the classes they define are "addable", and create a derived class having the two added classes as superclasses. That mechanism is perfectly valid with metaclasses as well - therefore, so just we "have some actual code", follows an example of "meta-meta" class that allows one to compose "metaclasses" for a class just by adding them on class declaration:
class MM(type):
def __add__(cls, other):
metacls = cls.__class__
return metacls(cls.__name__ + other.__name__, (cls, other), {})
class M1(type, metaclass=MM):
def __new__(metacls, name, bases, namespace):
namespace["M1"] = "here"
print("At M1 creation")
return super().__new__(metacls, name, bases, namespace)
class M2(type, metaclass=MM):
def __new__(metacls, name, bases, namespace):
namespace["M2"] = "there"
print("At M2 creation")
return super().__new__(metacls, name, bases, namespace)
And we can see that working on the interactive console:
In [22]: class Base(metaclass = M1 + M2):
...: pass
...:
At M1 creation
At M2 creation
Note that as different metaclasses in Python are usually difficult to combine, this can actually be useful by allowing a user-made metaclass to be combined with a library's or stdlib one, without this one having to be explicitly declared as parent of the former:
In [23]: import abc
In [24]: class Combined(metaclass=M1 + abc.ABCMeta):
...: pass
...:
At M1 creation
The class system in Smalltalk is an interesting one to study. In Smalltalk, everything is an object and every object has a class. This doesn't imply that the hierarchy goes to infinity. If I remember correctly, it goes something like:
5 -> Integer -> Integer class -> Metaclass -> Metaclass class -> Metaclass -> ... (it loops)
Where '->' denotes "is an instance of".