I have an esoteric question involving Python metaclasses. I am creating a Python package for web-server-side code that will make it easy to access arbitrary Python classes via client-side proxies. My proxy-generating code needs a catalog of all of the Python classes that I want to include in my API. To create this catalog, I am using the __metaclass__ special attribute to put a hook into the class-creation process. Specifically, all of the classes in the "published" API will subclass a particular base class, PythonDirectPublic, which itself has a __metaclass__ that has been set up to record information about the class creation.
So far so good. Where it gets complicated is that I want my PythonDirectPublic itself to inherit from a third-party class (enthought.traits.api.HasTraits). This third-party class also uses a __metaclass__.
So what's the right way of managing two metaclasses? Should my metaclass be a subclass of Enthought's metaclass? Or should I just invoke Enthought's metaclass inside my metaclass's __new__ method to get the type object that I will return? Or is there some other mystical incantation to use in this particular circumstance?
Should my metaclass be a subclass of Enthought's metaclass?
I believe that is in fact your only choice. If the metaclass of a derived class is not a subclass of the metaclasses of all of its bases then Python will throw a TypeError when you try to create the derived class. So your metaclass for PythonDirectPublic should look something like
class DerivedMetaClass(BaseMetaClass):
def __new__(cls, name, bases, dct):
# Do your custom memory allocation here, if any
# Now let base metaclass do its memory allocation stuff
return BaseMetaClass.__new__(cls, name, bases, dct)
def __init__(cls, name, bases, dct):
# Do your custom initialization here, if any
# This, I assume, is where your catalog creation stuff takes place
# Now let base metaclass do its initialization stuff
super(DerivedMetaClass, cls).__init__(name, bases, dct)
If you don't have access to the definition of the metaclass for your third-party base class, you can replace BaseMetaClass with enthought.traits.api.HasTraits.__metaclass__. It's wordy, but it will work.
Specifically, all of the classes in the "published" API will subclass a particular base class, PythonDirectPublic
Rather than adding another metaclass, you could recursively use the result of PythonDirectPublic.subclasses().
Related
I've been learning about metaclasses, and I was wondering if it's possible to add a new attribute to every class that's defined in python, not just those which inherit explicitly from a custom metaclass.
I can add a new attribute explicitly using a custom metaclass like this
class NewAttrMeta(type):
def __new__(cls, name, bases, attrs):
attrs['new_attr'] = 'new_thing'
return super().__new__(cls, name, bases, attrs)
class A(metaclass=NewAttrMeta):
...
print(A.new_attr)
$ 'new_thing'
But is it possible to force a change like this on every class that's defined, not just the ones which explicitly inherit from your custom metaclass?
I thought maybe as all classes are of type type, if I overwrote type itself with my custom metaclass, then all new classes might then inherit from it. And as metaclasses are subclasses of type, then all classes defined that way would still be valid...
class NewAttrMeta(type):
def __new__(cls, name, bases, attrs):
attrs['new_attr'] = 'new_thing'
return super().__new__(cls, name, bases, attrs)
type = NewMagicMeta
But this only works if type is passed in explicitly again:
class A(type):
...
print(A.new_attr)
$ 'new_thing'
class B():
...
print(B.new_attr)
$ AttributeError: type object 'A' has no attribute 'new_attr'
Why on earth am I trying to do this? I wanted to see if I could locally implement a version of the rejected PEP 472: "Support for indexing with keyword arguments" by overriding the __getitem__ method of every class which defined any version of __getitem__. I'm only doing this for fun, so I would be interested in any insights or alternative ways to do that (the hackier the better!).
Python does not allow modification of built-in declared types. That
means that dictionaries, lists, classes defineds in extensions like
Numpy.ndarrays are "frozen" from Python code.
Even if that where possible, changing the metaclass for all classes would not change classes already defined. So, list, etc... would not have affected. You could arrange your program so that it could "install" your class creation hooks before importing any other modules with class definition, though - so it could affect classes written in Python code. (Classes created in extensions are defined in C code and do not go through the metaclass-class creation process anyway)
type is referenced as the metaclass for object, so even if you change type in the builtins - which is possible, that won't automatically be used as a metaclass for anyone. It is used by default because it is what is returned by type(object)
All that said, it is possible to create something that would seek through all existing classes in a running Python program, and, whenever the class is defined in Python, to decorate a __getitem__ method if it exists, to accept keyword parameters.
But then:
The support with indexing arguments as proposed in PEP 472 requires changes to the parser and to the language specification - simply accepting keyword arguments in __getitem__ won´t make a[b=1] work, or not be a syntax error. One would still have to write a.__getitem__(b=1)
An index name in __getitem__ is something very specific for a kind of objects. There is no way it would make sense for any class designed without that in mind. If a is a list, what a[fish='golden'] would mean? And what if a is a dict?
All in all, you'd already have a very cool class if you would come up with something for which it makes sense to have name passed in the index - and then you could just have any method to retrieve it, and use the regular parentheses notation for that a.get(fish="gold"), or even, if you write the __call__ method: a(fish="gold")
I'm in the process of migrating from 2.7 to 3.x and I'm trying to understand the __prepare__ method for metaclasses introduced in PEP3115.
In most of the examples I've seen, implementations of this method ignore the parameters (name, bases, and **kwargs) simply returns a custom dictionary that does something interesting to the namespace provided to the __new__ and __init__ methods for the metaclass. Even the example in PEP3115 does nothing with the parameters.
I don't doubt that there is some good reason for the signature of __prepare__ but I haven't seen the use case.
What are some good examples that demonstrate the rational for making the signature of __prepare__ take these parameters?
__prepare__ will create a namespace for the class like you said, so we can do some logic inside of it like this:
class MyMeta(type):
#classmethod
def __prepare__(metacls, klass_name, bases):
namespace = {'s': 'my string',
'description': 'N/A'}
if klass_name.endswith('Base'):
namespace.update({'description': 'base class'})
return namespace
class KlassBase(metaclass=MyMeta):
def __init__(self, value):
self.value = value
class SubKlass(KlassBase):
def __init__(self, value):
super().__init__(value)
print(KlassBase(5).s)
print(KlassBase(5).description)
print(SubKlass(5).s)
print(SubKlass(5).description)
And you got:
my string
base class
my string
N/A
The reason why we don't do it, because same things could be done in other part of the meta class like : __new__, __init__, or be overrided by the latter. So most of time, we won't do it in __prepare__
The following is an image of the class creation work-flow, which is much more clearer:
[sorry I cannot find the original source of this pic]
When you look at __prepare__, and the state of Python class creation mechanism at the time, it is pretty much clear that what was really needed was a mechanism to enable attribute order preservation.
This way, one would be able to create classes that would describe data records with ordered fields, which is pretty much what humans would expect when describing a record. (Try to imagine a form that each time it is rendered, it shuffles the field ordes, so half the time you will be filling in the country you live in before the country).
Instead of a fixed mechanism to just enable the class body namespace to be an collections.OrderedDict, they came up with __prepare__ which enables this easily, with a single line returning a new OrderedDict instance.
But __prepare__ can have so many uses and abuses, that I think no one really thought of all the possibilities. The parameters ou mentioned are avaliable at the time it is called, and since it exists, there is no reason whatsoever for they not to be passed to the function. Why to cripple one of knowing the class' name inside the __prepare__ function?
So, it is just a powerful and flexible mechanism put in place, and not necessarily all possible use cases were thought of when making it. The "Ordered Attributes" thing n the other hand is so important that it became the default for Python 3.6, with PEP 520, even without any custom metaclass declaration.
__prepare__ receivog bases, for example, would allow one to pre-populate the namespace with certain objects that would be found in the superclass namespaces, overriding the inheritance attribute access, for example.
Or it could simply check the class name against a pre-existing registry, and either raise or pre-populate stuff from there.
One super-easy usage would be to pre-populate __name__ for example, so that class attributes could make use of it:
import collections
class M(type):
#classmethod
def __prepare__(metacls, name, bases):
ns = collections.OrderedDict()
ns["__name__"] = name
class T(metaclass=M):
__table__ = f"_{__name__.lower()}"
trivia
Due to the way functions work as methods in Python 3, one interesting related thing that is not documented anywhere is that if __prepare__ is not explicitly decorated to be a classmethod, it works as a staticmethod, and the target class name is passed directly in the first parameter to it.
I've got a mixin class, that adds some functionality to inheriting classes, but the mixin requires some class attributes to be present, for simplicity let's say only one property handlers. So this would be the usage of the mixin:
class Mixin:
pass
class Something(Mixin):
handlers = {}
The mixin can't function without this being defined, but I really don't want to specify the handlers in every class that I want to use the mixin with. So I solved this by writing a metaclass:
class MixinMeta:
def __new__(mcs, *args, **kwargs):
cls = super().__new__(mcs, *args, **kwargs)
cls.handlers = {}
return cls
class Mixin(metaclass=MixinMeta):
pass
And this works exactly how I want it to. But I'm thinking this can become a huge problem, since metaclasses don't work well together (I read various metaclass conflicts can only be solved by creating a new metaclass that resolves those conflicts).
Also, I don't want to make the handlers property a property of the Mixin class itself, since that would mean having to store handlers by their class names inside the Mixin class, complicating the code a bit. I like having each class having their handlers on their own class - it makes working with them simpler, but clearly this has drawbacks.
My question is, what would be a better way to implement this? I'm fairly new to metaclasses, but they seem to solve this problem well. But metaclass conflicts are clearly a huge issue when dealing with complex hierarchies without having to define various metaclasses just to resolve those conflicts.
Your problem is very real, and Python folks have thought of this for Python 3.6 (still unrealsed) on. For now (up to Python 3.5), if your attributes can wait to exist until your classes are first instantiated, you could put cod to create a (class) attribute on the __new__ method of your mixin class itself - thus avoiding the (extra) metaclass:
class Mixin:
def __new__(cls):
if not hasattr(cls, handlers):
cls.handlers = {}
return super().__new__(cls)
For Python 3.6 on, PEP 487 defines a __init_subclass__ special method to go on the mixin class body. This special method is not called for the mixin class itself, but will be called at the end of type.__new__ method (the "root" metaclass) for each class that inherits from your mixin.
class Mixin:
def __init_subclass__(cls, **kwargs):
cls.handlers = {}
return super().__init_subclass__(**kwargs)
As per the PEP's background text, the main motivation for this is exactly what led you to ask your question: avoid the need for meta-classes when simple customization of class creation is needed, in order to reduce the chances of needing different metaclasses in a project, and thus triggering a situation of metaclass conflict.
class MType(type):
pass
class MClass(object):
__metaclass__ = MType
a = MType
From the above snippet is there anyway I can create an instance of MClass given a?
(It's probably important to note that I won't be able to simply contruct MClass())
Here's a horrific way of doing it. Although, I'll be honest.. I'm not sure what the difference between setting __metaclass__ as a module property is vs. inheritance.
class MType(type):
pass
class MClass(MType):
pass
a = MType
print a.__mro__[0].__subclasses__(a)
Yields:
[<class '__main__.MClass'>]
There are two straightforward ways I can think of to find the instances of a metaclass. One is to have the metaclass keep track of a list of its instances as they are created, as follows:
class MType(type):
instances = [] #list of instances of the metaclass
def __init__(cls, name, bases, dct):
MType.instances.append(cls) #append to list of instances
super(MType, cls).__init__(name, bases, dct)
class MClass(object):
__metaclass__ = MType
Now MType.instances is [<class '__main__.MClass'>].
Another way is to use the
inspect module to look through the classes in, say, a given module, and use isinstance to check if they are instances of MType. This isn't necessarily a crazy thing to do, either - for example, various unit testing suites work by looking through a module for classes and functions that appear to be tests, based on their names and/or inherited classes - I'm not really sure why you would want to do this with a metaclass, though.
No, if you have b = MClass, you can simply do obj = b(), but the metaclass does not "know" about the class(es) it is set as the metaclass of.
You could call __new__ on a if you have the class in a variable, whether b or MClass.
I have a number of atomic classes (Components/Mixins, not really sure what to call them) in a library I'm developing, which are meant to be subclassed by applications. This atomicity was created so that applications can only use the features that they need, and combine the components through multiple inheritance.
However, sometimes this atomicity cannot be ensured because some component may depend on another one. For example, imagine I have a component that gives a graphical representation to an object, and another component which uses this graphical representation to perform some collision checking. The first is purely atomic, however the latter requires that the current object already subclassed this graphical representation component, so that its methods are available to it. This is a problem, because we have to somehow tell the users of this library, that in order to use a certain Component, they also have to subclass this other one. We could make this collision component sub class the visual component, but if the user also subclasses this visual component, it wouldn't work because the class is not on the same level (unlike a simple diamond relationship, which is desired), and would give the cryptic meta class errors which are hard to understand for the programmer.
Therefore, I would like to know if there is any cool way, through maybe metaclass redefinition or using class decorators, to mark these unatomic components, and when they are subclassed, the additional dependency would be injected into the current object, if its not yet available. Example:
class AtomicComponent(object):
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent(object):
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
Can someone give me an hint on how I can do this? or if it is even possible...
edit:
Since it is debatable that the meta class solution is the best one, I'll leave this unaccepted for 2 days.
Other solutions might be to improve error messages, for example, doing something like UserClass2 would give an error saying that UnAtomicComponent already extends this component. This however creates the problem that it is impossible to use two UnAtomicComponents, given that they would subclass object on different levels.
"Metaclasses"
This is what they are for! At time of class creation, the class parameters run through the
metaclass code, where you can check the bases and change then, for example.
This runs without error - though it does not preserve the order of needed classes
marked with the "depends" decorator:
class AutoSubclass(type):
def __new__(metacls, name, bases, dct):
new_bases = set()
for base in bases:
if hasattr(base, "_depends"):
for dependence in base._depends:
if not dependence in bases:
new_bases.add(dependence)
bases = bases + tuple(new_bases)
return type.__new__(metacls, name, bases, dct)
__metaclass__ = AutoSubclass
def depends(*args):
def decorator(cls):
cls._depends = args
return cls
return decorator
class AtomicComponent:
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent:
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
(I removed inheritance from "object", as I declared a global __metaclass__ variable. All classs will still be new style class and have this metaclass. Inheriting from object or another class does override the global __metaclass__variable, and a class level __metclass__ will have to be declared)
-- edit --
Without metaclasses, the way to go is to have your classes to properly inherit from their dependencies. Tehy will no longer be that "atomic", but, since they could not work being that atomic, it may be no matter.
In the example bellow, classes C and D would be your User classes:
>>> class A(object): pass
...
>>> class B(A, object): pass
...
>>>
>>> class C(B): pass
...
>>> class D(B,A): pass
...
>>>