I have a weird and unusual use case for metaclasses where I'd like to change the __metaclass__ of a base class after it's been defined so that its subclasses will automatically use the new __metaclass__. But that oddly doesn't work:
class MetaBase(type):
def __new__(cls, name, bases, attrs):
attrs["y"] = attrs["x"] + 1
return type.__new__(cls, name, bases, attrs)
class Foo(object):
__metaclass__ = MetaBase
x = 5
print (Foo.x, Foo.y) # prints (5, 6) as expected
class MetaSub(MetaBase):
def __new__(cls, name, bases, attrs):
attrs["x"] = 11
return MetaBase.__new__(cls, name, bases, attrs)
Foo.__metaclass__ = MetaSub
class Bar(Foo):
pass
print(Bar.x, Bar.y) # prints (5, 6) instead of (11, 12)
What I'm doing may very well be unwise/unsupported/undefined, but I can't for the life of me figure out how the old metaclass is being invoked, and I'd like to least understand how that's possible.
EDIT: Based on a suggestion made by jsbueno, I replaced the line Foo.__metaclass__ = MetaSub with the following line, which did exactly what I wanted:
Foo = type.__new__(MetaSub, "Foo", Foo.__bases__, dict(Foo.__dict__))
The problem is the __metaclass__ attribute is not used when inherited, contrary to what you might expect. The 'old' metaclass isn't called either for Bar. The docs say the following about how the metaclass is found:
The appropriate metaclass is determined by the following precedence
rules:
If dict['__metaclass__'] exists, it is used.
Otherwise, if there is at least one base class, its metaclass is used (this looks for a __class__ attribute first and if not found,
uses its type).
Otherwise, if a global variable named __metaclass__ exists, it is used.
Otherwise, the old-style, classic metaclass (types.ClassType) is used.
So what is actually used as metaclass in your Bar class is found in the parent's __class__ attribute and not in the parent's __metaclass__ attribute.
More information can be found on this StackOverflow answer.
The metaclass information for a class is used at the moment it is created (either parsed as a class block, or dynamically, with a explicit call to the metaclass). It can't be changed because the metaclass usually does make changes at class creation time - the created class type is the metaclasse. Its __metaclass__ attribute is irrelevant once it is created.
However, it is possible to create a copy of a given class, and have the copy bear a different metclass than the original class.
On your example, if instead of doing:
Foo.__metaclass__ = MetaSub
you do:
Foo = Metasub("Foo", Foo.__bases__, dict(Foo.__dict__))
You will achieve what you intended. The new Foo is for all effects equal its predecessor, but with a different metaclass.
However, previously existing instances of Foo won't be considered an instance of the new Foo - if you need that, you better create the Foo copy with a different name instead.
Subclasses use the __metaclass__ of their parent.
The solution to your use-case is to program the parent's __metaclass__ so that it will have different behaviors for the parent than for its subclasses. Perhaps have it inspect the class dictionary for a class variable and implement different behaviors depending on its value
(this is the technique type uses to control whether or not instances are given a dictionary depending on the whether or not __slots__ is defined).
Related
In Python, I can create a class method using the #classmethod decorator:
>>> class C:
... #classmethod
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> C.f()
f called with cls=<class '__main__.C'>
Alternatively, I can use a normal (instance) method on a metaclass:
>>> class M(type):
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> class C(metaclass=M):
... pass
...
>>> C.f()
f called with cls=<class '__main__.C'>
As shown by the output of C.f(), these two approaches provide similar functionality.
What are the differences between using #classmethod and using a normal method on a metaclass?
As classes are instances of a metaclass, it is not unexpected that an "instance method" on the metaclass will behave like a classmethod.
However, yes, there are differences - and some of them are more than semantic:
The most important difference is that a method in the metaclass is not "visible" from a class instance. That happens because the attribute lookup in Python (in a simplified way - descriptors may take precedence) search for an attribute in the instance - if it is not present in the instance, Python then looks in that instance's class, and then the search continues on the superclasses of the class, but not on the classes of the class. The Python stdlib make use of this feature in the abc.ABCMeta.register method.
That feature can be used for good, as methods related with the class themselves are free to be re-used as instance attributes without any conflict (but a method would still conflict).
Another difference, though obvious, is that a method declared in the metaclass can be available in several classes, not otherwise related - if you have different class hierarchies, not related at all in what they deal with, but want some common functionality for all classes, you'd have to come up with a mixin class, that would have to be included as base in both hierarchies (say for including all classes in an application registry). (NB. the mixin may sometimes be a better call than a metaclass)
A classmethod is a specialized "classmethod" object, while a method in the metaclass is an ordinary function.
So, it happens that the mechanism that classmethods use is the "descriptor protocol". While normal functions feature a __get__ method that will insert the self argument when they are retrieved from an instance, and leave that argument empty when retrieved from a class, a classmethod object have a different __get__, that will insert the class itself (the "owner") as the first parameter in both situations.
This makes no practical differences most of the time, but if you want access to the method as a function, for purposes of adding dynamically adding decorator to it, or any other, for a method in the metaclass meta.method retrieves the function, ready to be used, while you have to use cls.my_classmethod.__func__ to retrieve it from a classmethod (and then you have to create another classmethod object and assign it back, if you do some wrapping).
Basically, these are the 2 examples:
class M1(type):
def clsmethod1(cls):
pass
class CLS1(metaclass=M1):
pass
def runtime_wrap(cls, method_name, wrapper):
mcls = type(cls)
setattr(mcls, method_name, wrapper(getatttr(mcls, method_name)))
def wrapper(classmethod):
def new_method(cls):
print("wrapper called")
return classmethod(cls)
return new_method
runtime_wrap(cls1, "clsmethod1", wrapper)
class CLS2:
#classmethod
def classmethod2(cls):
pass
def runtime_wrap2(cls, method_name, wrapper):
setattr(cls, method_name, classmethod(
wrapper(getatttr(cls, method_name).__func__)
)
)
runtime_wrap2(cls1, "clsmethod1", wrapper)
In other words: apart from the important difference that a method defined in the metaclass is visible from the instance and a classmethod object do not, the other differences, at runtime will seem obscure and meaningless - but that happens because the language does not need to go out of its way with special rules for classmethods: Both ways of declaring a classmethod are possible, as a consequence from the language design - one, for the fact that a class is itself an object, and another, as a possibility among many, of the use of the descriptor protocol which allows one to specialize attribute access in an instance and in a class:
The classmethod builtin is defined in native code, but it could just be coded in pure python and would work in the exact same way. The 5 line class bellow can be used as a classmethod decorator with no runtime differences to the built-in #classmethod" at all (though distinguishable through introspection such as calls toisinstance, and evenrepr` of course):
class myclassmethod:
def __init__(self, func):
self.__func__ = func
def __get__(self, instance, owner):
return lambda *args, **kw: self.__func__(owner, *args, **kw)
And, beyond methods, it is interesting to keep in mind that specialized attributes such as a #property on the metaclass will work as specialized class attributes, just the same, with no surprising behavior at all.
When you phrase it like you did in the question, the #classmethod and metaclasses may look similar but they have rather different purposes. The class that is injected in the #classmethod's argument is usually used for constructing an instance (i.e. an alternative constructor). On the other hand, the metaclasses are usually used to modify the class itself (e.g. like what Django does with its models DSL).
That is not to say that you can't modify the class inside a classmethod. But then the question becomes why didn't you define the class in the way you want to modify it in the first place? If not, it might suggest a refactor to use multiple classes.
Let's expand the first example a bit.
class C:
#classmethod
def f(cls):
print(f'f called with cls={cls}')
Borrowing from the Python docs, the above will expand to something like the following:
class ClassMethod(object):
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
class C:
def f(cls):
print(f'f called with cls={cls}')
f = ClassMethod(f)
Note how __get__ can take either an instance or the class (or both), and thus you can do both C.f and C().f. This is unlike the metaclass example you give which will throw an AttributeError for C().f.
Moreover, in the metaclass example, f does not exist in C.__dict__. When looking up the attribute f with C.f, the interpreter looks at C.__dict__ and then after failing to find, looks at type(C).__dict__ (which is M.__dict__). This may matter if you want the flexibility to override f in C, although I doubt this will ever be of practical use.
In your example, the difference would be in some other classes that will have M set as their metaclass.
class M(type):
def f(cls):
pass
class C(metaclass=M):
pass
class C2(metaclass=M):
pass
C.f()
C2.f()
class M(type):
pass
class C(metaclass=M):
#classmethod
def f(cls):
pass
class C2(metaclass=M):
pass
C.f()
# C2 does not have 'f'
Here is more on metaclasses
What are some (concrete) use-cases for metaclasses?
Both #classmethod and Metaclass are different.
Everything in python is an object. Every thing means every thing.
What is Metaclass ?
As said every thing is an object. Classes are also objects in fact classes are instances of other mysterious objects formally called as meta-classes. Default metaclass in python is "type" if not specified
By default all classes defined are instances of type.
Classes are instances of Meta-Classes
Few important points are to understand metioned behaviour
As classes are instances of meta classes.
Like every instantiated object, like objects(instances) get their attributes from class. Class will get it's attributes from Meta-Class
Consider Following Code
class Meta(type):
def foo(self):
print(f'foo is called self={self}')
print('{} is instance of {}: {}'.format(self, Meta, isinstance(self, Meta)))
class C(metaclass=Meta):
pass
C.foo()
Where,
class C is instance of class Meta
"class C" is class object which is instance of "class Meta"
Like any other object(instance) "class C" has access it's attributes/methods defined in it's class "class Meta"
So, decoding "C.foo()" . "C" is instance of "Meta" and "foo" is method calling through instance of "Meta" which is "C".
First argument of method "foo" is reference to instance not class unlike "classmethod"
We can verify as if "class C" is instance of "Class Meta
isinstance(C, Meta)
What is classmethod?
Python methods are said to be bound. As python imposes the restriction that method has to be invoked with instance only.
Sometimes we might want to invoke methods directly through class without any instance (much like static members in java) with out having to create any instance.By default instance is required to call method. As a workaround python provides built-in function classmethod to bind given method to class instead of instance.
As class methods are bound to class. It takes at least one argument which is reference to class itself instead of instance (self)
if built-in function/decorator classmethod is used. First argument
will be reference to class instead of instance
class ClassMethodDemo:
#classmethod
def foo(cls):
print(f'cls is ClassMethodDemo: {cls is ClassMethodDemo}')
As we have used "classmethod" we call method "foo" without creating any instance as follows
ClassMethodDemo.foo()
Above method call will return True. Since first argument cls is indeed reference to "ClassMethodDemo"
Summary:
Classmethod's receive first argument which is "a reference to class(traditionally referred as cls) itself"
Methods of meta-classes are not classmethods. Methods of Meta-classes receive first argument which is "a reference to instance(traditionally referred as self) not class"
I am attempting to modify a value in a class __dict__ directly using something like X.__dict__['x'] += 1. It is impossible to do the modification like that because a class __dict__ is actually a mappingproxy object that does not allow direct modification of values. The reason for attempting direct modification or equivalent is that I am trying to hide the class attribute behind a property defined on the metaclass with the same name. Here is an example:
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = 0
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x']
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
__class__.__dict__['x'] += 1
This is example shows a scheme for creating an auto-incremented ID for each instance of Class. The line __class__.__dict__['x'] += 1 can not be replaced by setattr(__class__, 'x', __class__.x + 1) because x is a property with no setter in Meta. It would just change a TypeError from mappingproxy into an AttributeError from property.
I have tried messing with __prepare__, but that has no effect. The implementation in type already returns a mutable dict for the namespace. The immutable mappingproxy seems to get set in type.__new__, which I don't know how to avoid.
I have also attempted to rebind the entire __dict__ reference to a mutable version, but that failed as well: https://ideone.com/w3HqNf, implying that perhaps the mappingproxy is not created in type.__new__.
How can I modify a class dict value directly, even when shadowed by a metaclass property? While it may be effectively impossible, setattr is able to do it somehow, so I would expect that there is a solution.
My main requirement is to have a class attribute that appears to be read only and does not use additional names anywhere. I am not absolutely hung up on the idea of using a metaclass property with an eponymous class dict entry, but that is usually how I hide read only values in regular instances.
EDIT
I finally figured out where the class __dict__ becomes immutable. It is described in the last paragraph of the "Creating the Class Object" section of the Data Model reference:
When a new class is created by type.__new__, the object provided as the namespace parameter is copied to a new ordered mapping and the original object is discarded. The new copy is wrapped in a read-only proxy, which becomes the __dict__ attribute of the class object.
Probably the best way: just pick another name. Call the property x and the dict key '_x', so you can access it the normal way.
Alternative way: add another layer of indirection:
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = [0]
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x'][0]
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
__class__.__dict__['x'][0] += 1
That way you don't have to modify the actual entry in the class dict.
Super-hacky way that might outright segfault your Python: access the underlying dict through the gc module.
import gc
class Meta(type):
def __new__(cls, name, bases, attrs, **kwargs):
attrs['x'] = 0
return super().__new__(cls, name, bases, attrs)
#property
def x(cls):
return cls.__dict__['x']
class Class(metaclass=Meta):
def __init__(self):
self.id = __class__.x
gc.get_referents(__class__.__dict__)[0]['x'] += 1
This bypasses critical work type.__setattr__ does to maintain internal invariants, particularly in things like CPython's type attribute cache. It is a terrible idea, and I'm only mentioning it so I can put this warning here, because if someone else comes up with it, they might not know that messing with the underlying dict is legitimately dangerous.
It is very easy to end up with dangling references doing this, and I have segfaulted Python quite a few times experimenting with this. Here's one simple case that crashed on Ideone:
import gc
class Foo(object):
x = []
Foo().x
gc.get_referents(Foo.__dict__)[0]['x'] = []
print(Foo().x)
Output:
*** Error in `python3': double free or corruption (fasttop): 0x000055d69f59b110 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x2b32d5977bcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x2b32d597df96]
/lib/x86_64-linux-gnu/libc.so.6(+0x7778e)[0x2b32d597e78e]
python3(+0x2011f5)[0x55d69f02d1f5]
python3(+0x6be7a)[0x55d69ee97e7a]
python3(PyCFunction_Call+0xd1)[0x55d69efec761]
python3(PyObject_Call+0x47)[0x55d69f035647]
... [it continues like that for a while]
And here's a case with wrong results and no noisy error message to alert you to the fact that something has gone wrong:
import gc
class Foo(object):
x = 'foo'
print(Foo().x)
gc.get_referents(Foo.__dict__)[0]['x'] = 'bar'
print(Foo().x)
Output:
foo
foo
I make absolutely no guarantees as to any safe way to use this, and even if things happen to work out on one Python version, they may not work on future versions. It can be fun to fiddle with, but it's not something to actually use. Seriously, don't do it. Do you want to explain to your boss that your website went down or your published data analysis will need to be retracted because you took this bad idea and used it?
This probably counts as an "additional name" you don't want, but I've implemented this using a dictionary in the metaclass where the keys are the classes. The __next__ method on the metaclass makes the class itself iterable, such that you can just do next() to get the next ID. The dunder method also keeps the method from being available through the instances. The dictionary storing the next id has a name starting with a double underscore, so it's not easily discoverable from any of the classes that use it. The incrementing ID functionality is thus entirely contained in the metaclass.
I tucked the assignment of the id into a __new__ method on a base class, so you don't have to worry about it in __init__. This also allows you to del Meta so all the machinery is a little harder to get to.
class Meta(type):
__ids = {}
#property
def id(cls):
return __class__.__ids.setdefault(cls, 0)
def __next__(cls):
id = __class__.__ids.setdefault(cls, 0)
__class__.__ids[cls] += 1
return id
class Base(metaclass=Meta):
def __new__(cls, *args, **kwargs):
self = object.__new__(cls)
self.id = next(cls)
return self
del Meta
class Class(Base):
pass
class Brass(Base):
pass
c0 = Class()
c1 = Class()
b0 = Brass()
b1 = Brass()
assert (b0.id, b1.id, c0.id, c1.id) == (0, 1, 0, 1)
assert (Class.id, Brass.id) == (2, 2)
assert not hasattr(Class, "__ids")
assert not hasattr(Brass, "__ids")
Note that I've used the same name for the attribute on both the class and the object. That way Class.id is the number of instances you've created, while c1.id is the ID of that specific instance.
My main requirement is to have a class attribute that appears to be read only and does not use additional names anywhere. I am not absolutely hung up on the idea of using a metaclass property with an eponymous class dict entry, but that is usually how I hide read only values in regular instances.
What you are asking for is a contradiction: If your example worked, then __class__.__dict__['x'] would be an "additional name" for the attribute. So clearly we need a more specific definition of "additional name." But to come up with that definition, we need to know what you are trying to accomplish (NB: The following goals are not mutually exclusive, so you may want to do all of these things):
You want to make the value completely untouchable, except within the Class.__init__() method (and the same method of any subclasses): This is unPythonic and quite impossible. If __init__() can modify the value, then so can anyone else. You might be able to accomplish something like this if the modifying code lives in Class.__new__(), which the metaclass dynamically creates in Meta.__new__(), but that's extremely ugly and hard to understand.
You want the code that manipulates the value to be "nicely encapsulated": Write a method in the metaclass that increments the private value (or does whatever other modification you need), and provide a read-only metaclass property that accesses it under the public name.
You are concerned about a subclass accidentally clashing names with the private name: Prefix the private name with a double underscore to invoke automatic name mangling. While this is usually seen as a bit unPythonic, it is appropriate for cases where name collisions may be less obvious to subclass authors, such as the internal names of a metaclass colliding with the internal names of a regular class instantiated from it.
I was reading the answers to Usage of __slots__? and I noticed that one of the examples is :
from collections import namedtuple
class MyNT(namedtuple('MyNT', 'bar baz')):
"""MyNT is an immutable and lightweight object"""
__slots__ = ()
I saw that the __init__ of namedtuple was called when it was being subclassed by MyNT
I went ahead and tested it for myself and made this code which is my first attempt to understand such behavior:
class objectP():
def __init__(self,name):
print('object P inited')
class p(objectP('asd')):
pass
I got an error stating that 4 objects were "passed" so I changed it to
class objectP():
def __init__(self,*a):
print('object P inited')
class p(objectP('asd')):
pass
which now produces an output of
object P inited
object P inited
What does the line of code above mean? Calling __init__ when subclassing?
Why is object P inited printed twice?
The example code you list at the top of your question works because the namedtuple function (which isn't actually a class itself) returns a class. You're inheriting from that returned class, not from namedtuple itself.
The same structure doesn't work when you use it in your other code because calling the ObjectP class returns an instance when you call it, and that instance isn't a class that can be inherited from.
You can write a function that returns a class, like namedtuple does. You can also write a class who's instances are other classes. That's called a "metatype" and in Python 3 metatypes need to inherit from the type class.
class MyMeta(type):
def __new__(meta, name, bases, dct):
print("creating a new type named", name)
return super().__new__(meta, name, bases, dct)
class MyClass1(MyMeta("Base", (), {})): # you can inherit from an instance
pass
class MyClass2(metaclass=MyMeta): # or use the special metaclass syntax
pass
While metaclasses are neat, they can be a bit confusing if you're new to them. It gets a bit metaphysical, with type being an instance of itself (and a subclass of object for good measure). This stuff is the magical core of Python's type system, and you don't really need to understand it to use classes in ordinary ways.
How can you access class values from within the top level class scope? What I mean by that is, how do you do something like:
class FooClass(object):
zeroith_base = __bases__[0]
.
.
.
What I'm specifically trying to do in this case is derive the metaclasses of all base classes to dynamically generate a metaclass that subclasses all the base classes' metaclasses to get past metclass conflict problems. I found http://code.activestate.com/recipes/204197-solving-the-metaclass-conflict/, and while all the concepts make sense to me, the actual code of the recipe is just beyond my ability to follow it. I don't want to use code I can't understand though, so instead, I tried to implement my own, more rudimentary system, but I'm stuck at square one trying to inspect the class object during creation.
You cannot inspect a class prior to its creation, and it has not yet been created yet until the suite of statements, or class body, have finished executing. The first time you have access to this information would be in the MetaClass.__new__ method of the class creating the class in question, or the execution of the thing creating the class in question, which technically need not be a meta-class or a class at all (as in the example below).
Here is a very rough prototype that probably does not work in all cases, but works in the simple case, and is probably easier to follow than the recipe.
def meta_class_synthesize(name, bases, attrmap):
seen = set()
seen_add = seen.add
metas = [type(base) for base in bases]
metas = tuple([
meta for meta in metas
if meta is not type and meta not in seen and not seen_add(meta)])
if not metas:
return type(name, bases, attrmap)
elif len(metas) == 1:
return metas[0](name, bases, attrmap)
newmeta_name = "__".join(meta.__name__ for meta in metas)
newmeta = type(newmeta_name, metas, {})
return newmeta(name, bases, attrmap)
class M_A(type):
pass
class M_B(type):
pass
class A:
__metaclass__ = M_A
class B:
__metaclass__ = M_B
class C(A, B):
__metaclass__ = meta_class_synthesize
print type(C) # prints "<class '__main__.M_A__M_B'>"
You'll find that __bases__ is not part of the class namespace. The class namespace is passed to the metaclass as the third parameter; the bases are passed as the second parameter. They are totally separate until the class is created.
So what you'll need to do is write a metaclass that synthesizes the metaclass you want, then uses that to create the class. I have no idea if that'll actually work, but I can't see any reason why it wouldn't.
class MType(type):
pass
class MClass(object):
__metaclass__ = MType
a = MType
From the above snippet is there anyway I can create an instance of MClass given a?
(It's probably important to note that I won't be able to simply contruct MClass())
Here's a horrific way of doing it. Although, I'll be honest.. I'm not sure what the difference between setting __metaclass__ as a module property is vs. inheritance.
class MType(type):
pass
class MClass(MType):
pass
a = MType
print a.__mro__[0].__subclasses__(a)
Yields:
[<class '__main__.MClass'>]
There are two straightforward ways I can think of to find the instances of a metaclass. One is to have the metaclass keep track of a list of its instances as they are created, as follows:
class MType(type):
instances = [] #list of instances of the metaclass
def __init__(cls, name, bases, dct):
MType.instances.append(cls) #append to list of instances
super(MType, cls).__init__(name, bases, dct)
class MClass(object):
__metaclass__ = MType
Now MType.instances is [<class '__main__.MClass'>].
Another way is to use the
inspect module to look through the classes in, say, a given module, and use isinstance to check if they are instances of MType. This isn't necessarily a crazy thing to do, either - for example, various unit testing suites work by looking through a module for classes and functions that appear to be tests, based on their names and/or inherited classes - I'm not really sure why you would want to do this with a metaclass, though.
No, if you have b = MClass, you can simply do obj = b(), but the metaclass does not "know" about the class(es) it is set as the metaclass of.
You could call __new__ on a if you have the class in a variable, whether b or MClass.