I wrote a metaclass to handle the creation of two classes at class definition (with one accessible as an attribute of the other).
MRE with logic removed:
class MyMeta(type):
def __new__(mcs, clsname, bases, namespace, **kwds):
result = super().__new__(mcs, clsname, bases, namespace)
# Omitted logic to prefix namespace
print(namespace)
# namespace.pop('__classcell__', None)
disambiguated_cls = super().__new__(mcs, clsname + 'Prefixed', bases, namespace)
result.disambiguated_cls = disambiguated_cls
return result
class A(metaclass=MyMeta):
#classmethod
def f(cls, x):
return x + 1
class B(A):
#classmethod
def f(cls, x):
# return A.f(x)
return super().f(x)
Anytime super is called (e.g. in class B definition) a __classcell__ object is added to the namespace. If that __classcell__ object is duplicated (e.g. in the creation of disambiguated_cls) the following exception is raised:
TypeError: __class__ set to <class '__main__.B'> defining 'B' as <class '__main__.B'>
Looks like there are two solutions (corresponding to the two commented lines in the example):
Don't call super (just directly call function on the superclass: A.f(x))
Pop the __classcell__ out of the namespace
The first means manual resolution of superclasses and I'm not sure what downstream effects the latter can have.
For reference the docs on __classcell__ in metaclasses state:
In CPython 3.6 and later, the __class__ cell is passed to the metaclass as a __classcell__ entry in the class namespace. If present, this must be propagated up to the type.__new__ call in order for the class to be initialised correctly. Failing to do so will result in a RuntimeError in Python 3.8.
Is there another option? What is the best way to do this?
It feels like the simpler way is to partially create a new "blank" class with a __classcell__ and populate it with the namespace of the original class before calling type.__new__ for the second class.
If that works, it might be worth using it - otherwise, if there would be no multiple inheritance or complicated hierarchy involved, hard-coding the call to the superclass might work. However - it seems to be that hardcoding the call is impossible, since calling the hard-coded method inside the "disambiguated" class would call the outer ("result") class in the parent.
One point to take care is that all functions (including functions decorated with #classmethod) have their closures bound to the value provided in the original __classcell__. So, in order to have the same functions to participate as methods in a second class they have to be rebound to a fresh __classcell__ value, one created along the second class. Rebindind a function closure is not that straightforward, since functions are immutable. The way to do it is to create a new "function object" with the __code__ and other attributes of the original function, passing in a new closure. Once that is done, the rest is straightforward
from types import FunctionType
class MyMeta(type):
def __new__(mcls, clsname, bases, namespace, **kwds):
result = super().__new__(mcls, clsname, bases, namespace)
# Omitted logic to prefix namespace
print(namespace)
# the upside of having "nested_process" here instead of
# inside the other metaclass is transparent
# read access to the attributes of the original class:
def nested_process(stubname, stubbases, stub_namespace):
new_namespace = namespace.copy()
classcell = stub_namespace["__classcell__"]
new_namespace["__classcell__"] = classcell
for m_name, method in new_namespace.items():
is_classmeth = False
if isinstance(method, classmethod):
is_classmeth = True
method = method.__func__
if not isinstance(method, FunctionType):
continue
if not (free:=method.__code__.co_freevars) or free[0] != "__class__":
continue
method = FunctionType(
method.__code__, method.__globals__, method.__name__,
method.__defaults__,
closure=(classcell,)
)
if is_classmeth:
method = classmethod(method)
new_namespace[m_name] = method
# maybe modify namespace __qualname__.
# maybe rewrite "bases" to point to prefixed versions?
return clsname + "Prefixed", bases, new_namespace
class disambiguated_cls(metaclass=NestedMeta, clone_hook=nested_process):
def _stub(self):
_ = __class__
result.disambiguated_cls = disambiguated_cls
return result
class NestedMeta(MyMeta):
# the new syntethic class need this modified __new__.
# but as it also inherits from a class created by
# "MyMeta", this metaclass must inherit from it
# otherwise "metaclass conflict" is raised.
def __new__(mcls, clsname, bases, namespace, clone_hook):
clsname, bases, namespace = clone_hook(clsname, bases, namespace)
# surprise: we have to hardcode "type" here:
return type.__new__(mcls, clsname, bases, namespace)
# if "super()" is desired, than an extra KWD parameter
# can be passed to "MyMeta.__new__" so it skip its nesting+prefixed logic
# ensure a eventual __init__ from MyMeta is not called:
__init__ = lambda *args, **kw: None
class A(metaclass=MyMeta):
#classmethod
def f(cls, x):
return x + 1
class B(A):
#classmethod
def f(cls, x):
return super().f(x)
Related
PEP 3115 has the following example of using the __prepare__ method of a metaclass (print statements are mine):
# The custom dictionary
class member_table(dict):
def __init__(self):
self.member_names = []
def __setitem__(self, key, value):
print(f'in __setitem__{key, value}')
# if the key is not already defined, add to the
# list of keys.
if key not in self:
self.member_names.append(key)
# Call superclass
dict.__setitem__(self, key, value)
# The metaclass
class OrderedClass(type):
# The prepare function
#classmethod
def __prepare__(metacls, name, bases): # No keywords in this case
print('in __prepare__')
return member_table()
# The metaclass invocation
def __new__(cls, name, bases, classdict):
print('in __new__')
# Note that we replace the classdict with a regular
# dict before passing it to the superclass, so that we
# don't continue to record member names after the class
# has been created.
result = type.__new__(cls, name, bases, dict(classdict))
result.member_names = classdict.member_names
return result
print('before MyClass')
class MyClass(metaclass=OrderedClass):
print('in MyClass 1')
# method1 goes in array element 0
def method1(self):
pass
print('in MyClass 2')
# method2 goes in array element 1
def method2(self):
pass
print('in MyClass 3')
Running this, prints this:
before MyClass
in __prepare__
in __setitem__('__module__', '__main__')
in __setitem__('__qualname__', 'MyClass')
in MyClass 1
in __setitem__('method1', <function MyClass.method1 at 0x7fa70414da60>)
in MyClass 2
in __setitem__('method2', <function MyClass.method2 at 0x7fa70414daf0>)
in MyClass 3
in __new__
So it seems like when MyClass is executed, execution first goes to the class's metaclass's __prepare__ method which returns member_table() (who/what uses this return value?), then something sets the class's __module__ and __qualname__, then executes the class body, which sets the class's methods (method1 and method2), then the __new__ method is called with the return value of __prepare__ as the classdict argument value to __new__ (who/what is passing along this value?).
I tried to step through the execution in thonny's debugger, but that threw an error. I also tried stepping through execution in pythontutor.com, but that wasn't granular enough. I pdb'ed it, but it was hard to follow what was going on. Finally, I added some print statements, which are present in the code above.
The result of the prepare() is the namespace argument that gets passed to __new__. It is the namespace in which the body of the class is evaluated (see [1]).
So within the newly created class, you can see the values of MyClass.__module__, MyClass.__qualname__, etc because they are being assigned in the namespace object of MyClass.
Most uses of metaclasses have no need for prepare(), and an ordinary namespace is used.
[1] https://docs.python.org/3.9/reference/datamodel.html?highlight=__prepare__#preparing-the-class-namespace
I have a MetaClass that, at the moment, simply returns an obj of a class.
From this MetaClass, I derieved two different classes. The difference in the classes is a dictionary that gets passed to __init__ for one of the classes but not for the other one.
If this dict only contains one single entry, I want python to return an instance of the other class instead of the one that was actually called.
class _MetaClass(type):
def __new__(cls, name, bases, dct):
return super(_MetaClass, cls).__new__(name, bases, **dct)
class Class1(object):
__metaclass__ = _MetaClass
def __init__(self, someargs):
pass
class Class2(object):
__metaclass__ = _MetaClass
def __init__(self, someargs, kwargs):
if len(kwargs) == 1:
return Class1(someargs)
else:
pass
TestInstance = Class2("foo", {"bar":"foo"}) #should now be Class1 instance because only one item in dct
If the dict, like in this case, has only len(dct) == 1, then it should create an instance of Class1 with the "foo" passed to its __init__ instead of returning an instance of Class2 as it normally would
I tried to implement the __new__ and __init__ methods for the MetaClass, but I could not figure out how to check the arguments that are actually passes on new class instantiation
You could create a handlerMethod to deal with the issue
def handler(yourDict):
if len(yourDict) == 1:
return Class2(yourDict)
else:
return Class1(yourDict)
and then call the handler instead of the constructor
TestInstance = handler(yourDict)
Here is an executable code which works in Python 2.7 but results in an error in Python 3.6:
import six
class AMeta(type):
def __new__(cls, name, bases, attrs):
module = attrs.pop('__module__')
new_attrs = {'__module__': module}
classcell = attrs.pop('__classcell__', None)
if classcell is not None:
new_attrs['__classcell__'] = classcell
new = super(AMeta, cls).__new__(
cls, name, bases, new_attrs)
new.duplicate = False
legacy = super(AMeta, cls).__new__(
cls, 'Legacy' + name, (new,), new_attrs)
legacy.duplicate = True
return new
#six.add_metaclass(AMeta)
class A():
def pr(cls):
print('a')
class B():
def pr(cls):
print('b')
class C(A,B):
def pr(cls):
super(C, cls).pr() # not shown with new_attrs
B.pr(cls)
print('c') # not shown with new_attrs
c = C()
c.pr()
# Expected result
# a
# b
# c
I get the following error:
Traceback (most recent call last):
File "test.py", line 28, in <module>
class C(A,B):
TypeError: __class__ set to <class '__main__.LegacyC'> defining 'C' as <class '__main__.C'>
C is inherit from A that is generated with the metaclass AMeta. They are tests classes and AMeta's goal is to execute all the tests with 2 different file folders: the default one and the legacy one.
I found a way to remove thise error by removing classcell from attrs, then returning new = super(AMeta, cls).new(cls, name, bases, attrs) (not new_attrs) but it doesn't seem right, and if it is, I'd like to know why.
The goal of new_attrs resulted from this SO question or from the documentation where it states basically the opposite: when modifying the attrs, make sure to keep classcell because it is deprecated in Python 3.6 and will result in an error in Python 3.8.
Note that in this case, it removes the pr definition because they weren't passed to new_attrs, thus prints 'b' instead of 'abc', but is irrelevant for this problem.
Is there a way to call multiple super().new inside the new of a metaclass AMeta, and then call them from a class C inheriting from the class inheriting A ?
Without nesting inheritance, the error doesn't appear, like this:
import six
class AMeta(type):
def __new__(cls, name, bases, attrs):
new = super(AMeta, cls).__new__(
cls, name, bases, attrs)
new.duplicate = False
legacy = super(AMeta, cls).__new__(
cls, 'Duplicate' + name, (new,), attrs)
legacy.duplicate = True
return new
#six.add_metaclass(AMeta)
class A():
def pr(cls):
print('a')
a = A()
a.pr()
# Result
# a
Then maybe it is A's role to do something to fix it?
Thanks by advance,
What your problem is I can figure out, and how to work around it
The problem is that when you do what you are doing, you are passing the same cell object to both copies of your class: the original and the legacy one.
As it exists in two classes at once, it conflicts with the other place it is in use when one tries to make use of it - super() will pick the wrong ancestor class when called.
cell objects are picky, they are created in native code, and can't be created or configured on the Python side. I could figure out a way of creating the class copy by having a method that will return a fresh cell object, and passing that as __classcell__.
(I also tried to simply run copy.copy/copy.deepcopy on the classcell object -before resorting to my cellfactory bellow - it does not work)
In order to reproduce the problem and figure out a solution I made a simpler version of your metaclass, Python3 only.
from types import FunctionType
legacies = []
def classcellfactory():
class M1(type):
def __new__(mcls, name, bases, attrs, classcellcontainer=None):
if isinstance(classcellcontainer, list):
classcellcontainer.append(attrs.get("__classcell__", None))
container = []
class T1(metaclass=M1, classcellcontainer=container):
def __init__(self):
super().__init__()
return container[0]
def cellfactory():
x = None
def helper():
nonlocal x
return helper.__closure__[0]
class M(type):
def __new__(mcls, name, bases, attrs):
cls1 = super().__new__(mcls, name + "1", bases, attrs)
new_attrs = attrs.copy()
if "__classcell__" in new_attrs:
new_attrs["__classcell__"] = cellclass = cellfactory()
for name, obj in new_attrs.items():
if isinstance(obj, FunctionType) and obj.__closure__:
new_method = FunctionType(obj.__code__, obj.__globals__, obj.__name__, obj.__defaults__, (cellclass, ))
new_attrs[name] = new_method
cls2 = super().__new__(mcls, name + "2", bases, new_attrs)
legacies.append(cls2)
return cls1
class A(metaclass=M):
def meth(self):
print("at A")
class B(A):
pass
class C(B,A):
def meth(self):
super().meth()
C()
So, not only I create a nested-function in order to have the Python runtime create a separate cell object, that I then use in the cloned class - but also, methods that make use of the cellclass have to be re-created with a new __closure__ that points to the new cell var.
Without recreating the methods, they won't work in the clonned class - as super() in the cloned-class' methods will expect the cell pointing to the cloned class itself, but it points to the original one.
Fortunately, methods in Python 3 are plain functions - that makes the code simpler. However, that code won't run in Python 2 - so, just enclose it in an if block not to run on Python2. As the __cellclass__ attribute does not even exist there, there is no problem at all.
After pasting the code above in a Python shell I can run both methods and super() works:
In [142]: C().meth()
at A
In [143]: legacies[-1]().meth()
at A
I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:
class ShowMeList(object):
def __init__(self, it):
self._data = list(it)
def __getattr__(self, name):
attr = object.__getattribute__(self._data, name)
if callable(attr):
def wrapper(*a, **kw):
print "before the call"
result = attr(*a, **kw)
print "after the call"
return result
return wrapper
return attr
If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
If I don't inherit from object (first line class ShowMeList:) everything works as expected:
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
How do I accomplish this intercept with new style classes?
For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.
class Wrapper(object):
"""Wrapper class that provides proxy access to some internal instance."""
__wraps__ = None
__ignore__ = "class mro new init setattr getattr getattribute"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
# provide proxy access to regular attributes of wrapped object
def __getattr__(self, name):
return getattr(self._obj, name)
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Usage:
class DictWrapper(Wrapper):
__wraps__ = dict
wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))
# make sure it worked....
assert "b" in wrapped_dict # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3) # __eq__
assert "'a': 1" in str(wrapped_dict) # __str__
assert wrapped_dict.__doc__.startswith("dict()") # __doc__
Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.
Consider the following:
>>> class C:
x = 1
def __init__(self):
self.y = 2
def __getattr__(self, attr):
print(attr)
>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z
The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).
On your example, the __repr__ and many other magic methods are already defined in the object class.
One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.
As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.
Cut and copy from the documentation:
For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
I'm trying to intercept calls to python's double underscore magic methods in new style classes. This is a trivial example but it show's the intent:
class ShowMeList(object):
def __init__(self, it):
self._data = list(it)
def __getattr__(self, name):
attr = object.__getattribute__(self._data, name)
if callable(attr):
def wrapper(*a, **kw):
print "before the call"
result = attr(*a, **kw)
print "after the call"
return result
return wrapper
return attr
If I use that proxy object around list I get the expected behavior for non-magic methods but my wrapper function is never called for magic methods.
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
<__main__.ShowMeList object at 0x9640eac>
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
If I don't inherit from object (first line class ShowMeList:) everything works as expected:
>>> l = ShowMeList(range(8))
>>> l #call to __repr__
before the call
after the call
[0, 1, 2, 3, 4, 5, 6, 7]
>>> l.append(9)
before the call
after the call
>> len(l._data)
9
How do I accomplish this intercept with new style classes?
For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.
class Wrapper(object):
"""Wrapper class that provides proxy access to some internal instance."""
__wraps__ = None
__ignore__ = "class mro new init setattr getattr getattribute"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
# provide proxy access to regular attributes of wrapped object
def __getattr__(self, name):
return getattr(self._obj, name)
# create proxies for wrapped object's double-underscore attributes
class __metaclass__(type):
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
Usage:
class DictWrapper(Wrapper):
__wraps__ = dict
wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))
# make sure it worked....
assert "b" in wrapped_dict # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3) # __eq__
assert "'a': 1" in str(wrapped_dict) # __str__
assert wrapped_dict.__doc__.startswith("dict()") # __doc__
Using __getattr__ and __getattribute__ are the last resources of a class to respond to getting an attribute.
Consider the following:
>>> class C:
x = 1
def __init__(self):
self.y = 2
def __getattr__(self, attr):
print(attr)
>>> c = C()
>>> c.x
1
>>> c.y
2
>>> c.z
z
The __getattr__ method is only called when nothing else works (It will not work on operators, and you can read about that here).
On your example, the __repr__ and many other magic methods are already defined in the object class.
One thing can be done, thought, and it is to define those magic methods and make then call the __getattr__ method. Check this other question by me and its answers (link) to see some code doing that.
As of the answers to Asymmetric behavior for __getattr__, newstyle vs oldstyle classes (see also the Python docs), modifying access to "magic" methods with __getattr__ or __getattribute__ is just not possible with new-style classes. This restriction makes the interpreter much faster.
Cut and copy from the documentation:
For old-style classes, special methods are always looked up in exactly the same way as any other method or attribute.
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.