dynamic inheritance in Python through a decorator - python

I found this post where a function is used to inherit from a class:
def get_my_code(base):
class MyCode(base):
def initialize(self):
...
return MyCode
my_code = get_my_code(ParentA)
I would like to do something similar, but with a decorator, something like:
#decorator(base)
class MyClass(base):
...
Is this possible?
UPDATE
Say you have a class Analysis that is used throughout your code. Then you realize that you want to use a wrapper class Transient that is just a time loop on top of the analysis class. If in the code I replace the analysis class, but Transient(Analysis) everything breaks because an analysis class is expected, and thus all its attributes. The problem is that I can't just get to define class Transient(Analysis) in this way because there are plenty of analysis classes. I thought the best way to do this would be to have some sort of dynamic inheritance. Right now I use aggregation to redirect the functionality to the analysis class inside transient.

A class decorator actually gets the class already built - and instantiated (as a class object). It can perform changes on it's dict, and even wrap its methods with other decorators.
However, it means the class already has its bases set - and these can't be ordinarily changed. That implies you have to, in some ay rebuild the class inside the decorator code.
However, if the class'methods make use of parameterless super or __class__ cell variable, those are already set in the member functions (that in Python 3 are the same as unbound methods) you can't just create a new class and set those methods as members on the new one.
So, there might be a way, but it will be non-trivial. And as I pointed out in the comment above, I d like to understand what you'd like to be able to achieve with this, since one could just put the base class on the class declaration itself, instead of using it on the decorator configuration.
I've crafted a function that, as described above, creates a new class, "clonning" the original and can re-build all methods that use __class__ or super: it returns the new class which is functionally identical to the orignal one, but with the bases exchanged. If used in a decorator as requested (decorator code included), it will simply change the class bases. It can't handle decorated methods (other than classmethod and staticmethod), and don't take care of naming details - such as qualnames or repr for the methods.
from types import FunctionType
def change_bases(cls, bases, metaclass=type):
class Changeling(*bases, metaclass=metaclass):
def breeder(self):
__class__ #noQA
cell = Changeling.breeder.__closure__
del Changeling.breeder
Changeling.__name__ = cls.__name__
for attr_name, attr_value in cls.__dict__.items():
if isinstance(attr_value, (FunctionType, classmethod, staticmethod)):
if isinstance(attr_value, staticmethod):
func = getattr(cls, attr_name)
elif isinstance(attr_value, classmethod):
func = attr_value.__func__
else:
func = attr_value
# TODO: check if func is wrapped in decorators and recreate inner function.
# Although reaplying arbitrary decorators is not actually possible -
# it is possible to have a "prepare_for_changeling" innermost decorator
# which could be made to point to the new function.
if func.__closure__ and func.__closure__[0].cell_contents is cls:
franken_func = FunctionType(
func.__code__,
func.__globals__,
func.__name__,
func.__defaults__,
cell
)
if isinstance(attr_value, staticmethod):
func = staticmethod(franken_func)
elif isinstance(attr_value, classmethod):
func = classmethod(franken_func)
else:
func = franken_func
setattr(Changeling, attr_name, func)
continue
setattr(Changeling, attr_name, attr_value)
return Changeling
def decorator(bases):
if not isinstance(base, tuple):
bases = (bases,)
def stage2(cls):
return change_bases(cls, bases)
return stage2

Related

Does the final decorator help in preventing method overriding?

I am trying to create a final method in my class, where I want that it cannot be overridden by any sub-class, just like when creating a final class using final decorator which cannot be inherited.
from final_class import final
class Dummy:
def show(self):
print("show id running from dummy")
#final
def display(self):
print("display from dummy")
class Demo(Dummy):
def show(self):
print("show from demo")
def display(self):
print("display from demo")
d = Demo()
d.display()
I think we should get an error when accessing the display method from Demo, but when I run the program it gives "display from demo".
So what am I missing? I have checked final annotation and decorators in python3.8 but it talks about typechecking in typing packages while I was trying it from the final_class package.
As seem in the comments, the 3rd party library final_class.final is one thing: a class decorator that will prevent, at runtime, that a class is further inherited, anf typing.final which ships with Python, and is intended to decorate both classes and methods, but which has no enforcing behavior during program execution - it will, instead, make any compliant static analysis tool to raise an error in the type-checking stage.
It is, due to Python flexibility and dynamism, possible to create a final decorator for methods that will be enforced at runtime: i.e. whenever a subclass is created overriding a method marked as final in the inheritance chain, a RuntimeError, or other custom error can be raised.
The idea is that whenever a new class is created, both methods on the metaclass and the __init_subclass__ method of the bases is called - so, if one wants to create a custom metaclass or custom base-class to be used along with such a #final decorator, it should be something more or less straightforward.
What would be less straightforward would be such a decorator that would work regardless of an specific base class or custom-metaclass - and this also can be done: by injecting in the class being constructed an __init_subclass__ method which will perform a check of violation of the final clause.
The complicated part is to co-exist with eventual pre-existing __init_subclass__ methods which also need to be called, either on the same class or in any superclass, as well as emulate the working of super(), since we are creating a method outside the class body. The decorator code can inspect the context from which its called and inject a __init_subclass__ there, taking some care:
import sys
def final(method):
f = sys._getframe().f_back
_init_subclass_meth = "super"
def __init_subclass__(cls, *args, **kw):
# all of these are readonly, so nonlocal is optional:
# nonlocal final_methods, _init_subclass_meth, _original_init_subclass, __init_subclass__
# In a normal __init_subclass__, one can know about the class in which
# a method is declared, and call super(), via the `__class__`
# magic variable. But that won't work for a method defined
# outside the class and inkected in it.
# the line bellow should retrieve the equivalent to __class__
current_class = next(supercls for supercls in cls.__mro__ if getattr(supercls.__dict__.get("__init_subclass__", None), "__func__", None) is __init_subclass__)
for meth_name in cls.__dict__:
if meth_name in final_methods:
raise RuntimeError(f"Final method {meth_name} is redeclared in subclass {cls.__name__} from {current_class.__name__}")
if _init_subclass_meth == "wrap":
return _original_init_subclass(cls, *args, **kwd)
return super(current_class, None).__init_subclass__(*args, **kw)
__init_subclass__._final_mark = True
if "__init_subclass__" in f.f_locals and not getattr(f.f_locals["__init_subclass__"], "_final_mark", False):
_init_subclass_meth = "wrap"
_original_init_subclass = f.f_locals["__init_subclass__"]
# locals assignment: will work in this case because the caller context
# is a class body, inside which `f_locals` refers usually to a
# plain dict (unless a custom metaclass changed it).
# This normally would not work (= no effect) in an ordinary frame,
# represnting a plain function or method in execution:
f.f_locals["__init_subclass__"] = __init_subclass__
final_methods = f.f_locals.setdefault("_final_methods", set())
final_methods.add(method.__name__)
return method
class A:
#final
def b(self):
print("final b")
And this will raise an error:
class B(A):
def b(self):
# RuntimeError expected
...

Inheritance from metaclasses in Python

I've got simple metaclass, that turns methods of classes starting with "get_" to properties:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
new_attr = {}
for name, val in future_class_attr.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val)
else:
new_attr[name] = val
return type.__new__(mcs, future_class_name, future_class_parents, new_attr)
Imagine I have TestClass:
class TestClass():
def __init__(self, x: int):
self._x = x
def get_x(self):
print("this is property")
return self._x
I want it to work like this: I create some new class that kinda inherits from them both
class NewTestClass(TestClass, PropertyConvertMetaclass):
pass
and I could reuse their both methods like this:
obj = NewTestClass(8)
obj.get_x() # 8
obj.x # 8
As I take it, I should create a new class, lets name it PropertyConvert and make NewTestClass inherit from It:
class PropertyConvert(metaclass=PropertyConvertMetaclass):
pass
class NewTestClass(TestClass, PropertyConvert):
pass
But it doesn't help, I still can't use new property method with NewClassTest. How can I make PropertyConvert inherit all the methods from its brother, not doing anything inside NewClassTest, changing only PropertyConverterMetaclass or PropertyConverter? I'm new to metaclasses, so I'm sorry, if this question might seem silly.
When you do TestClass():, the body of the class is run in a namespace which becomes the class __dict__. The metaclass just informs the construction of that namespace via __new__ and __init__. In this case, you have set up the metaclass of TestClass to be type.
When you inherit from TestClass, e. g. with class NewTestClass(TestClass, PropertyConverter):, the version of PropertyConvertMetaclass you wrote operates on the __dict__ of NewTestClass only. TestClass has been created at that point, with no properties, because its metaclass way type, and the child class is empty, so you see no properties.
There are a couple of possible solutions here. The simpler one, but out of reach because of your assignment, is to do class TestClass(metaclass=PropertyConvertMetaclass):. All children of TestClass will have PropertyConvertMetaclass and so all getters will be converted to properties.
The alternative is to look carefully at the arguments of PropertyConvertMetaclass.__new__. Under normal circumstances, you only operate on the future_class_attr attribute. However, you have access to future_class_bases as well. If you want to upgrade the immediate siblings of PropertyConverter, that's all you need:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
# The loop is the same for each base __dict__ as for future_class_attr,
# so factor it out into a function
def update(d):
for name, value in d.items():
# Don't check for dunders: dunder can't start with `get_`
if name.startswith('get_') and callable(value):
prop = name[4:]
# Getter and setter can't be defined in separate classes
if 'set_' + prop in d and callable(d['set_' + prop]):
setter = d['set_' + prop]
else:
setter = None
if 'del_' + prop in d and callable(d['del_' + prop]):
deleter = d['del_' + prop]
else:
deleter = None
future_class_attr[prop] = property(getter, setter, deleter)
update(future_class_dict)
for base in future_class_parents:
# Won't work well with __slots__ or custom __getattr__
update(base.__dict__)
return super().__new__(mcs, future_class_name, future_class_parents, future_class_attr)
This is probably adequate for your assignment, but lacks a certain amount of finesse. Specifically, there are two deficiencies that I can see:
There is no lookup beyond the immediate base classes.
You can't define a getter in one class and a setter in another.
To address the first issue, you will have to traverse the MRO of the class. As #jsbueno suggests, this is easier to do on the fully constructed class using __init__ rather than the pre-class dictionary. I would solve the second issue by making a table of available getters and setters before making any properties. You could also make the properties respect MRO by doing this. The only complication with using __init__ is that you have to call setattr on the class rather than simply updating its future __dict__.
class PropertyConvertMetaclass(type):
def __init__(cls, class_name, class_parents, class_attr):
getters = set()
setters = set()
deleters = set()
for base in cls.__mro__:
for name, value in base.__dict__.items():
if name.startswith('get_') and callable(value):
getters.add(name[4:])
if name.startswith('set_') and callable(value):
setters.add(name[4:])
if name.startswith('del_') and callable(value):
deleters.add(name[4:])
for name in getters:
def getter(self, *args, **kwargs):
return getattr(super(cls, self), 'get_' + name)(*args, **kwargs)
if name in setters:
def setter(self, *args, **kwargs):
return getattr(super(cls, self), 'set_' + name)(*args, **kwargs)
else:
setter = None
if name in deleters:
def deleter(self, *args, **kwargs):
return getattr(super(cls, self), 'del_' + name)(*args, **kwargs)
else:
deleter = None
setattr(cls, name, property(getter, setter, deleter)
Anything that you do in the __init__ of a metaclass can just as easily be done with a class decorator. The main difference is that the metaclass will apply to all child classes, while a decorator only applies where it is used.
There is nothing "impossible" there.
It is a problem that, however unusual, can be solved with metaclasses.
Your approach is good - the problem you got is that when you look into the "future_class_attr" (also known as the namespace in the classbody), it only contains the methods and attributes for the class currently being defined . In your examples, NewTestClass is empty, and so is "future_class_attr".
The way to overcome that is to check instead on all base classes, looking for the methods that match the pattern you are looking for, and then creating the appropriate property.
Doing this correctly before creating the target class would be tricky - for one would have to do attribute searching in the correct mro (method resolution order) of all superclasses -and there can be a lot of corner cases. (but note it is not "impossible", nonetheless)
But nothing prevents you of doing that after creating the new class. For that, you can just assign the return value of super().__new__(mcls, ...) to a variable (by the way, prefer using super().__new__ instead of hardcoding type.__new__: this allows your metaclass to be colaborative and be combined with, say, collections.ABC or enum.Enum). That variable is them your completed class and you can use dir on it to check for all attribute and method names, already consolidating all superclasses - then, just create your new properties and assign then to the newly created class with setattr(cls_variable, property_name, property_object).
Better yet, write the metaclass __init__ instead of its __new__ method: you retrieve the new class already created, and can proceed to introspecting it with dir and adding the properties immediately. (don't forget to call super().__init__(...) even though your class don't need it.)
Also, note that since Python 3.6, the same results can be achieved with no metaclass at all, if one just implements the needed logic in the __init_subclass__ method of a base class.
One of the solutions of my problem is parsing parents' dicts in PropertyConvertMetaclass:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
new_attr = {}
for parent in future_class_parents:
for name, val in parent.__dict__.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val, parent.__dict__['set_' + name[4:]])
new_attr[name] = val
for name, val in future_class_attr.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val, future_class_attr['set_'+name[4:]])
new_attr[name] = val
return type.__new__(mcs, future_class_name, future_class_parents, new_attr)

Python naming convention for setter method name without args

I have been writing python code with classes that will have a method called something like:
def set_log_paths(self):
The thing is, this method doesn't take an argument, it determines what some values should be based on other values of self. Is it inappropriate to use the word "set" in this case? I ask because it isn't a direct getter or a setter as one would use in a language with private members.
Is there a common conventional word to use in my method name?
f you don't pass any values, and instead, compute the value at the moment the method is called, based on current values, it is reasonable that the verb describing the action be "update" - therefore update_log_paths().
Just double check you really need this design, and what are the chances of you/other users of your class forgetting calling these "update" methods.
Python's introspection easily allows adopting some elements from "reactive programing", which could be used to trigger these updater methods when the values they depend upon are changed.
One optimal choice for such an architecture would be a descriptor for your properties that upon having __set__ called would check a class-level registry to "see" if events should be triggered, and then one decorator that would enable you to list the attributes that would trigger it. A base class with a proper __init_subclass__ method could set everything up.
Let's suppose you will have the "base properties" on your class as annotated attributes in the class body - the descritptor, decorator and base-class code for this to work could be something along:
from functools import wraps
from collections import ChainMap
class EventDescriptor:
def __init__(self, name, default):
self.name = name
self.default = default
def __get__(self, instance, owner):
if not instance:
return self
return instance.__dict__[self.name] if self.name in instance.__dict__ else self.default
def __set__(self, instance, value):
instance.__dict__[self.name] = value
triggers = instance._post_change_registry.get(self.name, [])
for trigger in triggers:
getattr(instance, trigger)()
def triggered_by(*args):
def decorator(func):
func._triggered_by = args
return func
return decorator
class EventPropertyMixin:
def __init_subclass__(cls, **kw):
super.__init_subclass__(**kw)
for property_name, type_ in cls.__annotations__.items():
if not hasattr(cls, property_name):
raise TypeError("Properties without default values not supported in this example code")
# It would also be trivial to implement runtime type-checking in this point (and on the descriptor code)
setattr(cls, property_name, EventDescriptor(property_name, getattr(cls, property_name)))
# collects all registries in ancestor-classes, preserving order:
post_change_registry = ChainMap()
for ancestor in cls.__mro__[:0:-1]:
if hasattr(ancestor, "_post_change_registry"):
post_change_registry = post_change_registy.new_child(ancestor._post_change_registry)
post_change_registry = post_change_registry.new_child({})
for method_name, method in cls.__dict__.items():
if callable(method) and hasattr(method, "_triggered_by"):
for property_name in method._triggered_by:
triggers = post_change_registry.setdefault(property_name, [])
if method_name not in triggers:
triggers.append(method_name)
cls._post_change_registry = post_change_registry
class Test(EventPropertyMixin):
path1: str = ""
path2: str = ""
#triggered_by("path1", "path2")
def update_log_paths(self):
self.log_paths = self.path1 + self.path2
And let's this working:
In [2]: t = Test()
In [3]: t.path1 = "/tmp"
In [4]: t.path2 = "/inner"
In [5]: t.log_paths
Out[5]: '/tmp/inner'
So, this is complicated code, but code that usually would lie inside a framework, or in base utility libraries - with these 50 lines of code, you could be using Python to work for you, and have it call the updating methods, so their name won't matter at all! :-)
(ok, this code is way overkill for the question asked - but I was in a mood to produce something like this before sleeping tonight - disclaimer: I had not tested the inheritance-related corner cases covered in here)

Is there a way apply a decorator to a Python method that needs informations about the class?

When you decorate a method, it is not bound yet to the class, and therefor doesn't have the im_class attribute yet. I looking for a way to get the information about the class inside the decorator. I tried this:
import types
def decorator(method):
def set_signal(self, name, value):
print name
if name == 'im_class':
print "I got the class"
method.__setattr__ = types.MethodType(set_signal, method)
return method
class Test(object):
#decorator
def bar(self, foo):
print foo
But it doesn't print anything.
I can imagine doing this:
class Test(object):
#decorator(klass=Test)
def bar(self, foo):
print foo
But if I can avoid it, it would make my day.
__setattr__ is only called on explicit object.attribute = assignments; building a class does not use attribute assignment but builds a dictionary (Test.__dict__) instead.
To access the class you have a few different options though:
Use a class decorator instead; it'll be passed the completed class after building it, you could decorate individual methods on that class by replacing them (decorated) in the class. You could use a combination of a function decorator and a class decorator to mark which methods are to be decorated:
def methoddecoratormarker(func):
func._decorate_me = True
return func
def realmethoddecorator(func):
# do something with func.
# Note: it is still an unbound function here, not a method!
return func
def classdecorator(klass):
for name, item in klass.__dict__.iteritems():
if getattr(item, '_decorate_me', False):
klass.__dict__[name] = realmethoddecorator(item)
You could use a metaclass instead of a class decorator to achieve the same, of course.
Cheat, and use sys._getframe() to retrieve the class from the calling frame:
import sys
def methoddecorator(func):
callingframe = sys._getframe(1)
classname = callingframe.f_code.co_name
Note that all you can retrieve is the name of the class; the class itself is still being built at this time. You can add items to callingframe.f_locals (a mapping) and they'll be made part of the new class object.
Access self whenever the method is called. self is a reference to the instance after all, and self.__class__ is going to be, at the very least, a sub-class of the original class the function was defined in.
My strict answer would be: It's not possible, because the class does not yet exist when the decorator is executed.
The longer answer would depend on your very exact requirements. As I wrote, you cannot access the class if it does not yet exists. One solution would be, to mark the decorated method to be "transformed" later. Then use a metaclass or class decorator to apply your modifications after the class has been created.
Another option involves some magic. Look for the implementation of the implements method in zope.interfaces. It has some access to the information about the class which is just been parsed. Don't know if it will be enough for your use case.
You might want to take a look at descriptors. They let you implement a __get__ that is used when an attribute is accessed, and can return different things depending on the object and its type.
Use method decorators to add some marker attributes to the interesting methods, and use a metaclass which iterates over the methods, finds the marker attributes, and does the logic. The metaclass code is run when the class is created, so it has a reference to the newly created class.
class MyMeta(object):
def __new__(...):
...
cls = ...
... iterate over dir(cls), find methods having .is_decorated, act on them
return cls
def decorator(f):
f.is_decorated = True
return f
class MyBase(object):
__metaclass__ = MyMeta
class MyClass(MyBase):
#decorator
def bar(self, foo):
print foo
If you worry about that the programmer of MyClass forgets to use MyBase, you can forcibly set the metaclass in decorator, by exampining the globals dicitionary of the caller stack frame (sys._getframe()).

How to find class of bound method during class construction in Python 3.1?

i want to write a decorator that enables methods of classes to become visible to other parties; the problem i am describing is, however, independent of that detail. the code will look roughly like this:
def CLASS_WHERE_METHOD_IS_DEFINED( method ):
???
def foobar( method ):
print( CLASS_WHERE_METHOD_IS_DEFINED( method ) )
class X:
#foobar
def f( self, x ):
return x ** 2
my problem here is that the very moment that the decorator, foobar(), gets to see the method, it is not yet callable; instead, it gets to see an unbound version of it. maybe this can be resolved by using another decorator on the class that will take care of whatever has to be done to the bound method. the next thing i will try to do is to simply earmark the decorated method with an attribute when it goes through the decorator, and then use a class decorator or a metaclass to do the postprocessing. if i get that to work, then i do not have to solve this riddle, which still puzzles me:
can anyone, in the above code, fill out meaningful lines under CLASS_WHERE_METHOD_IS_DEFINED so that the decorator can actually print out the class where f is defined, the moment it gets defined? or is that possibility precluded in python 3?
When the decorator is called, it's called with a function as its argument, not a method -- therefore it will avail nothing to the decorator to examine and introspect its method as much as it wants to, because it's only a function and carries no information whatsoever about the enclosing class. I hope this solves your "riddle", although in the negative sense!
Other approaches might be tried, such as deep introspection on nested stack frames, but they're hacky, fragile, and sure not to carry over to other implementations of Python 3 such as pynie; I would therefore heartily recommend avoiding them, in favor of the class-decorator solution that you're already considering and is much cleaner and more solid.
As I mentioned in some other answers, since Python 3.6 the solution to this problem is very easy thanks to object.__set_name__ which gets called with the class object that is being defined.
We can use it to define a decorator that has access to the class in the following way:
class class_decorator:
def __init__(self, fn):
self.fn = fn
def __set_name__(self, owner, name):
# do something with "owner" (i.e. the class)
print(f"decorating {self.fn} and using {owner}")
# then replace ourself with the original method
setattr(owner, name, self.fn)
Which can then be used as a normal decorator:
>>> class A:
... #class_decorator
... def hello(self, x=42):
... return x
...
decorating <function A.hello at 0x7f9bedf66bf8> and using <class '__main__.A'>
>>> A.hello
<function __main__.A.hello(self, x=42)>
This is a very old post, but introspection isn't the way to solve this problem, because it can be more easily solved with a metaclass and a bit of clever class construction logic using descriptors.
import types
# a descriptor as a decorator
class foobar(object):
owned_by = None
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
# a proxy for `func` that gets used when
# `foobar` is referenced from by a class
return self.func(*args, **kwargs)
def __get__(self, inst, cls=None):
if inst is not None:
# return a bound method when `foobar`
# is referenced from by an instance
return types.MethodType(self.func, inst, cls)
else:
return self
def init_self(self, name, cls):
print("I am named '%s' and owned by %r" % (name, cls))
self.named_as = name
self.owned_by = cls
def init_cls(self, cls):
print("I exist in the mro of %r instances" % cls)
# don't set `self.owned_by` here because
# this descriptor exists in the mro of
# many classes, but is only owned by one.
print('')
The key to making this work is the metaclass - it searches through the attributes defined on the classes it creates to find foobar descriptors. Once it does, it passes them information about the classes they are involved in through the descriptor's init_self and init_cls methods.
init_self is called only for the class which the descriptor is defined on. This is where modifications to foobar should be made, because the method is only called once. While init_cls is called for all classes which have access to the decorated method. This is where modifications to the classes foobar can be referenced by should be made.
import inspect
class MetaX(type):
def __init__(cls, name, bases, classdict):
# The classdict contains all the attributes
# defined on **this** class - no attribute in
# the classdict is inherited from a parent.
for k, v in classdict.items():
if isinstance(v, foobar):
v.init_self(k, cls)
# getmembers retrieves all attributes
# including those inherited from parents
for k, v in inspect.getmembers(cls):
if isinstance(v, foobar):
v.init_cls(cls)
example
# for compatibility
import six
class X(six.with_metaclass(MetaX, object)):
def __init__(self):
self.value = 1
#foobar
def f(self, x):
return self.value + x**2
class Y(X): pass
# PRINTS:
# I am named 'f' and owned by <class '__main__.X'>
# I exist in the mro of <class '__main__.X'> instances
# I exist in the mro of <class '__main__.Y'> instances
print('CLASS CONSTRUCTION OVER\n')
print(Y().f(3))
# PRINTS:
# 10

Categories

Resources