For putting methods of various classes into a global registry I'm using a decorator with a metaclass. The decorator tags, the metaclass puts the function in the registry:
class ExposedMethod (object):
def __init__(self, decoratedFunction):
self._decoratedFunction = decoratedFunction
def __call__(__self,*__args,**__kw):
return __self._decoratedFunction(*__args,**__kw)
class ExposedMethodDecoratorMetaclass(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, ExposedMethod):
WorkerFunctionRegistry.addWorkerToWorkerFunction(obj_name, name)
return type.__new__(mcs, name, bases, dct)
class MyClass (object):
__metaclass__ = DiscoveryExposedMethodDecoratorMetaclass
#ExposeDiscoveryMethod
def myCoolExposedMethod (self):
pass
I've now came to the point where two function registries are needed. The first thought was to subclass the metaclass and put the other registry in. For that the new method has simply to be rewritten.
Since rewriting means redundant code this is not what I really want. So, it would be nice if anyone could name a way how to put an attribute inside of the metaclass which is able to be read when new is executed. With that the right registry could be put in without having to rewrite new.
Your ExposedMethod instances do not behave as normal instance methods but rather like static methods -- the fact that you're giving one of them a self argument hints that you're not aware of that. You may need to add a __get__ method to the ExposedMethod class to make it a descriptor, just like function objects are -- see here for more on descriptors.
But there is a much simpler way, since functions can have attributes...:
def ExposedMethod(registry=None):
def decorate(f):
f.registry = registry
return f
return decorate
and in a class decorator (simpler than a metaclass! requires Python 2.6 or better -- in 2.5 or earlier you'll need to stick w/the metaclass or explicitly call this after the class statement, though the first part of the answer and the functionality of the code below are still perfectly fine):
def RegisterExposedMethods(cls):
for name, f in vars(cls).iteritems():
if not hasattr(f, 'registry'): continue
registry = f.registry
if registry is None:
registry = cls.registry
registry.register(name, cls.__name__)
return cls
So you can do:
#RegisterExposedMethods
class MyClass (object):
#ExposeMethod(WorkerFunctionRegistry)
def myCoolExposedMethod (self):
pass
and the like. This is easily extended to allowing an exposed method to have several registries, get the default registry elsewhere than from the class (it could be in the class decorator, for example, if that works better for you) and avoids getting enmeshed with metaclasses without losing any functionality. Indeed that's exactly why class decorators were introduced in Python 2.6: they can take the place of 90% or so of practical uses of metaclasses and are much simpler than custom metaclasses.
You can use a class attribute to point to the registry you want to use in the specialized metaclasses, e.g. :
class ExposedMethodDecoratorMetaclassBase(type):
registry = None
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.items():
if isinstance(obj, ExposedMethod):
mcs.registry.register(obj_name, name)
return type.__new__(mcs, name, bases, dct)
class WorkerExposedMethodDecoratorMetaclass(ExposedMethodDecoratorMetaclassBase):
registry = WorkerFunctionRegistry
class RetiredExposedMethodDecoratorMetaclass(ExposedMethodDecoratorMetaclassBase):
registry = RetiredFunctionRegistry
Thank you both for your answers. Both helped alot to find a proper way for my request.
My final solution to the problem is the following:
def ExposedMethod(decoratedFunction):
decoratedFunction.isExposed = True
return decoratedFunction
class RegisterExposedMethods (object):
def __init__(self, decoratedClass, registry):
self._decoratedClass = decoratedClass
for name, f in vars(self._decoratedClass).iteritems():
if hasattr(f, "isExposed"):
registry.addComponentClassToComponentFunction(name, self._decoratedClass.__name__)
# cloak us as the original class
self.__class__.__name__ = decoratedClass.__name__
def __call__(self,*__args,**__kw):
return self._decoratedClass(*__args,**__kw)
def __getattr__(self, name):
return getattr(self._decoratedClass, name)
On a Class I wish to expose methods from I do the following:
#RegisterExposedMethods
class MyClass (object):
#ExposedMethod
def myCoolExposedMethod (self):
pass
The class decorator is now very easy to be subclassed. Here is an example:
class DiscoveryRegisterExposedMethods (RegisterExposedMethods):
def __init__(self, decoratedClass):
RegisterExposedMethods.__init__(self,
decoratedClass,
DiscoveryFunctionRegistry())
With that the comment of Alex
Your ExposedMethod instances do not behave as normal instance methods ...
is no longer true, since the method is simply tagged and not wrapped.
Related
I've got simple metaclass, that turns methods of classes starting with "get_" to properties:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
new_attr = {}
for name, val in future_class_attr.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val)
else:
new_attr[name] = val
return type.__new__(mcs, future_class_name, future_class_parents, new_attr)
Imagine I have TestClass:
class TestClass():
def __init__(self, x: int):
self._x = x
def get_x(self):
print("this is property")
return self._x
I want it to work like this: I create some new class that kinda inherits from them both
class NewTestClass(TestClass, PropertyConvertMetaclass):
pass
and I could reuse their both methods like this:
obj = NewTestClass(8)
obj.get_x() # 8
obj.x # 8
As I take it, I should create a new class, lets name it PropertyConvert and make NewTestClass inherit from It:
class PropertyConvert(metaclass=PropertyConvertMetaclass):
pass
class NewTestClass(TestClass, PropertyConvert):
pass
But it doesn't help, I still can't use new property method with NewClassTest. How can I make PropertyConvert inherit all the methods from its brother, not doing anything inside NewClassTest, changing only PropertyConverterMetaclass or PropertyConverter? I'm new to metaclasses, so I'm sorry, if this question might seem silly.
When you do TestClass():, the body of the class is run in a namespace which becomes the class __dict__. The metaclass just informs the construction of that namespace via __new__ and __init__. In this case, you have set up the metaclass of TestClass to be type.
When you inherit from TestClass, e. g. with class NewTestClass(TestClass, PropertyConverter):, the version of PropertyConvertMetaclass you wrote operates on the __dict__ of NewTestClass only. TestClass has been created at that point, with no properties, because its metaclass way type, and the child class is empty, so you see no properties.
There are a couple of possible solutions here. The simpler one, but out of reach because of your assignment, is to do class TestClass(metaclass=PropertyConvertMetaclass):. All children of TestClass will have PropertyConvertMetaclass and so all getters will be converted to properties.
The alternative is to look carefully at the arguments of PropertyConvertMetaclass.__new__. Under normal circumstances, you only operate on the future_class_attr attribute. However, you have access to future_class_bases as well. If you want to upgrade the immediate siblings of PropertyConverter, that's all you need:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
# The loop is the same for each base __dict__ as for future_class_attr,
# so factor it out into a function
def update(d):
for name, value in d.items():
# Don't check for dunders: dunder can't start with `get_`
if name.startswith('get_') and callable(value):
prop = name[4:]
# Getter and setter can't be defined in separate classes
if 'set_' + prop in d and callable(d['set_' + prop]):
setter = d['set_' + prop]
else:
setter = None
if 'del_' + prop in d and callable(d['del_' + prop]):
deleter = d['del_' + prop]
else:
deleter = None
future_class_attr[prop] = property(getter, setter, deleter)
update(future_class_dict)
for base in future_class_parents:
# Won't work well with __slots__ or custom __getattr__
update(base.__dict__)
return super().__new__(mcs, future_class_name, future_class_parents, future_class_attr)
This is probably adequate for your assignment, but lacks a certain amount of finesse. Specifically, there are two deficiencies that I can see:
There is no lookup beyond the immediate base classes.
You can't define a getter in one class and a setter in another.
To address the first issue, you will have to traverse the MRO of the class. As #jsbueno suggests, this is easier to do on the fully constructed class using __init__ rather than the pre-class dictionary. I would solve the second issue by making a table of available getters and setters before making any properties. You could also make the properties respect MRO by doing this. The only complication with using __init__ is that you have to call setattr on the class rather than simply updating its future __dict__.
class PropertyConvertMetaclass(type):
def __init__(cls, class_name, class_parents, class_attr):
getters = set()
setters = set()
deleters = set()
for base in cls.__mro__:
for name, value in base.__dict__.items():
if name.startswith('get_') and callable(value):
getters.add(name[4:])
if name.startswith('set_') and callable(value):
setters.add(name[4:])
if name.startswith('del_') and callable(value):
deleters.add(name[4:])
for name in getters:
def getter(self, *args, **kwargs):
return getattr(super(cls, self), 'get_' + name)(*args, **kwargs)
if name in setters:
def setter(self, *args, **kwargs):
return getattr(super(cls, self), 'set_' + name)(*args, **kwargs)
else:
setter = None
if name in deleters:
def deleter(self, *args, **kwargs):
return getattr(super(cls, self), 'del_' + name)(*args, **kwargs)
else:
deleter = None
setattr(cls, name, property(getter, setter, deleter)
Anything that you do in the __init__ of a metaclass can just as easily be done with a class decorator. The main difference is that the metaclass will apply to all child classes, while a decorator only applies where it is used.
There is nothing "impossible" there.
It is a problem that, however unusual, can be solved with metaclasses.
Your approach is good - the problem you got is that when you look into the "future_class_attr" (also known as the namespace in the classbody), it only contains the methods and attributes for the class currently being defined . In your examples, NewTestClass is empty, and so is "future_class_attr".
The way to overcome that is to check instead on all base classes, looking for the methods that match the pattern you are looking for, and then creating the appropriate property.
Doing this correctly before creating the target class would be tricky - for one would have to do attribute searching in the correct mro (method resolution order) of all superclasses -and there can be a lot of corner cases. (but note it is not "impossible", nonetheless)
But nothing prevents you of doing that after creating the new class. For that, you can just assign the return value of super().__new__(mcls, ...) to a variable (by the way, prefer using super().__new__ instead of hardcoding type.__new__: this allows your metaclass to be colaborative and be combined with, say, collections.ABC or enum.Enum). That variable is them your completed class and you can use dir on it to check for all attribute and method names, already consolidating all superclasses - then, just create your new properties and assign then to the newly created class with setattr(cls_variable, property_name, property_object).
Better yet, write the metaclass __init__ instead of its __new__ method: you retrieve the new class already created, and can proceed to introspecting it with dir and adding the properties immediately. (don't forget to call super().__init__(...) even though your class don't need it.)
Also, note that since Python 3.6, the same results can be achieved with no metaclass at all, if one just implements the needed logic in the __init_subclass__ method of a base class.
One of the solutions of my problem is parsing parents' dicts in PropertyConvertMetaclass:
class PropertyConvertMetaclass(type):
def __new__(mcs, future_class_name, future_class_parents, future_class_attr):
new_attr = {}
for parent in future_class_parents:
for name, val in parent.__dict__.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val, parent.__dict__['set_' + name[4:]])
new_attr[name] = val
for name, val in future_class_attr.items():
if not name.startswith('__'):
if name.startswith('get_'):
new_attr[name[4:]] = property(val, future_class_attr['set_'+name[4:]])
new_attr[name] = val
return type.__new__(mcs, future_class_name, future_class_parents, new_attr)
I have been using the following (Jython 2.7) code to decorate functions in some classes:
import sys
import inspect
from decorator import decorator
def useless_decorator(method, *args, **kwargs):
#Does nothing yet :D
return method(*args, **kwargs)
class UselessMetaClass(type):
def __new__(cls, clsname, bases, dict):
for name, method in dict.items():
if not name.startswith('_') and inspect.isroutine(method):
dict[name] = decorator(useless_decorator, method)
return type.__new__(cls, clsname, bases, dict)
class Useless(object):
__metaclass__ = UselessMetaClass
The goal is to decorate all public functions (i.e. ones with names that don't start with an underscore) with the useless_decorator. Of course, this behaviour is only expected in classes that inherit from Useless.
Unfortunately I've been running into metaclass conflict errors. I've had great difficulty debugging them and I think they're occurring for reasons beyond my control (due to a third party library I'm using: Sikuli).
But, maybe I don't need to use a metaclass at all! Does anyone know a way to simulate my above code without using a metaclass?
I.E., Is there any other way to apply a decorator to all functions in a class?
(P.S. I know I could manually decorate each function, but that's not the solution I'm looking for)
Converting your metaclass to a class decorator should be straight forward. A class decorator simly receives the class as argument and returns the (modified) class:
def useless_class_decorator(cls):
for name, method in cls.__dict__.items():
if not name.startswith('_') and inspect.isroutine(method):
setattr(cls, name, decorator(useless_decorator, method))
return cls
The main difference here is that you can't direcly change cls.__dict__ here, as for new style classes that will be a dictproxy which does not support assignment, so you have to use setattr on the class instead. Then you simply create your class:
#useless_class_decorator
class Useless(object):
def method_to_decorate(self, *args, *kwargs):
...
However this won't affect subclasses of Useless, those would also have to be decorated using the class decorator. If that's not acceptable, then a metaclass may be the better option...
So there is a static method in base class, and sub classes should use it.
However, I need to know which sub class calls the static method.
The code is like this:
class BaseClass():
#staticmethod
def getname():
#some magic
class SubClassA(BaseClass):
pass
class SubClassB(BaseClass):
pass
SubClassA.getname() #hope to see 'SubClassA'
SubClassB.getname() #hope to see 'SubClassB'
Or, is this even possible?
not possible with staticmethod. is possible, however, with classmethod
class A(object):
#classmethod
def f(cls):
print cls
By definition, a staticmethod is not provided with a reference to either an invoking class or instance -- that's what "static" means in Python.
You can do some hacky things to achieve this with either metaclasses or decorators.
Note that I'm using the example idea of just getting the class's name from your post, however, you can modify this example to work with any kind of function you want to have as a static function. You would just define that function inside getnameable below, like I have defined getname and make use of the some_class argument below (and you'd use a different name than all the "getname" stuff I use here).
With decorators, you could do this:
def getnameable(some_class):
# Note, this could be any kind of static method that you want, not
# just for getting the name. And it can use `some_class` in whatever
# way is needed.
def getname():
return some_class.__name__
some_class.getname = staticmethod(getname)
return some_class
then this works:
In [334]: #getnameable
class SubClassA(BaseClass):
pass
.....:
In [335]: SubClassA.getname()
Out[335]: 'SubClassA'
but note that if you implemented this in BaseClass directly, then the class name which gets bound would be BaseClass, even in the children classes. So in this case, you'd need to put the decorator on every class you wanted.
Metaclasses offers away around that, by indicating that you want this business of decorating the class to be part of class-creation (not instance creation, mind you).
class GetNameableMeta(type):
def __new__(cls, name, bases, attrs):
temp_class = super(GetNameableMeta, cls).__new__(cls, name, bases, attrs)
return getnameable(temp_class)
class BaseClass(object):
__metaclass__ = GetNameableMeta
class SubClassA(BaseClass):
pass
Testing it out:
In [337]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:class GetNameableMeta(type):
: def __new__(cls, name, bases, attrs):
: temp_class = super(GetNameableMeta, cls).__new__(cls, name, bases, attrs)
: return getnameable(temp_class)
:
:class BaseClass(object):
: __metaclass__ = GetNameableMeta
:
:class SubClassA(BaseClass):
: pass
:--
In [338]: SubClassA.getname()
Out[338]: 'SubClassA'
In [339]: BaseClass.getname()
Out[339]: 'BaseClass'
Notice how much high-falutin code we needed to write to do this, when we had several reasonable alternatives:
Just ask directly for the __name__ attribute.
Make it a classmethod to begin with.
Use the decorators just where we need this and give up the inheritance part.
I suspect that in "the essence of Python" any of these is better than the machinery I describe above, since it makes for simpler, easier to understand code.
I feel like I have a pretty good grasp on using decorators when dealing with regular functions, but between using methods of base classes for decorators in derived classes, and passing parameters to said decorators, I cannot figure out what to do next.
Here is a snippet of code.
class ValidatedObject:
...
def apply_validation(self, field_name, code):
def wrap(self, f):
self._validations.append(Validation(field_name, code, f))
return f
return wrap
class test(ValidatedObject):
....
#apply_validation("_name", "oh no!")
def name_validation(self, name):
return name == "jacob"
If I try this as is, I get an "apply_validation" is not found.
If I try it with #self.apply_validation I get a "self" isn't found.
I've also been messing around with making apply_validation a class method without success.
Would someone please explain what I'm doing wrong, and the best way to fix this? Thank you.
The issue you're having is that apply_validation is a method, which means you need to call it on an instance of ValidatedObject. Unfortunately, at the time it is being called (during the definition of the test class), there is no appropriate instance available. You need a different approach.
The most obvious one is to use a metaclass that searches through its instance dictionaries (which are really class dictionaries) and sets up the _validations variable based on what it finds. You can still use a decorator, but it probably should be a global function, or perhaps a static method, and it will need to work differently. Here's some code, that uses a metaclass and a decorator that adds function attributes:
class ValidatedMeta(type):
def __new__(meta, name, bases, dct):
validations = [Validation(f._validation_field_name, f._validation_code, f)
for f in dct.values if hasattr(f._validation_field_name)]
dct["_validations"] = validations
super(ValidatedMeta, meta).__new__(meta, name, bases, dct)
def apply_validation(field_name, code):
def decorator(f):
f._validation_field_name = field_name
f._validation_code = code
return f
return decorator
def ValidatedObject(metaclass=ValidatedMeta):
pass
class test(ValidatedObject):
#apply_validation("_name", "oh no!")
def name_validation(self, name):
return name == "jacob"
After this code runs, test._validations will be [Validation("_name", "oh no!", test.name_validation)]. Note that the method that is be passed to Validation is unbound, so you'll need to pass it a self argument yourself when you call it (or perhaps drop the self argument and change the decorator created in apply_validation to return staticmethod(f)).
This code may not do what you want if you have validation methods defined at several levels of an inheritance hierarchy. The metaclass as written above only checks the immediate class's dict for methods with the appropriate attributes. If you need it include inherited methods in _validations too, you may need to modify the logic in ValidatedMeta.__new__. Probably the easiest way to go is to look for _validations attributes in the bases and concatenate the lists together.
Just an example for using decorators on class method:
from functools import wraps
def VALIDATE(dec):
#wraps(dec)
def _apply_validation(self, name):
self.validate(name)
return dec(self, name)
return _apply_validation
class A:
def validate(self, name):
if name != "aamir":
raise Exception, 'Invalid name "%s"' % name
class B(A):
#VALIDATE
def name_validation(self, name):
return name
b = B()
b.name_validation('jacob') # should raise exception
I'm defining several classes intended to be used for multiple inheritance, e.g.:
class A:
def __init__(self, bacon = None, **kwargs):
self.bacon = bacon
if bacon is None:
self.bacon = 100
super().__init__(**kwargs)
class Bacon(A):
def __init__(self, **kwargs):
"""Optional: bacon"""
super().__init__(**kwargs)
class Eggs(A):
def __init__(self, **kwargs):
"""Optional: bacon"""
super().__init__(**kwargs)
class Spam(Eggs, Bacon):
def __init__(self, **kwargs):
"""Optional: bacon"""
super().__init__(**kwargs)
However, I have multiple classes (e.g. possibly Bacon, A, and Spam, but not Eggs) that care about when their property bacon is changed. They don't need to modify the value, only to know what the new value is, like an event. Because of the Multiple Inheritance nature I have set up, this would mean having to notify the super class about the change (if it cares).
I know that it might be possible if I pass the class name to the method decorator, or if I use a class decorator. I don't want to have all the direct self-class referencing, having to create lots of decorators above each class, or forcing the methods to be the same name, as none of these sound very pythonic.
I was hoping to get syntax that looks something like this:
#on_change(bacon)
def on_bacon_change(self, bacon):
# read from old/new bacon
make_eggs(how_much = bacon)
I don't care about the previous value of bacon, so that bacon argument isn't necessary, if this is called after bacon is set.
Is it possible to check if a super class has a method with this
decorator?
If this isn't feasible, are there alternatives to passing events like
this, up through the multiple-inheritance chain?
EDIT:
The actual calling of the function in Spam would be done in A, by using a #property and #bacon.setter, as that would be the upper-most class that initializes bacon. Once it knows what function to call on self, the problem only lies in propagating the call up the MI chain.
EDIT 2:
If I override the attribute with a #bacon.setter, Would it be possible to determine whether the super() class has a setter for bacon?
What you call for would probably be nicely fit with a more complete framework of signals, and so on - maybe even invite for Aspected Oriented Programing.
Without going deep into it however, a metaclass and a decorator can do just what you are asking for - I came up with these, I hope they work for you.
If you'd like to evolve this in to something robust and usable, write me - if nothing like this exists out there, it wouldbe worth to keep an utility package in pipy for this.
def setattr_wrapper(cls):
def watcher_setattr(self, attr, val):
super(cls, self).__setattr__(attr, val)
watched = cls.__dict__["_watched_attrs"]
if attr in watched:
for method in watched[attr]:
getattr(self, method)(attr, val)
return watcher_setattr
class AttrNotifier(type):
def __new__(metacls, name, bases, dct):
dct["_watched_attrs"] = {}
for key, value in dct.items():
if hasattr(value, "_watched_attrs"):
for attr in getattr(value, "_watched_attrs"):
if not attr in dct["_watched_attrs"]:
dct["_watched_attrs"][attr] = set()
dct["_watched_attrs"][attr].add(key)
cls = type.__new__(metacls, name, bases, dct)
cls.__setattr__ = setattr_wrapper(cls)
return cls
def on_change(*args):
def decorator(meth):
our_args = args
#ensure that this decorator is stackable
if hasattr(meth, "_watched_attrs"):
our_args = getattr(meth, "_watched_attrs") + our_args
setattr(meth, "_watched_attrs", our_args)
return meth
return decorator
# from here on, example of use:
class A(metaclass=AttrNotifier):
#on_change("bacon")
def bacon_changed(self, attr, val):
print ("%s changed in %s to %s" % (attr, self.__class__.__name__, val))
class Spam(A):
#on_change("bacon", "pepper")
def changed(self, attr, val):
print ("%s changed in %s to %s" % (attr, self.__class__.__name__, val))
a = A()
a.bacon = 5
b = Spam()
b.pepper = 10
b.bacon = 20
(tested in Python 3.2 and Python 2.6 - changing the declaration of the "A" class for
Python 2 metaclass syntax)
edit - some words on what is being done
Here is what happens:
The metaclass picks all methods marked with the "on_close" decorator, and register then in a dictionary on the class - this dictionary is named _watched_attrs and it can be accessed as a normal class attribute.
The other thing the metaclass does is to override the __setattr__ method for the clas once it is created. This new __setattr__ just sets the attribute, and then checks the _wacthed_attrs dictionary if there are any methods on that class registered to be called when the attribute just changed has been modified - if so, it calls it.
The extra indirection level around watcher_setattr (which is the function that becomes each class's __setattr__ is there so that you can register different attributes to be watched on each class on the inheritance chain - all the classess have indepently acessible _watched_attrs dictionaries. If it was not for this, only the most specilized class on the inheritance chain _watched_attrs would be respected.
You are looking for python properties:
http://docs.python.org/library/functions.html#property
Search google for override superclass property setter resulted in this StackOverflow question:
Overriding inherited properties’ getters and setters in Python