I have a decorator method to check the argument types passed to a function.
def accepts(*types):
def check_accepts(f):
assert len(types) == f.func_code.co_argcount
def new_f(*args, **kwds):
for (a, t) in zip(args, types):
assert isinstance(a, t), \
"arg %r does not match %s" % (a,t)
return f(*args, **kwds)
new_f.func_name = f.func_name
return new_f
return check_accepts
Now, in a class (in classA.py), I want a method to only accept arguments of the same class:
class ClassA:
#accepts(WHATTOPUTHERE)
def doSomething(otherObject):
# Do something
In other classes I can just put ClassA in place of WHATTOPUTHERE, but inside classA.py, ClassA is not known. How can I pass the current class to the #accepts() function?
Use the function based version of the decorator and apply it after the class definition:
class ClassA:
def doSomething(otherObject):
# Do something
ClassA.doSomething = accepts(ClassA)(ClassA.doSomething)
Another way would be write a Metaclass that would automatically apply this after class creation:
class Meta(type):
def __new__(cls, clsname, bases, dct):
fields = ('doSomething', ) #Fields on which you want to apply the decorator
for name, val in dct.items():
if name in fields:
dct[name] = accepts(cls)(val)
return type.__new__(cls, clsname, bases, dct)
class ClassA(object):
__metaclass__ = Meta
def doSomething(otherObject):
pass
Instead of manually doing things like new_f.func_name = f.func_name, use functools.wraps. This would also preserve things like docstring, argument list etc.
from functools import wraps
def accepts(*types):
def check_accepts(f):
print "inside"
assert len(types) == f.func_code.co_argcount
#wraps(f)
def new_f(*args, **kwds):
wouldn't adding a self variable to the function doSomething and then referencing args[0] inside check_accepts (provided you add (*args) or extra argument to your definition) solve your problem? If the function doSomething is supposed to be a class method, still you can outsource this self. How?
add a self to some dummy method inside the class
make a decorator which populates a variable which accepts can somehow reach (like metadata)
make sure to call this additional method before the doSomething()
you've got the class instance! Enjoy!
NOTE: This is not the only way to store metadata like this and use it later, you can do it as you wish.
Related
I have a ReportEntry class
class ReportEntry(object):
def __init__(self):
# Many attributes defined here
... # Lot many setattr/getattr here
def validate(self):
# Lot of validation code in here
return self
Multiple other classes maintain has-a relation with ReportEntry class
class A(object):
def test1(self):
t1 = ReportEntry()
# Assign the attribute values to t1
return t1.validate()
def test2(self):
t2 = ReportEntry()
# Assign the attribute values to t2
return t2.validate()
And there are multiple such classes as A.
I need to enforce each ReportEntry class instance to call validate() on return or maybe just before return.
Basically, no instance of ReportEntry should escape validation since the final report generation will fail if something is missing.
How may I achieve that ?
You can write a class decorator:
import inspect
def validate_entries(cls):
def validator(fnc): # this is a function decorator ...
def wrapper(*args, **kwargs):
rval = fnc(*args, **kwargs)
if isinstance(rval, ReportEntry):
# print('validating')
return rval.validate()
return rval
return wrapper
for name, f in inspect.getmembers(cls, predicate=inspect.isfunction):
setattr(cls, name, validator(f)) # .. that we apply to all functions
return cls
Now you can define all A-like classes:
#validate_entries
class A(object):
# ...
This will validate any ReportEntry that is returned by any of A's methods.
There are two ways I can think about to go about this. I cannot say more without knowing more implementation details:
Decorate your methods: Where every return instance is run through the decorator function. You may want to put this as a stand-alone function or part of a class depending on your specific use case.
def validate(func):
return func().validate()
class A(object):
#validate
def test1(self):
t1 = ReportEntry()
# Assign the attribute values to t1
return t1
#validate
def test2(self):
t2 = ReportEntry()
# Assign the attribute values to t2
return t2
Updating the __setattr__ and decorate your class:
def always_validate(cls):
# save the old set attribute method
old_setattr = getattr(cls, '__setattr__', None)
def __setattr__(self, name, value):
# set the attribute
validate(name, value)
old_setattr(self, name, value)
cls.__setattr__ = __setattr__
return cls
and then you could decorate your ReportEntry:
#alway_validate
class ReportEntry(object):
...
One approach that I could think of is to define __enter__ and __exit__ methods where validate is called upon __exit__ in ReportEntry
class ReportEntry(object):
def __enter__(self):
return self
def __init__(self):
# Many attributes defined here
... # Lot many setattr/getattr here
def validate(self):
# Lot of validation code in here
return self
def __exit__(self, a,b,c):
self.validate()
return True
# And then use it as
with ReportEntry() as report:
...
But again, this will be enforced only when used with ReportEntry() as report:
I am attempting to wrap a class from a third-party package in such a way that my new class looks exactly like a subclass of the third-party class. The third-party class does not support inheritance, and it has nontrivial features, such as functions that have a __getitem__ method. I can wrap almost every attribute and method using a solution based on Wrapping a class whose methods return instances of that class and How can I intercept calls to python's "magic" methods in new style classes?. However, I still need to override the __init__ method of the third-party class. How can I do that? Note: I am using new-style classes.
Code so far:
import copy
class WrapperMetaclass(type):
"""
Works with the `Wrapper` class to create proxies for the wrapped object's magic methods.
"""
def __init__(cls, name, bases, dct):
def make_proxy(name):
def proxy(self, *args):
return getattr(self._obj, name)
return proxy
type.__init__(cls, name, bases, dct)
if cls.__wraps__:
ignore = set("__%s__" % n for n in cls.__ignore__.split())
for name in dir(cls.__wraps__):
if name.startswith("__"):
if name not in ignore and name not in dct:
setattr(cls, name, property(make_proxy(name)))
class Wrapper(object):
"""
Used to provide a (nearly) seamless inheritance-like interface for classes that do not support direct inheritance.
"""
__metaclass__ = WrapperMetaclass
__wraps__ = None
# note that the __init__ method will be ignored by WrapperMetaclass
__ignore__ = "class mro new init setattr getattr getattribute dict"
def __init__(self, obj):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
elif isinstance(obj, self.__wraps__):
self._obj = obj
else:
raise ValueError("wrapped object must be of %s" % self.__wraps__)
def __getattr__(self, name):
if name is '_obj':
zot = 1
orig_attr = self._obj.__getattribute__(name)
if callable(orig_attr) and not hasattr(orig_attr, '__getitem__'):
def hooked(*args, **kwargs):
result = orig_attr(*args, **kwargs)
if result is self._obj:
return self
elif isinstance(result, self.__wraps__):
return self.__class__(result)
else:
return result
return hooked
else:
return orig_attr
def __setattr__(self, attr, val):
object.__setattr__(self, attr, val)
if getattr(self._obj, attr, self._obj) is not self._obj: # update _obj's member if it exists
setattr(self._obj, attr, getattr(self, attr))
class ClassToWrap(object):
def __init__(self, data):
self.data = data
def theirfun(self):
new_obj = copy.deepcopy(self)
new_obj.data += 1
return new_obj
def __str__(self):
return str(self.data)
class Wrapped(Wrapper):
__wraps__ = ClassToWrap
def myfun(self):
new_obj = copy.deepcopy(self)
new_obj.data += 1
return new_obj
# can't instantiate Wrapped directly! This is the problem!
obj = ClassToWrap(0)
wr0 = Wrapped(obj)
print wr0
>> 0
print wr0.theirfun()
>> 1
This works, but for truly seamless inheritance-like behavior, I need to instantiate Wrapped directly, e.g.
wr0 = Wrapped(0)
which currently throws
ValueError: wrapped object must be of <class '__main__.ClassToWrap'>
I attempted to override by defining a new proxy for __init__ in WrapperMetaclass, but rapidly ran into infinite recursions.
My codebase is complex with users at different skill levels, so I can't afford to use monkey-patching or solutions that modify the definition of the example classes ClassToWrap or Wrapped. I am really hoping for an extension to the code above that overrides Wrapped.__init__.
Please note that this question is not simply a duplicate of e.g. Can I exactly mimic inheritance behavior with delegation by composition in Python?. That post does not have any answer that is nearly as detailed as what I'm already providing here.
It sounds like you just want Wrapper.__init__ method to work differently that it currently does. Rather than taking an already existing instance of the __wraps__ class, it should take the arguments that the other class expects in its constructor and built the instance for you. Try something like this:
def __init__(self, *args, **kwargs):
if self.__wraps__ is None:
raise TypeError("base class Wrapper may not be instantiated")
else:
self._obj = self.__wraps__(*args, **kwargs)
If you want Wrapper to remain the same for some reason, you could put the logic in a new Wrapped.__init__ method instead:
def __init__(self, data): # I'm explicitly naming the argument here, but you could use *args
super(self, Wrapped).__init__(self.__wraps__(data)) # and **kwargs to make it extensible
Suppose I have some python code, e.g. some class defined somewhere, which cannot be modified
class MyClass(object):
def __init__(self, arg1, arg2):
do_something...
def foo(self):
do_something
Now I want to add a trace capability, e.g. some mechanism from outside that traces each and every method call for the above class. I want to be able to print out when e.g, __init__ has been called, or foo or even the __del__ method of MyClass.
Is this possible to do, and how is this done best?
Create a proxy class that wraps the original class and then delegates the work after printing a trace:
class MyClassProxy(object):
def __init__(*args, **kwds):
print 'Initializing'
self.o = MyClass(*args, **kwds)
def foo(self):
print 'Fooing'
return self.o.foo()
You can create a trace decorator and attach it to all the methods of the class instance or class definition as shown in decorate_methods function.
import functools
import inspect
import types
class TestClass(object):
def func1(self):
pass
def func2(self, a, b):
pass
def trace(func):
#functools.wraps(func)
def decorator(*args, **kwargs):
print "TRACE:", func.__name__, args, kwargs
return func(*args, **kwargs)
return decorator
def decorate_methods(obj, decorator):
for name, func in inspect.getmembers(obj):
if isinstance(func, types.MethodType):
setattr(obj, name, decorator(func))
# Apply the decorator to a class instance
test1 = TestClass()
decorate_methods(test1, trace)
test1.func1()
test1.func2('bar1', b='bar2')
# Apply the decorator to the class definition
decorate_methods(TestClass, trace)
test2 = TestClass()
test2.func1()
test2.func2('bar1', b='bar2')
The output of the script will be:
TRACE: func1 () {}
TRACE: func2 ('bar1',) {'b': 'bar2'}
TRACE: func1 (<__main__.TestClass object at 0x7f5a8d888150>,) {}
TRACE: func2 (<__main__.TestClass object at 0x7f5a8d888150>, 'bar1') {'b': 'bar2'}
Use decorator as shown below:
def call_trace(orig_func):
def decorated_func(*args, **kwargs):
print "========>In function: " + orig_func.__name__ + "<========"
orig_func(*args, **kwargs)
return decorated_func
Apply this decorator to trace the function. It prints function name before entering the function.
Ex:
#call_trace
def foo(self):
do_something
Hope it helps.
[Update]: You can use metaclass, only thing you got to change is to add "metaclass" parameter to your class as shown below. As you can see, below code applies "call_trace" decorator to every function in the class "ExBase".
I tried this out yesterday, it worked fine. I am also new to python.:)
def call_trace(orig_func):
def inner_func(*args, **kwargs):
print ("function name:" + str(orig_func.__name__))
orig_func(*args, **kwargs)
return inner_func
class ExMeta(type):
def __new__(cls, name, bases, attrs):
for attr in attrs:
if hasattr(attrs[attr], '__call__'):
attrs[attr] = call_trace(attrs[attr])
return type.__new__(cls, name, bases, attrs)
class ExBase(metaclass=ExMeta):
x = "x"
y = "y"
def __init__(self):
self.__name = "name"
def getname(self):
return self.__name
b = ExBase()
b.getname()
Get the code for OnlinePythonTutor from github.com/pgbovine/OnlinePythonTutor/tree/master/v3.
You don't need to bother with all the JS stuff. Extract the files into some directory. You can run your scripts using python /path/to/my/OnlinePythonTutor-master/v3/generate_json_trace my_script.py
This will give you basically everything your program is doing in a step by step manner. It will probably be overkill so if you want look into the source code and the underlying source in bdb http://docs.python.org/2/library/bdb.html. The docs for bdb are horrible so I'm having trouble figuring out what exactly is going on but I think this is a pretty cool problem, good luck.
object of type A and Is there a way to programatically wrap a class object?
Given
class A(object):
def __init__(self):
## ..
def f0(self, a):
## ...
def f1(self, a, b):
## ..
I want another class that wraps an A, such as
class B(object):
def __init__(self):
self.a = A()
def f0(self,a):
try:
a.f0(a)
except (Exception),ex:
## ...
def f1(self, a, b):
try:
a.f1(a,b)
except (Exception),ex:
## ...
Is there a way to do create B.f0 & B.f1 by reflection/inspection of class A?
If you want to create class B by calling a function on a predefined class A, you can simply do B = wrap_class(A) with a function wrap_class that looks like this:
import copy
def wrap_class(cls):
'Wraps a class so that exceptions in its methods are caught.'
# The copy is necessary so that mutable class attributes are not
# shared between the old class cls and the new class:
new_cls = copy.deepcopy(cls)
# vars() is used instead of dir() so that the attributes of base classes
# are not modified, but one might want to use dir() instead:
for (attr_name, value) in vars(cls).items():
if isinstance(value, types.FunctionType):
setattr(new_cls, attr_name, func_wrapper(value))
return new_cls
B = wrap_class(A)
As Jürgen pointed out, this creates a copy of the class; this is only needed, however, if you really want to keep your original class A around (like suggested in the original question). If you don't care about A, you can simply decorate it with a wrapper that does not perform any copy, like so:
def wrap_class(cls):
'Wraps a class so that exceptions in its methods are caught.'
# vars() is used instead of dir() so that the attributes of base classes
# are not modified, but one might want to use dir() instead:
for (attr_name, value) in vars(cls).items():
if isinstance(value, types.FunctionType):
setattr(cls, attr_name, func_wrapper(value))
return cls
#wrap_class
class A(object):
… # Original A class, with methods that are not wrapped with exception catching
The decorated class A catches exceptions.
The metaclass version is heavier, but its principle is similar:
import types
def func_wrapper(f):
'Returns a version of function f that prints an error message if an exception is raised.'
def wrapped_f(*args, **kwargs):
try:
return f(*args, **kwargs)
except Exception, ex:
print "Function", f, "raised", ex
return wrapped_f
class ExceptionCatcher(type):
'Metaclass that wraps methods with func_wrapper().'
def __new__(meta, cname, bases, cdict):
# cdict contains the attributes of class cname:
for (attr_name, value) in cdict.items():
if isinstance(value, types.FunctionType): # Various attribute types can be wrapped differently
cdict[attr_name] = func_wrapper(value)
return super(meta, ExceptionCatcher).__new__(meta, cname, bases, cdict)
class B(object):
__metaclass__ = ExceptionCatcher # ExceptionCatcher will be used for creating class A
class_attr = 42 # Will not be wrapped
def __init__(self):
pass
def f0(self, a):
return a*10
def f1(self, a, b):
1/0 # Raises a division by zero exception!
# Test:
b = B()
print b.f0(3.14)
print b.class_attr
print b.f1(2, 3)
This prints:
31.4
42
Function <function f1 at 0x107812d70> raised integer division or modulo by zero
None
What you want to do is in fact typically done by a metaclass, which is a class whose instances are a class: this is a way of building the B class dynamically based on its parsed Python code (the code for class A, in the question). More information on this can be found in the nice, short description of metaclasses given in Chris's Wiki (in part 1 and parts 2-4).
Meta classes are an option, but generally hard to understand. As is too much reflection
if not needed in simple cases, because it is easy to catch too many (internal) functions. If the wrapped functions are a stable known set, and B might gain other functions, you can delegate explicitly function by function and still keep your error handling code in one place:
class B(object):
def __init__(self):
a = A()
self.f0 = errorHandler(a.f0)
self.f1 = errorHandler(a.f1)
You might do the assignments in a loop if they are many, using getattr/setattr.
The errorhandler function will need to return a function which wraps its argument with
error handling code.
def errorHandler(f):
def wrapped(*args, **kw):
try:
return f(*args, **kw)
except:
# log or something
return wrapped
You can also use errorhandler as decorator on new functions not delegating to the A instance.
def B(A):
...
#errorHandler
def f_new(self):
...
This solution keeps B simple and it is quite explicit what's going on.
You could try it old-school with __getattr__:
class B(object):
def __init__(self):
self.a = A()
def __getattr__(self, name):
a_method = getattr(a, name, None)
if not callable(a_method):
raise AttributeError("Unknown attribute %r" % name)
def wrapper(*args, **kwargs):
try:
return a_method(*args, **kwargs)
except Exception, ex:
# ...
return wrapper
Or with updating B's dict:
class B(object):
def __init__(self):
a = A()
for attr_name in dir(a):
attr = getattr(a, attr_name)
if callable(attr):
def wrapper(*args, **kwargs):
try:
return attr(*args, **kwargs)
except Exception, ex:
# ...
setattr(self, attr_name, wrapper) # or try self.__dict__[x] = y
I want to be able to ask a class's __init__ method what it's parameters are. The straightforward approach is the following:
cls.__init__.__func__.__code__.co_varnames[:code.co_argcount]
However, that won't work if the class has any decorators. It will give the parameter list for the function returned by the decorator. I want to get down to the original __init__ method and get those original parameters. In the case of a decorator, the decorator function is going to be found in the closure of the function returned by the decorator:
cls.__init__.__func__.__closure__[0]
However, it is more complicated if there are other things in the closure, which decorators may do from time to time:
def Something(test):
def decorator(func):
def newfunc(self):
stuff = test
return func(self)
return newfunc
return decorator
def test():
class Test(object):
#Something(4)
def something(self):
print Test
return Test
test().something.__func__.__closure__
(<cell at 0xb7ce7584: int object at 0x81b208c>, <cell at 0xb7ce7614: function object at 0xb7ce6994>)
And then I have to decide if I want to the parameters from decorator or the parameters from the original function. The function returned by the decorator could have *args and **kwargs for its parameters. What if there are multiple decorators and I have to decide which is the one I care about?
So what is the best way to find a function's parameters even when the function may be decorated? Also, what is the best way to go down a chain of decorators back to the decorated function?
Update:
Here is effectively how I am doing this right now (names have been changed to protect the identity of the accused):
import abc
import collections
IGNORED_PARAMS = ("self",)
DEFAULT_PARAM_MAPPING = {}
DEFAULT_DEFAULT_PARAMS = {}
class DICT_MAPPING_Placeholder(object):
def __get__(self, obj, type):
DICT_MAPPING = {}
for key in type.PARAMS:
DICT_MAPPING[key] = None
for cls in type.mro():
if "__init__" in cls.__dict__:
cls.DICT_MAPPING = DICT_MAPPING
break
return DICT_MAPPING
class PARAM_MAPPING_Placeholder(object):
def __get__(self, obj, type):
for cls in type.mro():
if "__init__" in cls.__dict__:
cls.PARAM_MAPPING = DEFAULT_PARAM_MAPPING
break
return DEFAULT_PARAM_MAPPING
class DEFAULT_PARAMS_Placeholder(object):
def __get__(self, obj, type):
for cls in type.mro():
if "__init__" in cls.__dict__:
cls.DEFAULT_PARAMS = DEFAULT_DEFAULT_PARAMS
break
return DEFAULT_DEFAULT_PARAMS
class PARAMS_Placeholder(object):
def __get__(self, obj, type):
func = type.__init__.__func__
# unwrap decorators here
code = func.__code__
keys = list(code.co_varnames[:code.co_argcount])
for name in IGNORED_PARAMS:
try: keys.remove(name)
except ValueError: pass
for cls in type.mro():
if "__init__" in cls.__dict__:
cls.PARAMS = tuple(keys)
break
return tuple(keys)
class BaseMeta(abc.ABCMeta):
def __init__(self, name, bases, dict):
super(BaseMeta, self).__init__(name, bases, dict)
if "__init__" not in dict:
return
if "PARAMS" not in dict:
self.PARAMS = PARAMS_Placeholder()
if "DEFAULT_PARAMS" not in dict:
self.DEFAULT_PARAMS = DEFAULT_PARAMS_Placeholder()
if "PARAM_MAPPING" not in dict:
self.PARAM_MAPPING = PARAM_MAPPING_Placeholder()
if "DICT_MAPPING" not in dict:
self.DICT_MAPPING = DICT_MAPPING_Placeholder()
class Base(collections.Mapping):
__metaclass__ = BaseMeta
"""
Dict-like class that uses its __init__ params for default keys.
Override PARAMS, DEFAULT_PARAMS, PARAM_MAPPING, and DICT_MAPPING
in the subclass definition to give non-default behavior.
"""
def __init__(self):
pass
def __nonzero__(self):
"""Handle bool casting instead of __len__."""
return True
def __getitem__(self, key):
action = self.DICT_MAPPING[key]
if action is None:
return getattr(self, key)
try:
return action(self)
except AttributeError:
return getattr(self, action)
def __iter__(self):
return iter(self.DICT_MAPPING)
def __len__(self):
return len(self.DICT_MAPPING)
print Base.PARAMS
# ()
print dict(Base())
# {}
At this point Base reports uninteresting values for the four contants and the dict version of instances is empty. However, if you subclass you can override any of the four, or you can include other parameters to the __init__:
class Sub1(Base):
def __init__(self, one, two):
super(Sub1, self).__init__()
self.one = one
self.two = two
Sub1.PARAMS
# ("one", "two")
dict(Sub1(1,2))
# {"one": 1, "two": 2}
class Sub2(Base):
PARAMS = ("first", "second")
def __init__(self, one, two):
super(Sub2, self).__init__()
self.first = one
self.second = two
Sub2.PARAMS
# ("first", "second")
dict(Sub2(1,2))
# {"first": 1, "second": 2}
Consider this decorator:
def rickroll(old_function):
return lambda junk, junk1, junk2: "Never Going To Give You Up"
class Foo(object):
#rickroll
def bar(self, p1, p2):
return p1 * p2
print Foo().bar(1, 2)
In it, the rickroll decorator takes the bar method, discards it, replaces it with a new function that ignores its differently-named (and possibly numbered!) parameters and instead returns a line from a classic song.
There are no further references to the original function, and the garbage collector can come and remove it any time it likes.
In such a case, I cannot see how you could find the parameter names p1 and p2. In my understanding, even the Python interpreter itself has no idea what they used to be called.