This question already has answers here:
How can I decorate an instance method with a decorator class?
(2 answers)
Closed 4 years ago.
While there are plenty of resources about using classes as decorators, I haven't been able to find any that deal with the problem of decorating methods. The goal of this question is to fix that. I will post my own solution, but of course everyone else is invited to post theirs as well.
Why the "standard" implementation doesn't work
The problem with the standard decorator class implementation is that python will not create a bound method of the decorated function:
class Deco:
def __init__(self, func):
self.func= func
def __call__(self, *args):
self.func(*args)
class Class:
#Deco
def hello(self):
print('hello world')
Class().hello() # throws TypeError: hello() missing 1 required positional argument: 'self'
A method decorator needs to overcome this hurdle.
Requirements
Taking the classes from the previous example, the following things are expected to work:
>>> i= Class()
>>> i.hello()
hello world
>>> i.hello
<__main__.Deco object at 0x7f4ae8b518d0>
>>> Class.hello is Class().hello
False
>>> Class().hello is Class().hello
False
>>> i.hello is i.hello
True
Ideally, the function's __doc__ and signature and similar attributes are preserved as well.
Usually when a method is accessed as some_instance.some_method(), python's descriptor protocol kicks in and calls some_method.__get__(), which returns a bound method. However, because the method has been replaced with an instance of the Deco class, that does not happen - because Deco is not a descriptor. In order to make Deco work as expected, it must implement a __get__ method that returns a bound copy of itself.
Implementation
Here's basic "do nothing" decorator class:
import inspect
import functools
from copy import copy
class Deco(object):
def __init__(self, func):
self.__self__ = None # "__self__" is also used by bound methods
self.__wrapped__ = func
functools.update_wrapper(self, func)
def __call__(self, *args, **kwargs):
# if bound to an object, pass it as the first argument
if self.__self__ is not None:
args = (self.__self__,) + args
#== change the following line to make the decorator do something ==
return self.__wrapped__(*args, **kwargs)
def __get__(self, instance, owner):
if instance is None:
return self
# create a bound copy
bound = copy(self)
bound.__self__ = instance
# update __doc__ and similar attributes
functools.update_wrapper(bound, self.__wrapped__)
# add the bound instance to the object's dict so that
# __get__ won't be called a 2nd time
setattr(instance, self.__wrapped__.__name__, bound)
return bound
To make the decorator do something, add your code in the __call__ method.
Here's one that takes parameters:
class DecoWithArgs(object):
#== change the constructor's parameters to fit your needs ==
def __init__(self, *args):
self.args = args
self.__wrapped__ = None
self.__self__ = None
def __call__(self, *args, **kwargs):
if self.__wrapped__ is None:
return self.__wrap(*args, **kwargs)
else:
return self.__call_wrapped_function(*args, **kwargs)
def __wrap(self, func):
# update __doc__ and similar attributes
functools.update_wrapper(self, func)
return self
def __call_wrapped_function(self, *args, **kwargs):
# if bound to an object, pass it as the first argument
if self.__self__ is not None:
args = (self.__self__,) + args
#== change the following line to make the decorator do something ==
return self.__wrapped__(*args, **kwargs)
def __get__(self, instance, owner):
if instance is None:
return self
# create a bound copy of this object
bound = copy(self)
bound.__self__ = instance
bound.__wrap(self.__wrapped__)
# add the bound decorator to the object's dict so that
# __get__ won't be called a 2nd time
setattr(instance, self.__wrapped__.__name__, bound)
return bound
An implementation like this lets us use the decorator on methods as well as functions, so I think it should be considered good practice.
Related
I do have a wrapperclass for a specific object giving it some extra methods.
However this object (wrapped or not) is often passed as an argument. In this case I want to past the original (wrapped) object allways.
Is there a way (I guess magic method) to overwrite what is coming back if I do the following in print:
class Foo:
pass
C = Foo()
print(C)
I do now that this is actually calling __repr__ (which needs to be a str). If I do the same for a function call
def Bar(obj):
pass
does get Bar a string or the actual class here?
How I wrap the object:
class Wrapper:
def __init__(self, obj):
self._obj = obj
def __getattr__(self, attr):
orig_attr = self._obj.__getattribute__(attr)
if callable(orig_attr):
def hooked(*args, **kwargs):
result = orig_attr(*args, **kwargs)
# prevent wrapped_class from becoming unwrapped
if result == self._obj:
return self
return result
return hooked
else:
return orig_attr
So if do:
C_wrapped = Wrapper(C)
and than
Bar(C_wrapped)
if would actually have it to act as
Bar(C)
even if pass C_wrapped
The solution i found is that the function Bar (or object in my case) needs to check if it gets a wrapped or unwrapped object. If it is wrapped, Bar unwraps it.
The downside is this induces coupling.
I have two classes, one of which is a wrapper of the other. A function in the original class uses a method called forward, but I want it to use the forward method of the wrapper class after it has been wrapped, not the original. For example:
class A:
def __init__(self):
self.a = 1
def forward(self, x):
return self.a + x
def main(self, x):
return self.forward(x) + 100
class Wrapper:
def __init__(self, A):
self.module = A
def forward(self, x):
# Long convoluted code.
# ..
# ..
return self.module.forward(x)
classA = A()
wrapperA = Wrapper(classA)
# Goal: Make classA.main(..) use the forward function from Wrapper instead.
Because the wrapper class has the long and convoluted code that needs to be run, I want all calls of forward from main to be such that it calls the forward from the wrapper class, not from the original.
Is there a way to do this in Python?
Reasons why I did not use inheritance:
Instantiating class A is memory intensive. If I receive class A object as input, I want to modify its core behavior. without instantiating another object.
classA can be of different object types in runtime.
--
An alternative way I thought of is to redefine main in Wrapper. However, the problem is doing this automatically for every method defined in A without hard coding.
In Python "everything is an object". Including classes, functions, and methods
on objects.
As such we can take any class, loop over all functions in that class and modify
them as needed.
Depending on the real code, the problem in the question might be better tackled
using decorators or meta-classes, depending on the dependencies of the wrapper
(what values does it need access to). I will not go into meta-classes as most needs for meta-classes can also be implemented using class-decorators, which are less error-prone.
As you mention in one of your comments that you may have several different classes that need to be wrapped the class-decorator solution might be a good candidate. This way you won't lose the inheritance tree of the wrapped class.
Here is an example not using either, but doing exactly as asked ;)
Using __new__
from functools import update_wrapper
class A:
def __init__(self):
self.a = 1
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
def main(self, x):
return self.forward(x) + 100
class Wrapper:
# Using __new__ instead of __init__ gives us complete control *how* the
# "Wrapper" instance is created. We use it to "pull in" methods from *A*
# and dynamically attach them to the `Wrapper` instance using `setattr`.
#
# Using __new__ is error-prone however, and using either meta-classes or
# even easier, decorators would be more maintainable.
def __new__(cls, A):
# instance will be our instance of thie `Wrapper` class. We start off
# with no defined functions, we will add those soon...
instance = super().__new__(cls)
instance.module = A
# We now walk over every "name" in the wrapped class
for funcname in dir(A):
# We skip anything starting with two underscores. They are most
# likely magic methods that we don't want to wrap with the
# additional code. The conditions what exactly we want to wrap, can
# be adapted as needed.
if funcname.startswith("__"):
continue
# We now need to get a reference to that attribute and check if
# it's callable. If not it is a member variable or something else
# and we can/should skip it.
func = getattr(A, funcname)
if not callable(func):
continue
# Now we "wrap" the function with our additional code. This is done
# in a separate function to keep __new__ somewhat clean
wrapped = Wrapper._wrap(func)
# After wrapping the function we can attach that new function ont
# our `Wrapper` instance
setattr(instance, funcname, wrapped)
return instance
#staticmethod
def _wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
class Demo2:
def foo(self):
print("yoinks")
classA = A()
otherInstance = Demo2()
wrapperA = Wrapper(classA)
wrapperB = Wrapper(otherInstance)
print(wrapperA.forward(10))
print(wrapperB.foo())
print("docstring is maintained: %r" % wrapperA.forward.__doc__)
Using a class decorator
With a class decorator, there is no need to override __new__ which can lead to hard to debug issues if not 100% properly implemented.
However, it has a key difference: It modifies the existing class "in-place", so the original class is lost in a way. Although you could keep a reference to it in the unlikely case that you need to.
Modifying this in-place does however also mean that you don't need to replace all your usages in your application with the new "wrapper" class, making it a lot easier to implement in an existing code-base and eliminating the risk that you forget to apply the wrapper on new instances.
from functools import update_wrapper
def _wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
def wrapper(cls):
for funcname in dir(cls):
# We skip anything starting with two underscores. They are most
# likely magic methods that we don't want to wrap with the
# additional code. The conditions what exactly we want to wrap, can
# be adapted as needed.
if funcname.startswith("__"):
continue
# We now need to get a reference to that attribute and check if
# it's callable. If not it is a member variable or something else
# and we can/should skip it.
func = getattr(cls, funcname)
if not callable(func):
continue
# Now we "wrap" the function with our additional code. This is done
# in a separate function to keep __new__ somewhat clean
wrapped = _wrap(func)
# After wrapping the function we can attach that new function ont
# our `Wrapper` instance
setattr(cls, funcname, wrapped)
return cls
#wrapper
class A:
def __init__(self):
self.a = 1
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
def main(self, x):
return self.forward(x) + 100
#wrapper
class Demo2:
def foo(self):
print("yoinks")
classA = A()
otherInstance = Demo2()
print(classA.forward(10))
print(otherInstance.foo())
print("docstring is maintained: %r" % classA.forward.__doc__)
Using function decorators
Another alternative, which diverges largely from the original question but may still prove insightful is using individual functions wrappers.
The code still used the same wrapper function, but here functions/methods are annotated individually.
This might give more flexibility by offering the possibility to leave some methods "unwrapped", but could easily lead to the wrapping code being executed more often than anticipated as demonstrated in the main() method.
from functools import update_wrapper
def wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
class A:
def __init__(self):
self.a = 1
#wrap
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
#wrap # careful: will be wrapped twice!
def main(self, x):
return self.forward(x) + 100
def foo(self):
print("yoinks")
classA = A()
print(">>> forward")
print(classA.forward(10))
print("<<< forward")
print(">>> main")
print(classA.main(100))
print("<<< main")
print(">>> foo")
print(classA.foo())
print("<<< foo")
You could inherit Wrapper from A, and use super to access the parent class.
class A:
def __init__(self, child):
self.a = 1
self.child = child
def forward(self, x):
return self.a + x
def main(self, x):
return self.child.forward(x) + 100
class Wrapper(A):
def __init__(self):
super(Wrapper, self).__init__(self, self)
def forward(x):
return "whatever"
wrapperA = Wrapper()
But if you wish to use class A, just inherit A from Wrapper. Otherwise, I can't figure out whats wrong. Please don't use functions indiscriminate. Make a class you wish to use, and another one act as a parent and don't mix roles.
#...
class A(Wrapper):
def __init__(self):
super(A, self).__init__(self)
#...
I am attempting to copy the functionality of the built-in property class / decorator; a very basic example of what I want to is this:
# If a condition is met, run the first function, else, the second.
#godspeed()
def test():
print(1, 2, 3, 4)
#test.else_()
def test():
print(5, 6, 7, 8)
Here's what I have so far:
import inspect
class godspeed_class():
def __init__(
self,
func,
args,
kwargs,
value,
):
self.func = func
self.args = args
self.kwargs = kwargs
self.value = value
def __call__(self):
if self.value:
self.func(*self.args, **self.kwargs)
else:
self.else_func(*self.else_args, **self.else_kwargs)
def else_(self, *args, **kwargs):
def wrapper(func):
self.else_func = func
self.else_args = args
self.else_kwargs = kwargs
return wrapper
def godspeed(*args, value = 0, **kwargs):
def wrapper(func):
_ = godspeed_class(func, args, kwargs, value)
inspect.stack(1)[1][0].f_globals[func.__name__] = _
return wrapper
I already know how to implement the condition parsing, but I am having trouble with storing the function under the else_ decorator in the class, so that I can call it if the condition isn't met.
In addition, despite injecting the new class directly into the global namespace, when I run print(test), it tells me it's a NoneType object.
Note: Code has been updated; however, it still gives me the "NoneType object" error.
You need to change both of your wrapper functions to return a callable object, probably the instance of your class. Otherwise you're going to have None as the value for the method, since the decorator syntax will assign the return value to the name of the decorated function, which means that even if your inspect hack works, it will get overwritten.
I'd suggest:
class godspeed_class():
... # __init__ and __call__ can remain the same
def else_(self, *args, **kwargs):
def wrapper(func):
self.else_func = func
self.else_args = args
self.else_kwargs = kwargs
return self # add return here
return wrapper
def godspeed(*args, value = 0, **kwargs):
def wrapper(func):
return godspeed_class(func, args, kwargs, value) # and here (rather than inspect stuff)
return wrapper
This will do the job for your example with a top-level test function. If you want to be able to decorate methods, you'll also need to add a __get__ method to the class to add binding behavior (otherwise you'll not get the self argument passed in to the wrapped method).
It's a bit misleading to use wrapper as the name there, as the inner functions are the actual decorators being used here (the top level godspeed function and the else_ method are decorator factories). Normally you use wrapper as a name of a function returned by a decorator (but you're using your class for that instead).
I'd also note that it's a bit strange that you're passing the arguments for the functions to the decorator factories, rather than having __call__ accept arguments that it passes along to the relevant function. It's a bit unusual for a decorator that leaves behind a callable (rather than something like property that works differently) to dramatically change a function's calling convention, as it may end up hard for a caller to know what arguments they're expected to pass in, if the function signature isn't representative any more.
A decorator is nothing magical. Basically, the #decorator syntax is just syntactic sugar, so this:
#mydecorator
def func():
pass
is just a convenient shortcut for
def func():
pass
func = mydecorator(func)
IOW, a "decorator" is a callable object that takes a callable as input and returns a callable (well, it's supposed to return a callable at least - you can actually return whatever, but then you'll break everyone's expectations).
Most often, the decorator is written as a simple function returning a closure over the decorated function:
def trace(func):
def wrapper(*args, **kw):
result = func(*args, **kw)
print("{}({}, {}) => {}". format(func, args, kw, result))
return result
return wrapper
#trace
def foo(x):
return 42 * x
But (since closures are the poor man's classes and classes the poor man's closures) you can also implement it as a callable class, in which case the initializer will receive the decorated func, which in turn will be replaced by the instance:
class trace(object):
def __init__(self, func):
self.func = func
def __call__(self, *args, **kw):
result = self.func(*args, **kw)
print("{}({}, {}) => {}". format(self.func, args, kw, result))
return result
#trace
def foo(x):
return 42 * x
Then you have "parameterized" decorators - the one that can take arguments. In this case you need two level of indirection, the top-level one (the one used as decorator) returning the actual decorator (the one that receives the function), ie:
def trace(out):
def really_trace(func):
def wrapper(*args, **kw):
result = func(*args, **kw)
out.write("{}({}, {}) => {}\n". format(func, args, kw, result))
return result
return wrapper
return really_trace
#trace(sys.stderr)
def foo(x):
return 42 * x
I leave the class-based implementation as an exercise to the reader ;-)
Now in your case, the fact that test ends up being None is quite simply due to the fact that your wrapper func forgets to return the godspeed_class instance as it should (instead messing with the function's f_globals, which, as you noticed, doesn't work as expected).
Since you didn't clearly explained what you're trying to achieve here ("something similar to property" isn't a proper spec), it's hard to provide a working solution, but as a starting point you may want to fix your godspeed func to behave as expected:
def godspeed(*args, value = 0, **kwargs):
def wrapper(func):
return godspeed_class(func, args, kwargs, value)
return wrapper
I have a test framework that requires test cases to be defined using the following class patterns:
class TestBase:
def __init__(self, params):
self.name = str(self.__class__)
print('initializing test: {} with params: {}'.format(self.name, params))
class TestCase1(TestBase):
def run(self):
print('running test: ' + self.name)
When I create and run a test, I get the following:
>>> test1 = TestCase1('test 1 params')
initializing test: <class '__main__.TestCase1'> with params: test 1 params
>>> test1.run()
running test: <class '__main__.TestCase1'>
The test framework searches for and loads all TestCase classes it can find, instantiates each one, then calls the run method for each test.
load_test(TestCase1(test_params1))
load_test(TestCase2(test_params2))
...
load_test(TestCaseN(test_params3))
...
for test in loaded_tests:
test.run()
However, I now have some test cases for which I don't want the __init__ method called until the time that the run method is called, but I have little control over the framework structure or methods. How can I delay the call to __init__ without redefining the __init__ or run methods?
Update
The speculations that this originated as an XY problem are correct. A coworker asked me this question a while back when I was maintaining said test framework. I inquired further about what he was really trying to achieve and we figured out a simpler workaround that didn't involve changing the framework or introducing metaclasses, etc.
However, I still think this is a question worth investigating: if I wanted to create new objects with "lazy" initialization ("lazy" as in lazy evaluation generators such as range, etc.) what would be the best way of accomplishing it? My best attempt so far is listed below, I'm interested in knowing if there's anything simpler or less verbose.
First Solution:use property.the elegant way of setter/getter in python.
class Bars(object):
def __init__(self):
self._foo = None
#property
def foo(self):
if not self._foo:
print("lazy initialization")
self._foo = [1,2,3]
return self._foo
if __name__ == "__main__":
f = Bars()
print(f.foo)
print(f.foo)
Second Solution:the proxy solution,and always implement by decorator.
In short, Proxy is a wrapper that wraps the object you need. Proxy could provide additional functionality to the object that it wraps and doesn't change the object's code. It's a surrogate which provide the abitity of control access to a object.there is the code come form user Cyclone.
class LazyProperty:
def __init__(self, method):
self.method = method
self.method_name = method.__name__
def __get__(self, obj, cls):
if not obj:
return None
value = self.method(obj)
print('value {}'.format(value))
setattr(obj, self.method_name, value)
return value
class test:
def __init__(self):
self._resource = None
#LazyProperty
def resource(self):
print("lazy")
self._resource = tuple(range(5))
return self._resource
if __name__ == '__main__':
t = test()
print(t.resource)
print(t.resource)
print(t.resource)
To be used for true one-time calculated lazy properties. I like it because it avoids sticking extra attributes on objects, and once activated does not waste time checking for attribute presence
Metaclass option
You can intercept the call to __init__ using a metaclass. Create the object with __new__ and overwrite the __getattribute__ method to check if __init__ has been called or not and call it if it hasn't.
class DelayInit(type):
def __call__(cls, *args, **kwargs):
def init_before_get(obj, attr):
if not object.__getattribute__(obj, '_initialized'):
obj.__init__(*args, **kwargs)
obj._initialized = True
return object.__getattribute__(obj, attr)
cls.__getattribute__ = init_before_get
new_obj = cls.__new__(cls, *args, **kwargs)
new_obj._initialized = False
return new_obj
class TestDelayed(TestCase1, metaclass=DelayInit):
pass
In the example below, you'll see that the init print won't occur until the run method is executed.
>>> new_test = TestDelayed('delayed test params')
>>> new_test.run()
initializing test: <class '__main__.TestDelayed'> with params: delayed test params
running test: <class '__main__.TestDelayed'>
Decorator option
You could also use a decorator that has a similar pattern to the metaclass above:
def delayinit(cls):
def init_before_get(obj, attr):
if not object.__getattribute__(obj, '_initialized'):
obj.__init__(*obj._init_args, **obj._init_kwargs)
obj._initialized = True
return object.__getattribute__(obj, attr)
cls.__getattribute__ = init_before_get
def construct(*args, **kwargs):
obj = cls.__new__(cls, *args, **kwargs)
obj._init_args = args
obj._init_kwargs = kwargs
obj._initialized = False
return obj
return construct
#delayinit
class TestDelayed(TestCase1):
pass
This will behave identically to the example above.
In Python, there is no way that you can avoid calling __init__ when you instantiate a class cls. If calling cls(args) returns an instance of cls, then the language guarantees that cls.__init__ will have been called.
So the only way to achieve something similar to what you are asking is to introduce another class that will postpone the calling of __init__ in the original class until an attribute of the instantiated class is being accessed.
Here is one way:
def delay_init(cls):
class Delay(cls):
def __init__(self, *arg, **kwarg):
self._arg = arg
self._kwarg = kwarg
def __getattribute__(self, name):
self.__class__ = cls
arg = self._arg
kwarg = self._kwarg
del self._arg
del self._kwarg
self.__init__(*arg, **kwarg)
return getattr(self, name)
return Delay
This wrapper function works by catching any attempt to access an attribute of the instantiated class. When such an attempt is made, it changes the instance's __class__ to the original class, calls the original __init__ method with the arguments that were used when the instance was created, and then returns the proper attribute. This function can be used as decorator for your TestCase1 class:
class TestBase:
def __init__(self, params):
self.name = str(self.__class__)
print('initializing test: {} with params: {}'.format(self.name, params))
class TestCase1(TestBase):
def run(self):
print('running test: ' + self.name)
>>> t1 = TestCase1("No delay")
initializing test: <class '__main__.TestCase1'> with params: No delay
>>> t2 = delay_init(TestCase1)("Delayed init")
>>> t1.run()
running test: <class '__main__.TestCase1'>
>>> t2.run()
initializing test: <class '__main__.TestCase1'> with params: Delayed init
running test: <class '__main__.TestCase1'>
>>>
Be careful where you apply this function though. If you decorate TestBase with delay_init, it will not work, because it will turn the TestCase1 instances into TestBase instances.
In my answer I'd like to focus on cases when one wants to instantiate a class whose initialiser (dunder init) has side effects. For instance, pysftp.Connection, creates an SSH connection, which may be undesired until it's actually used.
In a great blog series about conceiving of wrapt package (nit-picky decorator implementaion), the author describes Transparent object proxy. This code can be customised for the subject in question.
class LazyObject:
_factory = None
'''Callable responsible for creation of target object'''
_object = None
'''Target object created lazily'''
def __init__(self, factory):
self._factory = factory
def __getattr__(self, name):
if not self._object:
self._object = self._factory()
return getattr(self._object, name)
Then it can be used as:
obj = LazyObject(lambda: dict(foo = 'bar'))
obj.keys() # dict_keys(['foo'])
But len(obj), obj['foo'] and other language constructs which invoke Python object protocols (dunder methods, like __len__ and __getitem__) will not work. However, for many cases, which are limited to regular methods, this is a solution.
To proxy object protocol implementations, it's possible to use neither __getattr__, nor __getattribute__ (to do it in a generic way). The latter's documentation notes:
This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See Special method lookup.
As a complete solution is demanded, there are examples of manual implementations like werkzeug's LocalProxy and django's SimpleLazyObject. However a clever workaround is possible.
Luckily there's a dedicated package (based on wrapt) for the exact use case, lazy-object-proxy which is described in this blog post.
from lazy_object_proxy import Proxy
obj = Proxy(labmda: dict(foo = 'bar'))
obj.keys() # dict_keys(['foo'])
len(len(obj)) # 1
obj['foo'] # 'bar'
One alternative would be to write a wrapper that takes a class as input and returns a class with delayed initialization until any member is accessed. This could for example be done as this:
def lazy_init(cls):
class LazyInit(cls):
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
self._initialized = False
def __getattr__(self, attr):
if not self.__dict__['_initialized']:
cls.__init__(self,
*self.__dict__['args'], **self.__dict__['kwargs'])
self._initialized = True
return self.__dict__[attr]
return LazyInit
This could then be used as such
load_test(lazy_init(TestCase1)(test_params1))
load_test(lazy_init(TestCase2)(test_params2))
...
load_test(lazy_init(TestCaseN)(test_params3))
...
for test in loaded_tests:
test.run()
Answering your original question (and the problem I think you are actually trying to solve), "How can I delay the init call until an attribute is accessed?": don't call init until you access the attribute.
Said another way: you can make the class initialization simultaneous with the attribute call. What you seem to actually want is 1) create a collection of TestCase# classes along with their associated parameters; 2) run each test case.
Probably your original problem came from thinking you had to initialize all your TestCase classes in order to create a list of them that you could iterate over. But in fact you can store class objects in lists, dicts etc. That means you can do whatever method you have for finding all TestCase classes and store those class objects in a dict with their relevant parameters. Then just iterate that dict and call each class with its run() method.
It might look like:
tests = {TestCase1: 'test 1 params', TestCase2: 'test 2 params', TestCase3: 'test 3 params'}
for test_case, param in tests.items():
test_case(param).run()
Overridding __new__
You could do this by overriding __new__ method and replacing __init__ method with a custom function.
def init(cls, real_init):
def wrapped(self, *args, **kwargs):
# This will run during the first call to `__init__`
# made after `__new__`. Here we re-assign the original
# __init__ back to class and assign a custom function
# to `instances.__init__`.
cls.__init__ = real_init
def new_init():
if new_init.called is False:
real_init(self, *args, **kwargs)
new_init.called = True
new_init.called = False
self.__init__ = new_init
return wrapped
class DelayInitMixin(object):
def __new__(cls, *args, **kwargs):
cls.__init__ = init(cls, cls.__init__)
return object.__new__(cls)
class A(DelayInitMixin):
def __init__(self, a, b):
print('inside __init__')
self.a = sum(a)
self.b = sum(b)
def __getattribute__(self, attr):
init = object.__getattribute__(self, '__init__')
if not init.called:
init()
return object.__getattribute__(self, attr)
def run(self):
pass
def fun(self):
pass
Demo:
>>> a = A(range(1000), range(10000))
>>> a.run()
inside __init__
>>> a.a, a.b
(499500, 49995000)
>>> a.run(), a.__init__()
(None, None)
>>> b = A(range(100), range(10000))
>>> b.a, b.b
inside __init__
(4950, 49995000)
>>> b.run(), b.__init__()
(None, None)
Using cached properties
The idea is to do the heavy calculation only once by caching results. This approach will lead to much more readable code if the whole point of delaying initialization is improving performance.
Django comes with a nice decorator called #cached_property. I tend to use it a lot in both code and unit-tests for caching results of heavy properties.
A cached_property is a non-data descriptor. Hence once the key is set in instance's dictionary, the access to property would always get the value from there.
class cached_property(object):
"""
Decorator that converts a method with a single self argument into a
property cached on the instance.
Optional ``name`` argument allows you to make cached properties of other
methods. (e.g. url = cached_property(get_absolute_url, name='url') )
"""
def __init__(self, func, name=None):
self.func = func
self.__doc__ = getattr(func, '__doc__')
self.name = name or func.__name__
def __get__(self, instance, cls=None):
if instance is None:
return self
res = instance.__dict__[self.name] = self.func(instance)
return res
Usage:
class A:
#cached_property
def a(self):
print('calculating a')
return sum(range(1000))
#cached_property
def b(self):
print('calculating b')
return sum(range(10000))
Demo:
>>> a = A()
>>> a.a
calculating a
499500
>>> a.b
calculating b
49995000
>>> a.a, a.b
(499500, 49995000)
I think you can use a wrapper class to hold the real class you want to instance, and use call __init__ yourself in your code, like(Python 3 code):
class Wrapper:
def __init__(self, cls):
self.cls = cls
self.instance = None
def your_method(self, *args, **kwargs):
if not self.instance:
self.instnace = cls()
return self.instance(*args, **kwargs)
class YourClass:
def __init__(self):
print("calling __init__")
but it's a dump way, but without any trick.
I have a decorated function (simplified version):
class Memoize:
def __init__(self, function):
self.function = function
self.memoized = {}
def __call__(self, *args, **kwds):
hash = args
try:
return self.memoized[hash]
except KeyError:
self.memoized[hash] = self.function(*args)
return self.memoized[hash]
#Memoize
def _DrawPlot(self, options):
do something...
now I want to add this method to a pre-esisting class.
ROOT.TChain.DrawPlot = _DrawPlot
when I call this method:
chain = TChain()
chain.DrawPlot(opts)
I got:
self.memoized[hash] = self.function(*args)
TypeError: _DrawPlot() takes exactly 2 arguments (1 given)
why doesn't it propagate self?
The problem is that you have defined your own callable class then tried to use it as a method. When you use a function as an attribute, accessing the function as an attribute calls it its __get__ method to return something other than the function itself—the bound method. When you have your own class without defining __get__, it just returns your instance without implicitly passing self.
Descriptors are explained some on http://docs.python.org/reference/datamodel.html#descriptors if you are not familiar with them. The __get__, __set__, and __delete__ methods change how interacting with your object as an attribute works.
You could implement memoize as a function and use the built-in __get__ magic that functions already have
import functools
def memoize(f):
#functools.wraps(f)
def memoized(*args, _cache={}):
# This abuses the normally-unwanted behaviour of mutable default arguments.
if args not in _cache:
_cache[args] = f(*args)
return _cache[args]
return memoized
or by modifying your class along the lines of
import functools
class Memoize(object): #inherit object
def __init__(self, function):
self.function = function
self.memoized = {}
def __call__(self, *args): #don't accept kwargs you don't want.
# I removed "hash = args" because it shadowed a builtin function and
# because it was untrue--it wasn't a hash, it was something you intended for
# Python to hash for you.
try:
return self.memoized[args]
except KeyError:
self.memoized[args] = self.function(*args)
return self.memoized[args]
def __get__(self, obj, type):
if obj is None: #We looked up on the class
return self
return functools.partial(self, obj)
Note that both of these choke if any of the arguments you pass in are mutable (well, unhashable technically). This might be suitable for your case, but you may also want to deal with the case where args is unhashable.