Develop Reference

In Python, how do I change/insert code of a method by other objects/functions/methods

I am not new to python but I am far from being an expert (or intermediate). Right now, I play around with objects and their behavior (like setattr, monkey-patch, etc.). During this, I stumbled upon a problem where I do not have any idea on how this might work.
Imagine following code:
class MyCalculator():
def __init__(self):
pass
def addition(self, a, b):
return a + b
def substraction(self, a, b):
return a - b
import inspect
class Changing():
def __init__(self):
pass
def listUserMethods(self, myObject):
object_names = [object_name for object_name in inspect.getmembers(myObject) if (inspect.ismethod(object_name[1]))]
return object_names
def setMethodAttribute(self, myMethod):
pass
if __name__=="__main__":
myCalc = MyCalculator()
change = Changing()
Now, I would like that setMethodAttribute() will change the code of the method I provide itself. Like, inserting a print() statement before the rest of the original method is executed. E.g. printing the input parameter before executing the addition, etc.
In my case, this does not need to be done during runtime (even if this is very interesting to know). I could imagine, that using inheritance or something similar could be a way. Perhaps somebody has a great idea?
Thanks for the help!

The answer really depends what you are after.
Wrapping a method of a class (before runtime)
This is very typical use case of decorators (the #something above a function definition).
def with_printing(func):
def wrapper(*args, **kwargs):
print("Before calling method")
ret = func(*args, **kwargs)
print("After calling method")
return ret
return wrapper
class MyCalculator:
#with_printing
def addition(self, a, b):
print("calling addition")
return a + b
If you want to keep the docstring of the original method, you would use the functools.wraps().
Example output
mycalc = MyCalculator()
print(mycalc.addition(2, 3))
would print
Before calling method
calling addition
After calling method
5
Wrapping a method of an object instance (runtime)
Here is one implementation which changes the method of an object. Note that this changes the method of an instance and not every instance of that class.
class MyCalculator:
def addition(self, a, b):
print("calling addition")
return a + b
class Changing:
def set_method_attribute(self, obj, method_name):
method = getattr(obj, method_name)
def wrapper(*args, **kwargs):
print("Before calling method")
ret = method(*args, **kwargs)
print("After calling method")
return ret
setattr(obj, method_name, wrapper)
Example usage
# Create two calculator objects for testing
mycalc = MyCalculator()
mycalc2 = MyCalculator()
change = Changing()
# Before change
print(mycalc.addition(2, 3))
print("----")
# After change
change.set_method_attribute(mycalc, "addition")
print(mycalc.addition(2, 3))
print("----")
# The another calculator object remains unchanged
print(mycalc2.addition(2, 3))
will print
calling addition
5
----
Before calling method
calling addition
After calling method
5
----
calling addition
5

I recreated some of the code in a new way. Would you mind taking a look on it and telling me if this is a "good" way?
import inspect
class InBetween():
def __init__(self):
object_names = [object_name for object_name in inspect.getmembers(self) if (inspect.ismethod(object_name[1]) and (object_name[0] != 'with_print'))]
for name in object_names:
method = getattr(self, name[0])
wrapper = self.with_print(method)
setattr(self, name[0], wrapper)
def with_print(self, method):
def wrapper(*args, **kwargs):
print("before")
ret = method(*args, **kwargs)
print("after")
return ret
return wrapper
class MyCalculator(InBetween):
def __init__(self):
super().__init__()
def addition(self, a, b):
return a + b
def substraction(self, a, b):
return a - b
def multiply(self, a, b):
return a * b
if __name__=="__main__":
myCalc = MyCalculator()
print(myCalc.addition(2,5))
print(myCalc.multiply(2,5))
The basic idea is, that every class wich will inherit "InBetween" can add via super() the wrapper to each method. Without doing this manually in the script. My next idea is, to replace the print statements by logging etc. At the end, if I call the parent "init" infromation will be easily logged, and if not, "nothing" happens.
Love to hear other opinions on that!
Thank you all!

How to replace every function call with the wrapper call in the original class?

I have two classes, one of which is a wrapper of the other. A function in the original class uses a method called forward, but I want it to use the forward method of the wrapper class after it has been wrapped, not the original. For example:
class A:
def __init__(self):
self.a = 1
def forward(self, x):
return self.a + x
def main(self, x):
return self.forward(x) + 100
class Wrapper:
def __init__(self, A):
self.module = A
def forward(self, x):
# Long convoluted code.
# ..
# ..
return self.module.forward(x)
classA = A()
wrapperA = Wrapper(classA)
# Goal: Make classA.main(..) use the forward function from Wrapper instead.
Because the wrapper class has the long and convoluted code that needs to be run, I want all calls of forward from main to be such that it calls the forward from the wrapper class, not from the original.
Is there a way to do this in Python?
Reasons why I did not use inheritance:
Instantiating class A is memory intensive. If I receive class A object as input, I want to modify its core behavior. without instantiating another object.
classA can be of different object types in runtime.
--
An alternative way I thought of is to redefine main in Wrapper. However, the problem is doing this automatically for every method defined in A without hard coding.

In Python "everything is an object". Including classes, functions, and methods
on objects.
As such we can take any class, loop over all functions in that class and modify
them as needed.
Depending on the real code, the problem in the question might be better tackled
using decorators or meta-classes, depending on the dependencies of the wrapper
(what values does it need access to). I will not go into meta-classes as most needs for meta-classes can also be implemented using class-decorators, which are less error-prone.
As you mention in one of your comments that you may have several different classes that need to be wrapped the class-decorator solution might be a good candidate. This way you won't lose the inheritance tree of the wrapped class.
Here is an example not using either, but doing exactly as asked ;)
Using __new__
from functools import update_wrapper
class A:
def __init__(self):
self.a = 1
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
def main(self, x):
return self.forward(x) + 100
class Wrapper:
# Using __new__ instead of __init__ gives us complete control *how* the
# "Wrapper" instance is created. We use it to "pull in" methods from *A*
# and dynamically attach them to the `Wrapper` instance using `setattr`.
#
# Using __new__ is error-prone however, and using either meta-classes or
# even easier, decorators would be more maintainable.
def __new__(cls, A):
# instance will be our instance of thie `Wrapper` class. We start off
# with no defined functions, we will add those soon...
instance = super().__new__(cls)
instance.module = A
# We now walk over every "name" in the wrapped class
for funcname in dir(A):
# We skip anything starting with two underscores. They are most
# likely magic methods that we don't want to wrap with the
# additional code. The conditions what exactly we want to wrap, can
# be adapted as needed.
if funcname.startswith("__"):
continue
# We now need to get a reference to that attribute and check if
# it's callable. If not it is a member variable or something else
# and we can/should skip it.
func = getattr(A, funcname)
if not callable(func):
continue
# Now we "wrap" the function with our additional code. This is done
# in a separate function to keep __new__ somewhat clean
wrapped = Wrapper._wrap(func)
# After wrapping the function we can attach that new function ont
# our `Wrapper` instance
setattr(instance, funcname, wrapped)
return instance
#staticmethod
def _wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
class Demo2:
def foo(self):
print("yoinks")
classA = A()
otherInstance = Demo2()
wrapperA = Wrapper(classA)
wrapperB = Wrapper(otherInstance)
print(wrapperA.forward(10))
print(wrapperB.foo())
print("docstring is maintained: %r" % wrapperA.forward.__doc__)
Using a class decorator
With a class decorator, there is no need to override __new__ which can lead to hard to debug issues if not 100% properly implemented.
However, it has a key difference: It modifies the existing class "in-place", so the original class is lost in a way. Although you could keep a reference to it in the unlikely case that you need to.
Modifying this in-place does however also mean that you don't need to replace all your usages in your application with the new "wrapper" class, making it a lot easier to implement in an existing code-base and eliminating the risk that you forget to apply the wrapper on new instances.
from functools import update_wrapper
def _wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
def wrapper(cls):
for funcname in dir(cls):
# We skip anything starting with two underscores. They are most
# likely magic methods that we don't want to wrap with the
# additional code. The conditions what exactly we want to wrap, can
# be adapted as needed.
if funcname.startswith("__"):
continue
# We now need to get a reference to that attribute and check if
# it's callable. If not it is a member variable or something else
# and we can/should skip it.
func = getattr(cls, funcname)
if not callable(func):
continue
# Now we "wrap" the function with our additional code. This is done
# in a separate function to keep __new__ somewhat clean
wrapped = _wrap(func)
# After wrapping the function we can attach that new function ont
# our `Wrapper` instance
setattr(cls, funcname, wrapped)
return cls
#wrapper
class A:
def __init__(self):
self.a = 1
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
def main(self, x):
return self.forward(x) + 100
#wrapper
class Demo2:
def foo(self):
print("yoinks")
classA = A()
otherInstance = Demo2()
print(classA.forward(10))
print(otherInstance.foo())
print("docstring is maintained: %r" % classA.forward.__doc__)
Using function decorators
Another alternative, which diverges largely from the original question but may still prove insightful is using individual functions wrappers.
The code still used the same wrapper function, but here functions/methods are annotated individually.
This might give more flexibility by offering the possibility to leave some methods "unwrapped", but could easily lead to the wrapping code being executed more often than anticipated as demonstrated in the main() method.
from functools import update_wrapper
def wrap(func):
"""
Wraps *func* with additional code.
"""
# we define a wrapper function. This will execute all additional code
# before and after the "real" function.
def wrapped(*args, **kwargs):
print("before-call:", func, args, kwargs)
output = func(*args, **kwargs)
print("after-call:", func, args, kwargs, output)
return output
# Use "update_wrapper" to keep docstrings and other function metadata
# intact
update_wrapper(wrapped, func)
# We can now return the wrapped function
return wrapped
class A:
def __init__(self):
self.a = 1
#wrap
def forward(self, x):
"""
docstring (to demonstrate `update_wrapper`
"""
print("original forward")
return self.a + x
#wrap # careful: will be wrapped twice!
def main(self, x):
return self.forward(x) + 100
def foo(self):
print("yoinks")
classA = A()
print(">>> forward")
print(classA.forward(10))
print("<<< forward")
print(">>> main")
print(classA.main(100))
print("<<< main")
print(">>> foo")
print(classA.foo())
print("<<< foo")

You could inherit Wrapper from A, and use super to access the parent class.
class A:
def __init__(self, child):
self.a = 1
self.child = child
def forward(self, x):
return self.a + x
def main(self, x):
return self.child.forward(x) + 100
class Wrapper(A):
def __init__(self):
super(Wrapper, self).__init__(self, self)
def forward(x):
return "whatever"
wrapperA = Wrapper()
But if you wish to use class A, just inherit A from Wrapper. Otherwise, I can't figure out whats wrong. Please don't use functions indiscriminate. Make a class you wish to use, and another one act as a parent and don't mix roles.
#...
class A(Wrapper):
def __init__(self):
super(A, self).__init__(self)
#...

How to add methods of class to a list inside the class with a decorator

I would like to update a "class-wide" list from a decorator that decorates the class' methods and adds each decorated method to that list.
This is what came to mind:
def add(meth: callable):
Spam.eggs.append(func)
return meth
class Spam:
eggs = []
#add
def meth(self):
pass
This won't work though because Spam hasn't finished defining itself when #add is reached, and thus add raises a NameError, as pointed out in the comments.
I also tried a class method:
class Spam:
eggs = []
#classmethod
def add(cls, meth: callable):
cls.eggs.append(meth)
return meth
#add
def meth(self):
pass
But this doesn't work either because when #add is reached, add is bound to the classmethod decorated instance, which is not callable.
Here is what I need this for:
I have a class with several methods that take one argument (besides self) that transform that object in such a way that these methods may be composed with one another. I want to decorate each of these in such a way that they're automatically added to a list in the class.
E.g.:
from typing import List
def transform_meth(meth: callable):
TextProcessor.transforms.add(meth)
return meth
class TextProcessor:
transforms: List[callable] = []
#transform_meth
def m1(self, text):
return text
#transform_meth
def m2(self, text):
return text
def transform(self, text):
for transform in self.transforms:
text = transform(text)
return text
I could add the methods in the list manually, but I find the decorator to be clearer since it is close to the definition of the method, and thus it is easier to remember to decorate a new method when defining it than adding it to the list manually.

Your current approach fails because when transform_meth is called, TextProcessor isn't bound to anything yet (or if it is, that object gets overwritten when the class statement completes).
The simple solution would be to define transform_meth inside the class statement, so that it could simply declare transforms as a nonlocal variable. However, that won't work because a class statement doesn't establish a new scope.
Instead, you can define a function that creates the decorator, which takes the desired list (at that point a just a name in the body of the class statement, not from any assumed scope). That function returns a closure over the list argument
so that you can append to it.
def make_decorator(lst):
# *This* will be the function bound to the name 'transform_meth'
def _(meth):
lst.append(meth)
return meth
return _
class TextProcessor:
transforms: List[callable] = []
transform_meth = make_decorator(transforms)
#transform_meth
def m1(self, text):
return text
#transform_meth
def m2(self, text):
return text
def transform(self, text):
for transform in self.transforms:
text = transform(text)
return text
del transform_meth # Not needed anymore, don't create a class attribute

Since the arg of each method is self you can append to the object instance like so:
from functools import wraps
def appender(f):
#wraps(f)
def func(*args, **kwargs):
if f not in args[0].transforms:
args[0].transforms.append(f)
return f(*args, **kwargs)
return func
class Foo(object):
def __init__(self):
self.transforms = []
#appender
def m1(self, arg1):
return arg1
#appender
def m2(self, arg1):
return arg1
def transform(self, text):
methods = [f for f in dir(self) if not f.startswith("__") and callable(getattr(self,f)) and f != 'transform']
for f in methods:
text = getattr(self,f)(text)
return text
f = Foo()
f.transform('your text here')
print(f.transforms)
Output:
[<function Foo.m1 at 0x1171e4e18>, <function Foo.m2 at 0x1171e4268>]

How to create an object collection proxy in Python?

I'm trying to create an object collection proxy, which could do something like this:
class A:
def do_something():
# ...
class B:
def get_a():
return A()
class Proxy:
?
collection = [B(), B()]
proxy = Proxy(collection)
proxy.get_a().do_something()
# ^ for each B in collection get_a() and do_something()
What would be the best architecture / strategy for achieving this?
The key question, I guess is, how to cache the result of get_a() so I can then proxy do_something()
N.B. I don't expect proxy.get_a().do_something() to return anything sensible, it's only supposed to do things.

Simple enough... you may want to adapt it to do some more checking
class A(object):
def do_something(self):
print id(self), "called"
class B(object):
def get_a(self):
return A()
class Proxy(object):
def __init__(self, objs):
self._objs = objs
def __getattr__(self, name):
def func(*args, **kwargs):
return Proxy([getattr(o, name)(*args, **kwargs) for o in self._objs])
return func
collection = [B(), B()]
proxy = Proxy(collection)
proxy.get_a().do_something()
Results in:
4455571152 called
4455571216 called

The most pythonic way of going about this would probably be a list comprehension:
results = [b.get_a().do_something() for b in collection]
If you want to cache calls to B.get_a(), you can use memoization. A simple way of doing memoization yourself could look like this:
cache = None
# ...
class B:
def get_a(self):
global cache
if cache is None:
cache = A()
return cache
If you want to use caching in multiple places, you'll need to cache results based on keys in order to distinguish them, and for convenience's sake write a decorator that you can simply wrap functions with whose results you want to cache.
A good example of this is found in Python Algorithms: Mastering Basic Algorithms in the Python Language (see this question). Modified for your case, to not use the function arguments but the function name as cache key, it would look like this:
from functools import wraps
def memoize(func):
cache = {}
key = func.__name__
# wraps(func)
def wrap(*args):
if key not in cache:
cache[key] = func(*args)
return cache[key]
return wrap
class A:
def do_something(self):
return 1
class B:
#memoize
def get_a(self):
print "B.get_a() was called"
return A()
collection = [B(), B()]
results = [b.get_a().do_something() for b in collection]
print results
Output:
B.get_a() was called
[1, 1]

How are these type of python decorators written?

I'd like to write a decorator that would limit the number of times a function can be executed, something along the following syntax :
#max_execs(5)
def my_method(*a,**k):
# do something here
pass
I think it's possible to write this type of decorator, but I don't know how. I think a function won't be this decorator's first argument, right? I'd like a "plain decorator" implementation, not some class with a call method.
The reason for this is to learn how they are written. Please explain the syntax, and how that decorator works.

This is what I whipped up. It doesn't use a class, but it does use function attributes:
def max_execs(n=5):
def decorator(fn):
fn.max = n
fn.called = 0
def wrapped(*args, **kwargs):
fn.called += 1
if fn.called <= fn.max:
return fn(*args, **kwargs)
else:
# Replace with your own exception, or something
# else that you want to happen when the limit
# is reached
raise RuntimeError("max executions exceeded")
return wrapped
return decorator
max_execs returns a functioned called decorator, which in turn returns wrapped. decoration stores the max execs and current number of execs in two function attributes, which then get checked in wrapped.
Translation: When using the decorator like this:
#max_execs(5)
def f():
print "hi!"
You're basically doing something like this:
f = max_execs(5)(f)

Decorator is merely a callable that transforms a function into something else. In your case, max_execs(5) must be a callable that transforms a function into another callable object that will count and forward the calls.
class helper:
def __init__(self, i, fn):
self.i = i
self.fn = fn
def __call__(self, *args, **kwargs):
if self.i > 0:
self.i = self.i - 1
return self.fn(*args, **kwargs)
class max_execs:
def __init__(self, i):
self.i = i
def __call__(self, fn):
return helper(self.i, fn)
I don't see why you would want to limit yourself to a function (and not a class). But if you really want to...
def max_execs(n):
return lambda fn, i=n: return helper(i, fn)

There are two ways of doing it. The object-oriented way is to make a class:
class max_execs:
def __init__(self, max_executions):
self.max_executions = max_executions
self.executions = 0
def __call__(self, func):
#wraps(func)
def maybe(*args, **kwargs):
if self.executions < self.max_executions:
self.executions += 1
return func(*args, **kwargs)
else:
print "fail"
return maybe
See this question for an explanation of wraps.
I prefer the above OOP approach for this kind of decorator, since you've basically got a private count variable tracking the number of executions. However, the other approach is to use a closure, such as
def max_execs(max_executions):
executions = [0]
def actual_decorator(func):
#wraps(func)
def maybe(*args, **kwargs):
if executions[0] < max_executions:
executions[0] += 1
return func(*args, **kwargs)
else:
print "fail"
return maybe
return actual_decorator
This involved three functions. The max_execs function is given a parameter for the number of executions and returns a decorator that will restrict you to that many calls. That function, the actual_decorator, does the same thing as our __call__ method in the OOP example. The only weirdness is that since we don't have a class with private variables, we need to mutate the executions variable which is in the outer scope of our closure. Python 3.0 supports this with the nonlocal statement, but in Python 2.6 or earlier, we need to wrap our executions count in a list so that it can be mutated.

Without relying to a state in a class, you have to save the state (count) in the function itself:
def max_execs(count):
def new_meth(meth):
meth.count = count
def new(*a,**k):
meth.count -= 1
print meth.count
if meth.count>=0:
return meth(*a,**k)
return new
return new_meth
#max_execs(5)
def f():
print "invoked"
[f() for _ in range(10)]
It gives:
5
invoked
4
invoked
3
invoked
2
invoked
1
invoked
0
-1
-2
-3
-4

This method does not modify function internals, instead wraps it into a callable object.
Using class slows down execution by ~20% vs using the patched function!
def max_execs(n=1):
class limit_wrapper:
def __init__(self, fn, max):
self.calls_left = max
self.fn = fn
def __call__(self,*a,**kw):
if self.calls_left > 0:
self.calls_left -= 1
return self.fn(*a,**kw)
raise Exception("max num of calls is %d" % self.i)
def decorator(fn):
return limit_wrapper(fn,n)
return decorator
#max_execs(2)
def fun():
print "called"

I know you said you didn't want a class, but unfortunately that's the only way I can think of how to do it off the top of my head.
class mymethodwrapper:
def __init__(self):
self.maxcalls = 0
def mymethod(self):
self.maxcalls += 1
if self.maxcalls > 5:
return
#rest of your code
print "Code fired!"
Fire it up like this
a = mymethodwrapper
for x in range(1000):
a.mymethod()
The output would be:
>>> Code fired!
>>> Code fired!
>>> Code fired!
>>> Code fired!
>>> Code fired!