I am writing a class that sends slack messages to users when processes have finished. I thought it would be useful to provide a Jupyter magic so that users can be notified when the cell is executed.
The class already provides a decorator, so I figured I'd just wrap a cell execution in a decorated function.
from IPython.core.magic import register_cell_magic
from IPython import get_ipython
import functools
class MyClass(object):
def decorate(self, f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
r = f(*args, **kwargs)
print('Send a message here!')
return r
return wrapped
#register_cell_magic
def magic(self, line, cell):
ip = get_ipython()
#self.decorate
def f():
return ip.run_cell(cell)
return f()
So then I'd do:
obj = MyClass()
# ----- NEW CELL
%%obj.magic
'''do some stuff'''
But I get
>> UsageError: Cell magic `%%obj.magic` not found.
I found out that the magic is registered under its name (above, magic), so %%magic works. But then the arguments are all messed up because there is no self in the mix.
I want the magic to be an instance method so that config (set in __init__ can be used). Is there any way to do this?
Here are a couple hacky solutions I don't want to implement unless I really have to:
Register a regular function with the instance as an argument. I don't want to add that line of code to the notebook, I want to use an instance method.
Register a regular function that constructs an instance on the fly.
This is the best I can come up with, and it's #1 on the list of the things I didn't want to do.
from IPython.core.magic import register_cell_magic
from IPython import get_ipython
import functools
class MyClass(object):
def decorate(self, f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
r = f(*args, **kwargs)
print('Send a message here!')
return r
return wrapped
def register_magic(self):
#register_cell_magic
def magic(line, cell):
ip = get_ipython()
#self.decorate
def f():
return ip.run_cell(cell)
return f()
Then
obj = MyClass()
obj.register_magic()
# ------
%%magic
...
Related
I'm trying to find a way to dynamically add methods to a class through decorator.
The decorator i have look like:
def deco(target):
def decorator(function):
#wraps(function)
def wrapper(self, *args, **kwargs):
return function(*args, id=self.id, **kwargs)
setattr(target, function.__name__, wrapper)
return function
return decorator
class A:
pass
# in another module
#deco(A)
def compute(id: str):
return do_compute(id)
# in another module
#deco(A)
def compute2(id: str):
return do_compute2(id)
# **in another module**
a = A()
a.compute() # this should work
a.compute2() # this should work
My hope is the decorator should add the compute() function to class A, any object of A should have the compute() method.
However, in my test, this only works if i explicitly import compute into where an object of A is created. I think i'm missing something obvious, but don't know how to fix it. appreciate any help!
I think this will be quite simpler using a decorator implemented as a class:
class deco:
def __init__(self, cls):
self.cls = cls
def __call__(self, f):
setattr(self.cls, f.__name__, f)
return self.cls
class A:
def __init__(self, val):
self.val = val
#deco(A)
def compute(a_instance):
print(a_instance.val)
A(1).compute()
A(2).compute()
outputs
1
2
But just because you can do it does not mean you should. This can become a debugging nightmare, and will probably give a hard time to any static code analyser or linter (PyCharm for example "complains" with Unresolved attribute reference 'compute' for class 'A')
Why doesn't it work out of the box when we split it to different modules (more specifically, when compute is defined in another module)?
Assume the following:
a.py
print('importing deco and A')
class deco:
def __init__(self, cls):
self.cls = cls
def __call__(self, f):
setattr(self.cls, f.__name__, f)
return self.cls
class A:
def __init__(self, val):
self.val = val
b.py
print('defining compute')
from a import A, deco
#deco(A)
def compute(a_instance):
print(a_instance.val)
main.py
from a import A
print('running main')
A(1).compute()
A(2).compute()
If we execute main.py we get the following:
importing deco and A
running main
Traceback (most recent call last):
A(1).compute()
AttributeError: 'A' object has no attribute 'compute'
Something is missing. defining compute is not outputted. Even worse, compute is never defined, let alone getting bound to A.
Why? because nothing triggered the execution of b.py. Just because it sits there does not mean it gets executed.
We can force its execution by importing it. Feels kind of abusive to me, but it works because importing a file has a side-effect: it executes every piece of code that is not guarded by if __name__ == '__main__, much like importing a module executes its __init__.py file.
main.py
from a import A
import b
print('running main')
A(1).compute()
A(2).compute()
outputs
importing deco and A
defining compute
running main
1
2
I want to create an Abstract Factory in order to abstract hardware differences between computers (say a RaspberryPi and an Arduino) in Python 2.7.
I am using the following implementation of an Abstract Factory:
'''
Provide a device-agnostic display interface
'''
from hardware import sysname
class DisplayBase(object):
def __init__(self):
pass
def show(self, message):
pass
def __str__(self):
return "DisplayBase"
def __repr__(self):
return self.__str__()
class RPIDisplay(DisplayBase):
def __new__(cls, *args, **kwargs):
from rpi_display import writeline
instance = super(RPIDisplay, cls).__new__(cls, *args, **kwargs)
return instance
def __str__(self):
return "RPIDisplay"
def show(self, message):
writeline(message)
class ArduinoDisplay(DisplayBase):
def __new__(cls, *args, **kwargs):
import arduino_display
instance = super(ArduinoDisplay, cls).__new__(cls, *args, **kwargs)
return instance
def __str__(self):
return "ArduinoDisplay"
def show(self, message):
return arduino_display.println(message)
class Display(DisplayBase): # Display Factory
def __new__(cls, *args, **kwargs):
platform = sysname()
if platform == "RaspberryPi":
return RPIDisplay()
elif platform == "Arduino":
return ArduinoDisplay()
else:
return MockDisplay()
if __name__ == "__main__":
display = Display()
print display
display.show("hello world")
The instantiation works fine, but when I try to run this, I get:
ArduinoDisplay
Traceback (most recent call last):
File "tt.py", line 56, in <module>
display.show("hello world")
File "tt.py", line 41, in show
return arduino_display.println(message)
NameError: global name 'arduino_display' is not defined
So the import of arduino_display does sorta work, but I cannot find a way to use it within the object.
Conditional imports are needed since different platforms will have different modules installed.
Any idea how to use those conditional imports?
I tried self.arduino_display and ArduinoDisplay.arduino_display but to no avail.
I could obviously catch import errors, as in, add to the top:
try:
import arduino_display
except:
pass
...and that would fail on a RPI, which would be fine, but there must be a better way...
The problem is due to import only binding names in its current scope. Doing an import in a function/method does not make it available in other methods.
Perform the imports where you actually need them. For example, ArduinoDisplay should import arduino_display where it is used:
class ArduinoDisplay(DisplayBase):
# no new, no import
def __str__(self):
return "ArduinoDisplay"
def show(self, message):
# import where needed
import arduino_display
return arduino_display.println(message)
Note that import is idempotent -- if the module has already been loaded before, import just binds the name again. This makes such nested import statements fast enough for most cases.
If your classes need many imports or speed is an issue, isolate classes into separate modules and import the entire module conditionally. You can directly assign the correct class using the common name, instead of having a dummy type that constructs another.
# ## display/arduino.py ##
# other systems accordingly
from .base import DisplayBase
# import once, globally
import arduino_display
class ArduinoDisplay(DisplayBase):
# no new, no import
def __str__(self):
return "ArduinoDisplay"
def show(self, message):
# import where needed
return arduino_display.println(message)
# ## display/__init__.py ##
from hardware import sysname
platform = sysname()
# resolve and import appropriate type once
if platform == "RaspberryPi":
from .rpi import RPIDisplay as Display
elif platform == "Arduino":
from .arduino import ArduinoDisplay as Display
else:
from .mock import MockDisplay as Display
Have you tried using from arduino_display import println then use println just like you did with the RPIs writeline?
Edit: missed the obvious...
You can do somthing like this:
from hardware import sysname
class Display(object):
def __init__(self, print_function, display_name):
self._print_function = print_function
self._display_name = display_name
def show(self, message):
self.print_function(message)
def __str__(self):
return self._display_name
def __repr__(self):
return self.__str__()
def create_display():
platform = sysname()
if platform == "RaspberryPi":
from rpi_display import writeline
return Display(writeline, "RPIDisplay")
elif platform == "Arduino":
from arduino_display import println
return Display(println, "ArduinoDisplay")
In case you need the classes and the nested factory, you can apply the same principle of storing the function object there too.
The problem is that the function you import aren't saved anywhere for the other methods to see them, so they are not visible anywhere else but locally. Try setting ArduinoDisplay.arduino_display = arduino_display after you import.
Here's some sample code that illustrates my meaning:
class ADisplay:
def __init__(self):
from builtins import print as display_write
def show(self, msg):
display_write(msg)
disp = ADisplay()
disp.show("mymsg")
This fails with NameError: name 'display_write' is not defined.
Now, bind the import to your class.
class ADisplay:
def __init__(self):
from builtins import print as display_write
ADisplay.display_write = display_write
def show(self, msg):
self.display_write(msg)
disp = ADisplay()
disp.show("mymsg")
This works. In fact you can even dispense with show by assigning it directly, if all those hardware print methods only take a string as a parameter and if you don't need to format or modify anything.
class ADisplay:
def __init__(self):
#do other init stuff...
pass
#only bind show the first time.
if getattr(ADisplay, "show", None):
return
from builtins import print as display_write
ADisplay.show = display_write
disp = ADisplay()
disp.show("mymsg")
I just started using decorators in python and thought it might be useful to write a decorator for pubsub.subscribe
I'm using Python 3.6 so functools.wraps should make the wrapper having the same signature as the wrapped function, but pub.sendMessage still complains about the additional unknown argument.
from pubsub import pub
from functools import wraps
def subscribe(message):
def subscribe_deco(func):
#wraps(func)
def wrapper(*args,**kwargs):
return func(*args, **kwargs)
pub.subscribe(wrapper, message)
return wrapper
return subscribe_deco
#subscribe('testmessage')
def test(bar='empty'):
print('foo ' + bar)
if __name__ == '__main__':
test('bar')
pub.sendMessage('testmessage')
pub.sendMessage('testmessage', bar='decorated')
Result:
foo bar
foo empty
pubsub.core.topicargspec.SenderUnknownMsgDataError: Some optional args unknown in call to sendMessage('('testmessage',)', bar): bar
Update:
I just realized that the inner wrapper is useless and I can just subscribe to the decorated function directly. But I'm not sure then, if I even need a decorator in that case ...
def subscribe(message):
def subscribe_deco(func):
pub.subscribe(func, message)
return func
return subscribe_deco
Let's consider this piece of code where I would like to create bar dynamically with a decorator
def foo():
def bar():
print "I am bar from foo"
print bar()
def baz():
def bar():
print "I am bar from baz"
print bar()
I thought I could create bar from the outside with a decorator:
def bar2():
print "I am super bar from foo"
setattr(foo, 'bar', bar2)
But the result is not what I was expecting (I would like to get I am super bar from foo:
>>> foo()
I am bar from foo
Is it possible to override a sub-function on an existing function with a decorator?
The actual use case
I am writing a wrapper for a library and to avoid boilerplate code I would like to simplify my work.
Each library function has a prefix lib_ and returns an error code. I would like to add the prefix to the current function and treat the error code. This could be as simple as this:
def call():
fname = __libprefix__ + inspect.stack()[1][3]
return_code = getattr(__lib__, fname)(*args)
if return_code < 0: raise LibError(fname, return_code)
def foo():
call()
The problem is that call might act differently in certain cases. Some library functions do not return an error_code so it would be easier to write it like
this:
def foo():
call(check_status=True)
Or much better in my opinion (this is the point where I started thinking about decorators):
#LibFunc(check_status=True)
def foo():
call()
In this last example I should declare call inside foo as a sub-function created dynamically by the decorator itself.
The idea was to use something like this:
class LibFunc(object):
def __init__(self,**kwargs):
self.kwargs = kwargs
def __call__(self, original_func):
decorator_self = self
def wrappee( *args, **kwargs):
def call(*args):
fname = __libprefix__ + original_func.__name__
return_code = getattr(__lib__, fname)(*args)
if return_code < 0: raise LibError(fname, return_code)
print original_func
print call
# <<<< The part that does not work
setattr(original_func, 'call', call)
# <<<<
original_func(*args,**kwargs)
return wrappee
Initially I was tempted to call the call inside the decorator itself to minimize the writing:
#LibFunc():
foo(): pass
Unfortunately, this is not an option since other things should sometime be done before and after the call:
#LibFunc():
foo(a,b):
value = c_float()
call(a, pointer(value), b)
return value.value
Another option that I thought about was to use SWIG, but again this is not an option because I will need to rebuild the existing library with the SWIG wrapping functions.
And last but not least, I may get inspiration from SWIG typemaps and declare my wrapper as this:
#LibFunc(check_exit = true, map = ('<a', '>c_float', '<c_int(b)')):
foo(a,b): pass
This looks like the best solution to me, but this is another topic and another question...
Are you married to the idea of a decorator? Because if your goal is bunch of module-level functions each of which wraps somelib.lib_somefunctionname, I don't see why you need one.
Those module-level names don't have to be functions, they just have to be callable. They could be a bunch of class instances, as long as they have a __call__ method.
I used two different subclasses to determine how to treat the return value:
#!/usr/bin/env python3
import libtowrap # Replace with the real library name.
class Wrapper(object):
'''
Parent class for all wrapped functions in libtowrap.
'''
def __init__(self, name):
self.__name__ = str(name)
self.wrapped_name = 'lib_' + self.__name__
self.wrapped_func = getattr(libtowrap, self.wrapped_name)
self.__doc__ = self.wrapped_func.__doc__
return
class CheckedWrapper(Wrapper):
'''
Wraps functions in libtowrap that return an error code that must
be checked. Negative return values indicate an error, and will
raise a LibError. Successful calls return None.
'''
def __call__(self, *args, **kwargs):
error_code = self.wrapped_func(*args, **kwargs)
if error_code < 0:
raise LibError(self.__name__, error_code)
return
class UncheckedWrapper(Wrapper):
'''
Wraps functions in libtowrap that return a useful value, as
opposed to an error code.
'''
def __call__(self, *args, **kwargs):
return self.wrapped_func(*args, **kwargs)
strict = CheckedWrapper('strict')
negative_means_failure = CheckedWrapper('negative_means_failure')
whatever = UncheckedWrapper('whatever')
negative_is_ok = UncheckedWrapper('negative_is_ok')
Note that the wrapper "functions" are assigned while the module is being imported. They are in the top-level module namespace, and not hidden by any if __name__ == '__main__' test.
They will behave like functions for most purposes, but there will be minor differences. For example, I gave each instance a __name__ that matches the name they're assigned to, not the lib_-prefixed name used in libtowrap... but I copied the original __doc__, which might refer to a prefixed name like lib_some_other_function. Also, testing them with isinstance will probably surprise people.
For more about decorators, and for many more annoying little discrepancies like the ones I mentioned above, see Graham Dumpleton's half-hour lecture "Advanced Methods for Creating Decorators" (PyCon US 2014; slides). He is the author of the wrapt module (Python Package Index; Git Hub; Read the Docs), which corrects all(?) of the usual decorator inconsistencies. It might solve your problem entirely (except for the old lib_-style names showing up in __doc__).
What's the best way to toggle decorators on and off, without actually going to each decoration and commenting it out? Say you have a benchmarking decorator:
# deco.py
def benchmark(func):
def decorator():
# fancy benchmarking
return decorator
and in your module something like:
# mymodule.py
from deco import benchmark
class foo(object):
#benchmark
def f():
# code
#benchmark
def g():
# more code
That's fine, but sometimes you don't care about the benchmarks and don't want the overhead. I have been doing the following. Add another decorator:
# anothermodule.py
def noop(func):
# do nothing, just return the original function
return func
And then comment out the import line and add another:
# mymodule.py
#from deco import benchmark
from anothermodule import noop as benchmark
Now benchmarks are toggled on a per-file basis, having only to change the import statement in the module in question. Individual decorators can be controlled independently.
Is there a better way to do this? It would be nice to not have to edit the source file at all, and to specify which decorators to use in which files elsewhere.
You could add the conditional to the decorator itself:
def use_benchmark(modname):
return modname == "mymodule"
def benchmark(func):
if not use_benchmark(func.__module__):
return func
def decorator():
# fancy benchmarking
return decorator
If you apply this decorator in mymodule.py, it will be enabled; if you apply it in othermodule.py, it will not be enabled.
I've been using the following approach. It's almost identical to the one suggested by CaptainMurphy, but it has the advantage that you don't need to call the decorator like a function.
import functools
class SwitchedDecorator:
def __init__(self, enabled_func):
self._enabled = False
self._enabled_func = enabled_func
#property
def enabled(self):
return self._enabled
#enabled.setter
def enabled(self, new_value):
if not isinstance(new_value, bool):
raise ValueError("enabled can only be set to a boolean value")
self._enabled = new_value
def __call__(self, target):
if self._enabled:
return self._enabled_func(target)
return target
def deco_func(target):
"""This is the actual decorator function. It's written just like any other decorator."""
def g(*args,**kwargs):
print("your function has been wrapped")
return target(*args,**kwargs)
functools.update_wrapper(g, target)
return g
# This is where we wrap our decorator in the SwitchedDecorator class.
my_decorator = SwitchedDecorator(deco_func)
# Now my_decorator functions just like the deco_func decorator,
# EXCEPT that we can turn it on and off.
my_decorator.enabled=True
#my_decorator
def example1():
print("example1 function")
# we'll now disable my_decorator. Any subsequent uses will not
# actually decorate the target function.
my_decorator.enabled=False
#my_decorator
def example2():
print("example2 function")
In the above, example1 will be decorated, and example2 will NOT be decorated. When I have to enable or disable decorators by module, I just have a function that makes a new SwitchedDecorator whenever I need a different copy.
I think you should use a decorator a to decorate the decorator b, which let you switch the decorator b on or off with the help of a decision function.
This sounds complex, but the idea is rather simple.
So let's say you have a decorator logger:
from functools import wraps
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
This is a very boring decorator and I have a dozen or so of these, cachers, loggers, things which inject stuff, benchmarking etc. I could easily extend it with an if statement, but this seems to be a bad choice; because then I have to change a dozen of decorators, which is not fun at all.
So what to do? Let's step one level higher. Say we have a decorator, which can decorate a decorator? This decorator would look like this:
#point_cut_decorator(logger)
def my_oddly_behaving_function
This decorator accepts logger, which is not a very interesting fact. But it also has enough power to choose if the logger should be applied or not to my_oddly_behaving_function. I called it point_cut_decorator, because it has some aspects of aspect oriented programming. A point cut is a set of locations, where some code (advice) has to be interwoven with the execution flow. The definitions of point cuts is usually in one place. This technique seems to be very similar.
How can we implement it decision logic. Well I have chosen to make a function, which accepts the decoratee, the decorator, file and name, which can only say if a decorator should be applied or not. These are the coordinates, which are good enough to pinpoint the location very precisely.
This is the implementation of point_cut_decorator, I have chosen to implement the decision function as a simple function, you could extend it to let it decide from your settings or configuration, if you use regexes for all 4 coordinates, you will end up with something very powerful:
from functools import wraps
myselector is the decision function, on true a decorator is applied on false it is not applied. Parameters are the filename, the module name, the decorated object and finally the decorator. This allows us to switch of behaviour in a fine grained manner.
def myselector(fname, name, decoratee, decorator):
print fname
if decoratee.__name__ == "test" and fname == "decorated.py" and decorator.__name__ == "logger":
return True
return False
This decorates a function, checks myselector and if myselector says go on, it will apply the decorator to the function.
def point_cut_decorator(d):
def innerdecorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
if myselector(__file__, __name__, f, d):
ps = d(f)
return ps(*args, **kwargs)
else:
return f(*args, **kwargs)
return wrapper
return innerdecorator
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
And this is how you use it:
#point_cut_decorator(logger)
def test(a):
print "hello"
return "world"
test(1)
EDIT:
This is the regular expression approach I talked about:
from functools import wraps
import re
As you can see, I can specify somewhere a couple of rules, which decides a decorator should be applied or not:
rules = [{
"file": "decorated.py",
"module": ".*",
"decoratee": ".*test.*",
"decorator": "logger"
}]
Then I loop over all rules and return True if a rule matches or false if a rule doesn't matches. By making rules empty in production, this will not slow down your application too much:
def myselector(fname, name, decoratee, decorator):
for rule in rules:
file_rule, module_rule, decoratee_rule, decorator_rule = rule["file"], rule["module"], rule["decoratee"], rule["decorator"]
if (
re.match(file_rule, fname)
and re.match(module_rule, name)
and re.match(decoratee_rule, decoratee.__name__)
and re.match(decorator_rule, decorator.__name__)
):
return True
return False
Here is what I finally came up with for per-module toggling. It uses #nneonneo's suggestion as a starting point.
Random modules use decorators as normal, no knowledge of toggling.
foopkg.py:
from toggledeco import benchmark
#benchmark
def foo():
print("function in foopkg")
barpkg.py:
from toggledeco import benchmark
#benchmark
def bar():
print("function in barpkg")
The decorator module itself maintains a set of function references for all decorators that have been disabled, and each decorator checks for its existence in this set. If so, it just returns the raw function (no decorator). By default the set is empty (everything enabled).
toggledeco.py:
import functools
_disabled = set()
def disable(func):
_disabled.add(func)
def enable(func):
_disabled.discard(func)
def benchmark(func):
if benchmark in _disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco
The main program can toggle individual decorators on and off during imports:
from toggledeco import benchmark, disable, enable
disable(benchmark) # no benchmarks...
import foopkg
enable(benchmark) # until they are enabled again
import barpkg
foopkg.foo() # no benchmarking
barpkg.bar() # yes benchmarking
reload(foopkg)
foopkg.foo() # now with benchmarking
Output:
function in foopkg
--> benchmarking bar((),{})
function in barpkg
<-- done
--> benchmarking foo((),{})
function in foopkg
<-- done
This has the added bug/feature that enabling/disabling will trickle down to any submodules imported from modules imported in the main function.
EDIT:
Here's class suggested by #nneonneo. In order to use it, the decorator must be called as a function ( #benchmark(), not #benchmark ).
class benchmark:
disabled = False
#classmethod
def enable(cls):
cls.disabled = False
#classmethod
def disable(cls):
cls.disabled = True
def __call__(cls,func):
if cls.disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco
I would implement a check for a config file inside the decorator's body. If benchmark has to be used according to the config file, then I would go to your current decorator's body. If not, I would return the function and do nothing more. Something in this flavor:
# deco.py
def benchmark(func):
if config == 'dontUseDecorators': # no use of decorator
# do nothing
return func
def decorator(): # else call decorator
# fancy benchmarking
return decorator
What happens when calling a decorated function ? # in
#benchmark
def f():
# body comes here
is syntactic sugar for this
f = benchmark(f)
so if config wants you to overlook decorator, you are just doing f = f() which is what you expect.
I don't think anyone has suggested this yet:
benchmark_modules = set('mod1', 'mod2') # Load this from a config file
def benchmark(func):
if not func.__module__ in benchmark_modules:
return func
def decorator():
# fancy benchmarking
return decorator
Each function or method has a __module__ attribute that is the name of the module where the function is defined. Create a whitelist (or blacklist if you prefer) of modules where benchmarking is to occur, and if you don't want to benchmark that module just return the original undecorated function.
another straight way:
# mymodule.py
from deco import benchmark
class foo(object):
def f():
# code
if <config.use_benchmark>:
f = benchmark(f)
def g():
# more code
if <config.use_benchmark>:
g = benchmark(g)
Here's a workaround to automatically toggle a decorator (here: #profile used by line_profiler):
if 'profile' not in __builtins__ or type(__builtins__) is not dict: profile=lambda x: None;
More info
This conditional (only if needed) instantiation of the profile variable (as an empty lambda function) prevents raising NameError when trying to import our module with user-defined functions where the decorator #profile is applied to every profiled user function. If I ever want to use the decorator for profiling - it will still work, not being overwritten (already existing in an external script kernprof that contains this decorator).