I would like to write a decorator to redirect stdout for the main function to a specific log file. The argument that the main function takes is - item, and I want the decorator to allocate a different log path with each item for the main function. How do I achieve this?
Currently I have:
def redirect_stdout(func):
def wrapper():
with open(f"{log_path}{item}.log", "w") as log, contextlib.redirect_stdout(log), contextlib.redirect_stderr(log):
func(item)
return wrapper()
#redirect_stdout
def main(item):
But I am not sure how the item argument goes into the decorator. Thanks!
What you are looking for is something like below
def redirect_stdout(func):
def wrapper(item):
with open(f"{log_path}{item}.log", "w") as log, contextlib.redirect_stdout(log), contextlib.redirect_stderr(log):
func(item)
return wrapper
To understand how this works you need to properly understand how the decorator works. Check below I have tried to explain how the decorator works. ==> I have used to indicate this is equivalent to.
#redirect_stdout
def main(item):
pass
== >
def main(item):
pass
main = redirect_stdout(main) = wrapper
---------
main(item) => wrapper(item)
I'm trying to understand the arguments that are passed to a pyramid view function.
The following example demonstrates a function wrapped with two different wrapppers. The only difference between the two wrappers is the signature. In the first wrapper, the first positional argument (obj) is explicit. In the second, it is included in *args.
import functools
from pyramid.config import Configurator
import webtest
def decorator_1(func):
#functools.wraps(func)
def wrapper(obj, *args, **kwargs): # <- obj
print('decorator_1')
print(type(obj), obj)
print(args)
print(kwargs)
return func(obj, *args, **kwargs) # <- obj
wrapper.__wrapped__ = func
return wrapper
def decorator_2(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
print('decorator_2')
print(args)
print(kwargs)
return func(*args, **kwargs)
wrapper.__wrapped__ = func
return wrapper
#decorator_1
def func_1(request):
return {'func': 'func_1'}
#decorator_2
def func_2(request):
return {'func': 'func_2'}
I would expect both wrapepd method to behave the same.
In decorator_1, I expect obj to be a request object and indeed it is.
In decorator_2, I would expect args[0] to be the same request object but it is not. It appears an additional first positional argument is passed before the request object.
def add_route(config, route, view, renderer="json"):
"""Helper for adding a new route-view pair."""
route_name = view.__name__
config.add_route(route_name, route)
config.add_view(view, route_name=route_name, renderer=renderer)
config = Configurator()
add_route(config, "/func_1", func_1)
add_route(config, "/func_2", func_2)
app = config.make_wsgi_app()
testapp = webtest.TestApp(app)
testapp.get("/func_1")
testapp.get("/func_2")
Output:
decorator_1
<class 'pyramid.request.Request'> GET /func_1 HTTP/1.0
Host: localhost:80
()
{}
decorator_2
(<pyramid.traversal.DefaultRootFactory object at 0x7f981da2ee48>, <Request at 0x7f981da2ea20 GET http://localhost/func_2>)
{}
Consequently, func_2 crashes because it receives a DefaultRootFactory object it does not expect.
I'd like to understand this discrepancy. How come the signature of the wrapper changes what pyramid passes to the wrapped function?
There is a mechanism at stake I don't understand, and I suspect it might be in Pyramid's logic.
I shared my findings in the webargs issue where this came up, but just in case anyone comes across this here:
Pyramid lets you write a view function with either of these signatures
def view(request):
...
def view(context, request):
...
The second calling convention is the original one, and the first is newer. So even though it is called an "alternate" in the pyramid docs, it is the default.
They use inspect.getfullargspec to see if the view takes a single positional parameter, and if so, wrap it to match the second convention. If the view doesn't match the first convention, it is assumed to match the second convention (which is false in this case).
I am using decorator: deferred_set_context to set the function module, name and time as the context string(basically this context is needed to trace in-case there are multiple calls to register_request)
#deferred_set_context()
def register_request(self, machine_model):
def deferred_set_context(name=None):
def __outer__(func):
context_name = name
if not context_name:
context_name = func.__name__
#defer.inlineCallbacks
def __inner__(*args, **kwargs):
r = None
with base.this_context_if_no_context(base.Context('%s.%s.%s' % (func.__module__, context_name,
datetime.utcnow().strftime(
'%Y-%m-%dT%H:%M:%S.%f')))):
r = yield func(*args, **kwargs)
defer.returnValue(r)
return __inner__
return __outer__
Now I need to set pass the machine_model name(which is argument of register_request) to this decorator: register_request. Something like this:
#deferred_set_context(name=machine_model)
def register_request(self, machine_model):
How shall I do it?
Note that the following two code snippets are equivalent
# first
#decorator
def func():
pass
# second
def func():
pass
func = decorator(func)
You need to make your decorator take a function as an argument. If you want to pass name in as a variable, then it should be a parameter in the function returned by the decorator.
Assuming you don't want to change the behavior of deferred_set_context or register_request, you could make a second shim decorator that spies on the correct function argument and passes it along to your original decorator.
Something like this:
def new_shim_decorator(register_request):
def wrapper(self, machine_model):
return deferred_set_context(machine_model)(self, machine_model)
return wrapper
#new_shim_decorator
def register_request(self, machine_model):
print("my_function called with {}".format(machine_model))
This is kind of a hacky solution in my opinion since this decorator that proxies to the original decorator would need to know the arguments passed to register_request and mirror them.
I would prefer to see a keyword-only argument that get be optionally get-ed from the keyword argument dictionary in either a shim decorator or the original:
def deferred_set_context_by_kwarg(kwarg_name):
def outer(func):
def wrapper(*args, **kwargs):
return deferred_set_context(name=kwargs.get(kwarg_name))(*args, **kwargs)
return wrapper
return outer
#deferred_set_context_by_kwarg('machine_model')
def register_request(self, *, machine_model):
print("my_function called with {}".format(machine_model))
Notice the *, machine_model. That * is a Python 3 feature that requires machine_model to be specified as a keyword argument. This allows our deferred_set_context_by_kwarg decorator to accept a string representing the name of the keyword argument we want and use that name to retrieve the keyword argument value and pass to deferred_set_context upon each function call.
If you are sure your function register_request takes always the same 2 arguments then pick them inside the decorator code.
I recently started learning about Decorators in Python and found the following piece of code which got me confused.
HANDLERS = {}
def handler_for(template):
def decorator(f):
HANDLERS[template] = f
return f
return decorator
def get_template(template, **kwargs):
#This function got me all confused. How does HANDLERS.get(template, None) works and how does it return demo_func() function?
return HANDLERS.get(template, None)
def email(template, **kwargs):
handler = get_template(template, **kwargs)
contents = handler(**kwargs)
print contents
#handler_for('demo')
def demo_func(**kwargs):
#Do something and return the String
if __name__ == "__main__":
email('demo')
I tried debugging with Python Debugger but I'm still getting confused as how the get_template() function calls the decorator which returns the function name.
I tried debugging with Python Debugger but I'm still getting confused as how the get_template() function calls the decorator which returns the function name.
Well… it doesn't. The decorator gets called by you using #handler_for('demo').
Decorators aren't as complicated as they seem, once you keep in mind that the following are equivalent:
#beans
def spam(eggs):
print(eggs)
def spam(eggs):
print(eggs)
spam = beans(spam)
That beans doesn't have to be just an identifier, it can be (almost) any expression, even a function call. So, your code is doing this:
demo_func = handler_for('demo')(demo_func)
handler_for takes 'demo' as the value of its template parameter. It then defines and returns a local function, decorator. So, when it's called, it has access to its own argument, f, and also to the closed-over template value from when it was constructed. That function is immediately called with demo_func as its f value, and it just stashes the function in the HANDLERS dictionary and returns it unchanged. So, inlining everything:
def demo_func(**kwargs):
#Do something and return the String
template = 'demo'
HANDLERS[template] = demo_func
demo_func = demo_func
So, because get_template just looks things up in HANDLERS, when you later call get_template('demo') it will find demo_func.
What's the best way to toggle decorators on and off, without actually going to each decoration and commenting it out? Say you have a benchmarking decorator:
# deco.py
def benchmark(func):
def decorator():
# fancy benchmarking
return decorator
and in your module something like:
# mymodule.py
from deco import benchmark
class foo(object):
#benchmark
def f():
# code
#benchmark
def g():
# more code
That's fine, but sometimes you don't care about the benchmarks and don't want the overhead. I have been doing the following. Add another decorator:
# anothermodule.py
def noop(func):
# do nothing, just return the original function
return func
And then comment out the import line and add another:
# mymodule.py
#from deco import benchmark
from anothermodule import noop as benchmark
Now benchmarks are toggled on a per-file basis, having only to change the import statement in the module in question. Individual decorators can be controlled independently.
Is there a better way to do this? It would be nice to not have to edit the source file at all, and to specify which decorators to use in which files elsewhere.
You could add the conditional to the decorator itself:
def use_benchmark(modname):
return modname == "mymodule"
def benchmark(func):
if not use_benchmark(func.__module__):
return func
def decorator():
# fancy benchmarking
return decorator
If you apply this decorator in mymodule.py, it will be enabled; if you apply it in othermodule.py, it will not be enabled.
I've been using the following approach. It's almost identical to the one suggested by CaptainMurphy, but it has the advantage that you don't need to call the decorator like a function.
import functools
class SwitchedDecorator:
def __init__(self, enabled_func):
self._enabled = False
self._enabled_func = enabled_func
#property
def enabled(self):
return self._enabled
#enabled.setter
def enabled(self, new_value):
if not isinstance(new_value, bool):
raise ValueError("enabled can only be set to a boolean value")
self._enabled = new_value
def __call__(self, target):
if self._enabled:
return self._enabled_func(target)
return target
def deco_func(target):
"""This is the actual decorator function. It's written just like any other decorator."""
def g(*args,**kwargs):
print("your function has been wrapped")
return target(*args,**kwargs)
functools.update_wrapper(g, target)
return g
# This is where we wrap our decorator in the SwitchedDecorator class.
my_decorator = SwitchedDecorator(deco_func)
# Now my_decorator functions just like the deco_func decorator,
# EXCEPT that we can turn it on and off.
my_decorator.enabled=True
#my_decorator
def example1():
print("example1 function")
# we'll now disable my_decorator. Any subsequent uses will not
# actually decorate the target function.
my_decorator.enabled=False
#my_decorator
def example2():
print("example2 function")
In the above, example1 will be decorated, and example2 will NOT be decorated. When I have to enable or disable decorators by module, I just have a function that makes a new SwitchedDecorator whenever I need a different copy.
I think you should use a decorator a to decorate the decorator b, which let you switch the decorator b on or off with the help of a decision function.
This sounds complex, but the idea is rather simple.
So let's say you have a decorator logger:
from functools import wraps
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
This is a very boring decorator and I have a dozen or so of these, cachers, loggers, things which inject stuff, benchmarking etc. I could easily extend it with an if statement, but this seems to be a bad choice; because then I have to change a dozen of decorators, which is not fun at all.
So what to do? Let's step one level higher. Say we have a decorator, which can decorate a decorator? This decorator would look like this:
#point_cut_decorator(logger)
def my_oddly_behaving_function
This decorator accepts logger, which is not a very interesting fact. But it also has enough power to choose if the logger should be applied or not to my_oddly_behaving_function. I called it point_cut_decorator, because it has some aspects of aspect oriented programming. A point cut is a set of locations, where some code (advice) has to be interwoven with the execution flow. The definitions of point cuts is usually in one place. This technique seems to be very similar.
How can we implement it decision logic. Well I have chosen to make a function, which accepts the decoratee, the decorator, file and name, which can only say if a decorator should be applied or not. These are the coordinates, which are good enough to pinpoint the location very precisely.
This is the implementation of point_cut_decorator, I have chosen to implement the decision function as a simple function, you could extend it to let it decide from your settings or configuration, if you use regexes for all 4 coordinates, you will end up with something very powerful:
from functools import wraps
myselector is the decision function, on true a decorator is applied on false it is not applied. Parameters are the filename, the module name, the decorated object and finally the decorator. This allows us to switch of behaviour in a fine grained manner.
def myselector(fname, name, decoratee, decorator):
print fname
if decoratee.__name__ == "test" and fname == "decorated.py" and decorator.__name__ == "logger":
return True
return False
This decorates a function, checks myselector and if myselector says go on, it will apply the decorator to the function.
def point_cut_decorator(d):
def innerdecorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
if myselector(__file__, __name__, f, d):
ps = d(f)
return ps(*args, **kwargs)
else:
return f(*args, **kwargs)
return wrapper
return innerdecorator
def logger(f):
#wraps(f)
def innerdecorator(*args, **kwargs):
print (args, kwargs)
res = f(*args, **kwargs)
print res
return res
return innerdecorator
And this is how you use it:
#point_cut_decorator(logger)
def test(a):
print "hello"
return "world"
test(1)
EDIT:
This is the regular expression approach I talked about:
from functools import wraps
import re
As you can see, I can specify somewhere a couple of rules, which decides a decorator should be applied or not:
rules = [{
"file": "decorated.py",
"module": ".*",
"decoratee": ".*test.*",
"decorator": "logger"
}]
Then I loop over all rules and return True if a rule matches or false if a rule doesn't matches. By making rules empty in production, this will not slow down your application too much:
def myselector(fname, name, decoratee, decorator):
for rule in rules:
file_rule, module_rule, decoratee_rule, decorator_rule = rule["file"], rule["module"], rule["decoratee"], rule["decorator"]
if (
re.match(file_rule, fname)
and re.match(module_rule, name)
and re.match(decoratee_rule, decoratee.__name__)
and re.match(decorator_rule, decorator.__name__)
):
return True
return False
Here is what I finally came up with for per-module toggling. It uses #nneonneo's suggestion as a starting point.
Random modules use decorators as normal, no knowledge of toggling.
foopkg.py:
from toggledeco import benchmark
#benchmark
def foo():
print("function in foopkg")
barpkg.py:
from toggledeco import benchmark
#benchmark
def bar():
print("function in barpkg")
The decorator module itself maintains a set of function references for all decorators that have been disabled, and each decorator checks for its existence in this set. If so, it just returns the raw function (no decorator). By default the set is empty (everything enabled).
toggledeco.py:
import functools
_disabled = set()
def disable(func):
_disabled.add(func)
def enable(func):
_disabled.discard(func)
def benchmark(func):
if benchmark in _disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco
The main program can toggle individual decorators on and off during imports:
from toggledeco import benchmark, disable, enable
disable(benchmark) # no benchmarks...
import foopkg
enable(benchmark) # until they are enabled again
import barpkg
foopkg.foo() # no benchmarking
barpkg.bar() # yes benchmarking
reload(foopkg)
foopkg.foo() # now with benchmarking
Output:
function in foopkg
--> benchmarking bar((),{})
function in barpkg
<-- done
--> benchmarking foo((),{})
function in foopkg
<-- done
This has the added bug/feature that enabling/disabling will trickle down to any submodules imported from modules imported in the main function.
EDIT:
Here's class suggested by #nneonneo. In order to use it, the decorator must be called as a function ( #benchmark(), not #benchmark ).
class benchmark:
disabled = False
#classmethod
def enable(cls):
cls.disabled = False
#classmethod
def disable(cls):
cls.disabled = True
def __call__(cls,func):
if cls.disabled:
return func
#functools.wraps(func)
def deco(*args,**kwargs):
print("--> benchmarking %s(%s,%s)" % (func.__name__,args,kwargs))
ret = func(*args,**kwargs)
print("<-- done")
return deco
I would implement a check for a config file inside the decorator's body. If benchmark has to be used according to the config file, then I would go to your current decorator's body. If not, I would return the function and do nothing more. Something in this flavor:
# deco.py
def benchmark(func):
if config == 'dontUseDecorators': # no use of decorator
# do nothing
return func
def decorator(): # else call decorator
# fancy benchmarking
return decorator
What happens when calling a decorated function ? # in
#benchmark
def f():
# body comes here
is syntactic sugar for this
f = benchmark(f)
so if config wants you to overlook decorator, you are just doing f = f() which is what you expect.
I don't think anyone has suggested this yet:
benchmark_modules = set('mod1', 'mod2') # Load this from a config file
def benchmark(func):
if not func.__module__ in benchmark_modules:
return func
def decorator():
# fancy benchmarking
return decorator
Each function or method has a __module__ attribute that is the name of the module where the function is defined. Create a whitelist (or blacklist if you prefer) of modules where benchmarking is to occur, and if you don't want to benchmark that module just return the original undecorated function.
another straight way:
# mymodule.py
from deco import benchmark
class foo(object):
def f():
# code
if <config.use_benchmark>:
f = benchmark(f)
def g():
# more code
if <config.use_benchmark>:
g = benchmark(g)
Here's a workaround to automatically toggle a decorator (here: #profile used by line_profiler):
if 'profile' not in __builtins__ or type(__builtins__) is not dict: profile=lambda x: None;
More info
This conditional (only if needed) instantiation of the profile variable (as an empty lambda function) prevents raising NameError when trying to import our module with user-defined functions where the decorator #profile is applied to every profiled user function. If I ever want to use the decorator for profiling - it will still work, not being overwritten (already existing in an external script kernprof that contains this decorator).