Develop Reference

python decorator takes 1 positional argument but 5 were given after modifying for pytest

I'd appreciate some help with the following code, as I'm still relatively new to Python, and despite countless days trying to figure out where i'm going wrong, i cant seem to spot the error i'm making.
I've adapted the following code from an article on medium to create a logging decorator and then enhanced it to try and "redact pandas df and dictionary" from the logs. Using functools caused me a problem with pytest and pytest fixtures. A post on stack overflow suggested dropping functools in favour of decorators.
def log_decorator(_func=None):
def log_decorator_info(func):
def log_decorator_wrapper(*args, **kwargs):
_logger = Logger()
logger_obj = _logger.get_logger()
args_passed_in_function = args_excl_df_dict(*args)
kwargs_passed_in_function = kwargs_excl_df_dict(**kwargs)
formatted_arguments = join_args_kwargs(args_passed_in_function,kwargs_passed_in_function)
py_file_caller = getframeinfo(stack()[1][0])
extra_args = { 'func_name_override': func.__name__,'file_name_override': os.path.basename(py_file_caller.filename) }
""" Before to the function execution, log function details."""
logger_obj.info(f"Begin function - Arguments: {formatted_arguments}", extra=extra_args)
try:
""" log return value from the function """
args_returned_from_function = args_excl_df_dict(func(*args))
kwargs_returned_from_function = []
formatted_arguments = join_args_kwargs(args_returned_from_function,kwargs_returned_from_function)
logger_obj.info(f"End function - Returned: {formatted_arguments}", extra=extra_args)
except:
"""log exception if occurs in function"""
error_raised = str(sys.exc_info()[1])
logger_obj.error(f"Exception: {str(sys.exc_info()[1])}",extra=extra_args)
msg_to_send = f"{func.__name__} {error_raised}"
send_alert(APP_NAME,msg_to_send,'error')
raise
return func(*args, **kwargs)
return decorator.decorator(log_decorator_wrapper, func)
if _func is None:
return log_decorator_info
else:
return log_decorator_info(_func)
Having adapted the above code i cant figure out what is causing the following error
args_returned_from_function = args_excl_df_dict(func(*args))
TypeError: test_me() takes 4 positional arguments but 5 were given
Other functions which the log decorator relies on
def args_excl_df_dict(*args):
args_list = []
for a in args:
if isinstance(a,(pd.DataFrame,dict)):
a = 'redacted from log'
args_list.append(repr(a))
else:
args_list.append(repr(a))
return args_list
def kwargs_excl_df_dict(**kwargs):
kwargs_list = []
for k, v in kwargs.items():
if isinstance(v,(dict,pd.DataFrame)):
v = 'redacted from log'
kwargs_list.append(f"{k}={v!r}")
else:
kwargs_list.append(f"{k}={v!r}")
return kwargs_list
def join_args_kwargs(args,kwargs):
formatted_arguments = ", ".join(args + kwargs)
return str(formatted_arguments)
This is the code calling the decorator
#log_decorator.log_decorator()
def test_me(a, b, c, d):
return a, b
test_me(string, number, dictionary, pandas_df)

I think the problem is that the wrapper is including the function as an argument to the function.
Try adding this line and see if it helps
args = args[1:]
intor your log_decorator_wrapper function towards the top. Like this.
def log_decorator(_func=None):
def log_decorator_info(func):
def log_decorator_wrapper(*args, **kwargs):
args = args[1:] # < -------------------here
_logger = Logger()
logger_obj = _logger.get_logger()
args_passed_in_function = args_excl_df_dict(*args)
kwargs_passed_in_function = kwargs_excl_df_dict(**kwargs)
formatted_arguments = join_args_kwargs(args_passed_in_function,kwargs_passed_in_function)
py_file_caller = getframeinfo(stack()[1][0])
extra_args = { 'func_name_override': func.__name__,'file_name_override': os.path.basename(py_file_caller.filename) }
""" Before to the function execution, log function details."""
logger_obj.info(f"Begin function - Arguments: {formatted_arguments}", extra=extra_args)
try:
""" log return value from the function """
args_returned_from_function = args_excl_df_dict(func(*args))
kwargs_returned_from_function = []
formatted_arguments = join_args_kwargs(args_returned_from_function,kwargs_returned_from_function)
logger_obj.info(f"End function - Returned: {formatted_arguments}", extra=extra_args)
except:
"""log exception if occurs in function"""
error_raised = str(sys.exc_info()[1])
logger_obj.error(f"Exception: {str(sys.exc_info()[1])}",extra=extra_args)
msg_to_send = f"{func.__name__} {error_raised}"
send_alert(APP_NAME,msg_to_send,'error')
raise
return func(*args, **kwargs)
return decorator.decorator(log_decorator_wrapper, func)
if _func is None:
return log_decorator_info
else:
return log_decorator_info(_func)

If your code is as is in your editor, maybe look at the indentation on the first three functions. Then start from there to move down

How to write python decorator that updates keyword argument?

The goal is to write a decorator that updates one keyword argument of the wrapped function.
In the following code wrapper attempts to update kwarg1:
import inspect [0/300678]
from functools import wraps
def override_me(arg, kwarg1="default kwarg1", kwarg2="default kwarg2"):
print(f"override_me {arg} kwarg1={kwarg1} kwarg2={kwarg2}")
def append_kwarg1(func):
original_kwarg1_default = (
inspect.signature(func).parameters["kwarg1"].default
)
#wraps(func)
def wrapper(*args, kwarg1=original_kwarg1_default, **kwargs):
func(*args, kwarg1=kwarg1 + "_patched!", **kwargs)
return wrapper
override_me = append_kwarg1(override_me)
override_me("passed_arg")
override_me("passed_arg", kwarg1="passed_kwarg1_named")
override_me("passed_arg", "passed_kwarg1_as_arg") # TypeError: override_me() got multiple values for argument 'kwarg1'
This, however, fails when kwarg1 is passed as a positional argument.
Edit:
Clarifications as pointed in comments: override_me signature cannot be changed (think: external module).

A working solution based on the fact that even a keyword argument has an index. Then depending on the length of args passed by caller we can determine if the argument of interest has been passed positionally or by keyword and update it either in args or kwargs:
import inspect
from functools import wraps
def override_me(arg, kwarg1="default kwarg1", kwarg2="default kwarg2"):
print(f"override_me {arg} kwarg1={kwarg1} kwarg2={kwarg2}")
def append_kwarg1(func):
params = inspect.signature(func).parameters
kwarg1_index = next(
x[0] for x in zip(range(len(params)), params.items()) if x[1][0] == "kwarg1"
)
def update(v):
return v + "_patched!"
#wraps(func)
def wrapper(*args, **kwargs):
if len(args) > kwarg1_index:
args = (
args[:kwarg1_index]
+ (update(args[kwarg1_index]),)
+ args[kwarg1_index + 1 :]
)
func(*args, **kwargs)
else:
kwargs["kwarg1"] = update(kwargs.get("kwarg1", params["kwarg1"].default))
func(*args, **kwargs)
return wrapper
override_me = append_kwarg1(override_me)
override_me("passed_arg", kwarg1="passed_kwarg1_named")
override_me("passed_arg")
override_me("passed_arg", "passed_kwarg1_as_arg")
override_me("passed_arg", "passed_kwarg1_as_arg", "passed_kwarg2_as_arg")

How to obtain tuple values from inside of decorator functions in Python

I am exploring decorator functions in Python. My objective is to return a tuple from a wrapper function inside of the decorator function, when the original function that is passed to the decorator functin also returns a tuple. My code snippet is pasted below:
def decorator_function(original_function):
def wrapper_function(*args, **kwargs):
s, o = original_function(*args, **kwargs)
return s, o
return wrapper_function
def test_function(name, command):
status = True
output = dict()
output['message'] = command + " " + name
return status, output
decorator_func_var = decorator_function(test_function("Kaushik", "Hello"))
ok, out = decorator_func_var()
print(ok)
print(out)
However, when I execute this, I get an error message as follows:
I am curious to know where I am going wrong with my code snippet and how can I obtain the tuple values when calling a decorated function. I would really appreciate any suggestions or feedback.

Your decorator takes a function as an argument, and returns a function. You just want
ok, out = decorator_function(test_function)("Kaushik", "Hello")
or a little bit more clearly:
wrapped_function = decorator_function(test_function)
ok, out = wrapped_function("Kaushik", "Hello")

def decorator_function(original_function):
def wrapper_function(*args, **kwargs):
s, o = original_function(*args, **kwargs)
return s, o
return wrapper_function
def test_function(name, command):
status = True
output = dict()
output['message'] = command + " " + name
return status, output
decorator_func_var = decorator_function(test_function)
ok, out = decorator_func_var("Kaushik", "Hello")
print(ok)
print(out)
You pass the parameters after initialising decorator_func_var

one decorator for 3 different functions

I am writing python API and I have one problem.
I have 3 different functions:
func1() -> return only text
func2(name) -> return text only but takes parameter
func3(name) -> this function create a file "name".txt
Now I have a problem with decorator, I want to create a log decorator that is called everytime function is called.
Problem is that I dont know how to simply do it, I know how to create it with no param or one param but I have no idea hot to create universal decorator that will work for all three functions.
Now i have something like this:
def log(func):
def wrapper(name):
func(name)
log = ('write something here')
f = open('log.txt', 'a+')
f.write(log + "\n")
f.close(name)
return wrapper

Your wrapper should accept an arbitrary number of arguments, with the *args and **kwargs syntax to capture both positional and keyword arguments. Make sure to return whatever the wrapped function returns:
def log(func):
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
log = ('write something here')
with open('log.txt', 'a+') as f:
f.write(log + "\n")
return result
return wrapper
You probably want to add in the #functools.wraps decorator; this copies across any documentation and other metadata from the original wrapped function to the new wrapper:
from functools import wraps
def log(func):
#wraps(func)
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
log = ('write something here')
with open('log.txt', 'a+') as f:
f.write(log + "\n")
return result
return wrapper
Last but not least, rather than reopening a log file yourself, take a look at the logging module to handle log files for you.

def log(func):
def wrapper(*args, **kwds):
log = func(*args, **kwds)
f = open('log.txt', 'a+')
f.write(log + "\n")
f.close()
return wrapper
#log
def func1():
return "Called function 1"
#log
def func2(name):
return "Called function 2 with " + name
#log
def func3(name):
f = open('name.txt', 'a+')
f.write(name + " from func3\n")
f.close()
return "Called function 3 with " + name
def main():
func1()
func2("func2")
func3("func3")
if __name__ == '__main__':
main()
Log.txt becomes:
Called function 1
Called function 2 with func2
Called function 3 with func3

getting sphinx to recognise correct signature

I've been trying to get my documentation in order for an open source project I'm working on, which involves a mirrored client and server API. To this end I have created a decorator that can most of the time be used to document a method that simply performs validation on its input. You can find a class full of these methods here and the decorator's implementation here.
The decorator, as you can see uses functools.wraps to preserve the docstring, and I thought also the signature, however the source code vs the generated documentation looks like this:
Source:
vs
Docs:
Does anyone know any way to have setH's generated documentation show the correct call signature? (without having a new decorator for each signature - there are hudreds of methods I need to mirror)
I've found a workaround which involved having the decorator not changing the unbound method, but having the class mutate the method at binding time (object instantiation) - this seems like a hack though, so any comments on this, or alternative ways of doing this, would be appreciated.

In PRAW, I handled this issue by having conditional decorators that return the original function (rather than the decorated function) when a sphinx build is occurring.
In PRAW's sphinx conf.py I added the following as a way to determine if SPHINX is currently building:
import os
os.environ['SPHINX_BUILD'] = '1'
And then in PRAW, its decorators look like:
import os
# Don't decorate functions when building the documentation
IS_SPHINX_BUILD = bool(os.getenv('SPHINX_BUILD'))
def limit_chars(function):
"""Truncate the string returned from a function and return the result."""
#wraps(function)
def wrapped(self, *args, **kwargs):
output_string = function(self, *args, **kwargs)
if len(output_string) > MAX_CHARS:
output_string = output_string[:MAX_CHARS - 3] + '...'
return output_string
return function if IS_SPHINX_BUILD else wrapped
The return function if IS_SPHINX_BUILD else wrapped line is what allows SPHINX to pick up the correct signature.
Relevant Source

I'd like to avoid reliance on too muck outside of the standard library, so while I have looked at the Decorator module, I have mainly tried to reproduce its functionality.... Unsuccessfully...
So I took a look at the problem from another angle, and now I have a partially working solution, which can mainly be described by just looking at this commit. It's not perfect as it relies on using partial, which clobbers the help in the REPL. The idea is that instead of replacing the function to which the decorator is applied, it is augmented with attributes.
+def s_repr(obj):
+ """ :param obj: object """
+ return (repr(obj) if not isinstance(obj, SikuliClass)
+ else "self._get_jython_object(%r)" % obj._str_get)
+
+
def run_on_remote(func):
...
- func.s_repr = lambda obj: (repr(obj)
- if not isinstance(obj, SikuliClass) else
- "self._get_jython_object(%r)" % obj._str_get)
-
- def _inner(self, *args):
- return self.remote._eval("self._get_jython_object(%r).%s(%s)" % (
- self._id,
- func.__name__,
- ', '.join([func.s_repr(x) for x in args])))
-
- func.func = _inner
+ gjo = "self._get_jython_object"
+ func._augment = {
+ 'inner': lambda self, *args: (self.remote._eval("%s(%r).%s(%s)"
+ % (gjo, self._id, func.__name__,
+ ', '.join([s_repr(x)for x in args]))))
+ }
#wraps(func)
def _outer(self, *args, **kwargs):
func(self, *args, **kwargs)
- if hasattr(func, "arg"):
- args, kwargs = func.arg(*args, **kwargs), {}
- result = func.func(*args, **kwargs)
- if hasattr(func, "post"):
+ if "arg" in func._augment:
+ args, kwargs = func._augment["arg"](self, *args, **kwargs), {}
+ result = func._augment['inner'](self, *args, **kwargs)
+ if "post" in func._augment:
return func.post(result)
else:
return result
def _arg(arg_func):
- func.arg = arg_func
- return _outer
+ func._augment['arg'] = arg_func
+ return func
def _post(post_func):
- func.post = post_func
- return _outer
+ func._augment['post'] = post_func
+ return func
def _func(func_func):
- func.func = func_func
- return _outer
- _outer.arg = _arg
- _outer.post = _post
- _outer.func = _func
- return _outer
+ func._augment['inner'] = func_func
+ return func
+
+ func.arg = _outer.arg = _arg
+ func.post = _outer.post = _post
+ func.func = _outer.func = _func
+ func.run = _outer.run = _outer
+ return func
So this doesn't actually change the unbound method, ergo the generated documentation stays the same. The second part of the trickery occurs at class initialisation.
class ClientSikuliClass(ServerSikuliClass):
""" Base class for types based on the Sikuli native types """
...
def __init__(self, remote, server_id, *args, **kwargs):
"""
:type server_id: int
:type remote: SikuliClient
"""
super(ClientSikuliClass, self).__init__(None)
+ for key in dir(self):
+ try:
+ func = getattr(self, key)
+ except AttributeError:
+ pass
+ else:
+ try:
+ from functools import partial, wraps
+ run = wraps(func.run)(partial(func.run, self))
+ setattr(self, key, run)
+ except AttributeError:
+ pass
self.remote = remote
self.server_id = server_id
So at the point where an instance of any class inheriting ClientSikuliClass is instantiated, an attempt is made to take the run property of each attribute of that instance and make that what is returned on attempting to get that attribute, and so the bound method
is now a partially applied _outer function.
So the issues with this are multiple:
Using partial at initilaisation results in losing the bound method information.
I worry about clobbering attributes that just so happen to have a run attribute...
So while I have an answer to my own question, I'm not quite satisfied by it.
Update
Ok so after a bit more work I ended up with this:
class ClientSikuliClass(ServerSikuliClass):
""" Base class for types based on the Sikuli native types """
...
def __init__(self, remote, server_id, *args, **kwargs):
"""
:type server_id: int
:type remote: SikuliClient
"""
super(ClientSikuliClass, self).__init__(None)
- for key in dir(self):
+
+ def _apply_key(key):
try:
func = getattr(self, key)
+ aug = func._augment
+ runner = func.run
except AttributeError:
- pass
- else:
- try:
- from functools import partial, wraps
- run = wraps(func.run)(partial(func.run, self))
- setattr(self, key, run)
- except AttributeError:
- pass
+ return
+
+ #wraps(func)
+ def _outer(*args, **kwargs):
+ return runner(self, *args, **kwargs)
+
+ setattr(self, key, _outer)
+
+ for key in dir(self):
+ _apply_key(key)
+
self.remote = remote
self.server_id = server_id
This prevents the loss of the documentation on the object. You'll also see that the func._augment attribute is accessed, even though it is not used, so that if it does not exist the object attribute will not be touched.
I'd be interested if anyone had any comments on this?

functools.wraps only preserves __name__,__doc__, and __module__. To preserve the signature as well take a look at Michele Simionato's Decorator module.

To expand my short comment on Ethan's answer, here is my original code using the functools package:
import functools
def trace(f):
"""The trace decorator."""
logger = ... # some code to determine the right logger
where = ... # some code to create a string describing where we are
#functools.wraps(f)
def _trace(*args, **kwargs):
logger.debug("Entering %s", where)
result = f(*args, **kwargs)
logger.debug("Leaving %s", where)
return result
return _trace
and here the code using the decorator package:
import decorator
def trace(f):
"""The trace decorator."""
logger = ... # some code to determine the right logger
where = ... # some code to create a string describing where we are
def _trace(f, *args, **kwargs):
logger.debug("Entering %s", where)
result = f(*args, **kwargs)
logger.debug("Leaving %s", where)
return result
return decorator.decorate(f, _trace)
We wanted to move the code to determine the right logger and where-string out of the actual function wrapper, for performance reasons. Hence the approach with the nested wrapper function, in both versions.
Both versions of the code work on Python 2 and Python 3, but the second version creates the correct prototypes for the decorated functions when using Sphinx & autodoc (without having to repeat the prototype in the autodoc statements, as suggested in this answer).
This is with cPython, I did not try Jython etc.