If I have to wrap an existing method, let us say wrapee() from a new method, say wrapper(), and the wrapee() provides default values for some arguments, how do I preserve its semantics without introducing unnecessary dependencies and maintenance? Let us say, the goal is to be able to use wrapper() in place of wrapee() without having to change the client code. E.g., if wrapee() is defined as:
def wrapee(param1, param2="Some Value"):
# Do something
Then, one way to define wrapper() is:
def wrapper(param1, param2="Some Value"):
# Do something
wrapee(param1, param2)
# Do something else.
However, wrapper() has to make assumptions on the default value for param2 which I don't like. If I have the control on wrapee(), I would define it like this:
def wrapee(param1, param2=None):
param2 = param2 or "Some Value"
# Do something
Then, wrapper() would change to:
def wrapper(param1, param2=None):
# Do something
wrapee(param1, param2)
# Do something else.
If I don't have control on how wrapee() is defined, how best to define wrapper()? One option that comes into mind is to use to create a dict with non-None arguments and pass it as dictionary arguments, but it seems unnecessarily tedious.
Update:
The solution is to use both the list and dictionary arguments like this:
def wrapper(param1, *args, **argv):
# Do something
wrapee(param1, *args, **argv)
# Do something else.
All the following calls are then valid:
wrapper('test1')
wrapper('test1', 'test2')
wrapper('test1', param2='test2')
wrapper(param2='test2', param1='test1')
Check out argument lists in the Python docs.
>>> def wrapper(param1, *stuff, **kargs):
... print(param1)
... print(stuff)
... print(args)
...
>>> wrapper(3, 4, 5, foo=2)
3
(4, 5)
{'foo': 2}
Then to pass the args along:
wrapee(param1, *stuff, **kargs)
The *stuff is a variable number of non-named arguments, and the **kargs is a variable number of named arguments.
I'd hardly say that it isn't tedious, but the only approach that I can think of is to introspect the function that you are wrapping to determine if any of its parameters have default values. You can get the list of parameters and then determine which one is the first that has default values:
from inspect import getargspec
method_signature = getargspec(method)
param_names = method_signature[0]
default_values = method_signature[3]
params = []
# If any of method's parameters has default values, we need
# to know the index of the first one that does.
param_with_default_loc = -1
if default_values is not None and len(default_values) > 0:
param_slice_index = len(default_values) * -1
param_with_default = param_names[param_slice_index:][0]
param_with_default_loc = param_names.index(param_with_default)
At that point, you can iterate over param_names, copying into the dict that is passed to wrappee. Once your index >= param_with_default_loc, you can obtain the default values by looking in the default_values list with an index of your index - param_with_default_loc.
Does that make any sesne?
Of course, to make this generic, you would to define it as a wrapper function, adding yet another layer of wrapping.
def wrapper(param1, param2=None):
if param2:
wrapee(param1, param2)
else:
wrapee(param1)
is this what you want?
#!/usr/bin/python
from functools import wraps
def my_decorator(f):
#wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
def f1(x, y):
print x, y
def f2(x, y="ok"):
print x, y
my_decorator(f1)(1,2)
my_decorator(f2)(1,2)
my_decorator(f2)(1)
adapted from http://koala/doc/python2.6-doc/html/library/functools.html#module-functools
Related
I'm newbie in Python, but the second time I encouter this problem.
Problem:
In some libraries there are functions with arguments. Sometimes there is argument as function, like this:
def somefun(fun):
x = [1,2,3]
z = fun(x)
return z
And I want to pass there some other function like this:
def func(x,y):
return x*y
which have more than one argument. I want to make one argument static, so somefun except func as argument.
Finally I want to make some kind of cycle where I can change static arg.
Something like this:
for i in xrange(1,9):
somefun(func(i,*))
Please do not offer me to change any functions. They are from library and it's not very comfortable to change them.
Thanks a lot!
You can use lambda statement:
somefun(lambda x: func(i, x))
It sure sounds like you are looking for functools.partial. From the docs:
functools.partial(func, *args, **keywords)
Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords.
In your example, you could pass partial(func, 10) as the argument to somefun. Or you could create the partial objects and use them in a loop:
for i in xrange(1,9):
somefun(partial(func, i))
My solution with decorator
from functools import wraps
import numpy as np
def p_decorate(f):
#wraps(f)
def wrapped(*args):
z = f(*args)
return z
return wrapped
#p_decorate
def myfunc(a,b):
"""My new function"""
z = np.dot(a,b)
return z
x = [1,2,3]
y = [4,2,0]
r = myfunc(x,y)
print (r)
print (myfunc.__name__)
print (myfunc.__doc__)
You can change myfunc as you wish.You can also insert more function layers.Without the use of this decorator factory,you would lose the name of myfunc and the docstring.
I am trying to create a set of functions in python that will all do a similar operation on a set of inputs. All of the functions have one input parameter fixed and half of them also need a second parameter. For the sake of simplicity, below is a toy example with only two functions.
Now, I want, in my script, to run the appropriate function, depending on what the user input as a number. Here, the user is the random function (so the minimum example works). What I want to do is something like this:
def function_1(*args):
return args[0]
def function_2(*args):
return args[0] * args[1]
x = 10
y = 20
i = random.randint(1,2)
f = function_1 if i==1 else function_2
return_value = f(x,y)
And it works, but it seems messy to me. I would rather have function_1 defined as
def function_1(x):
return x
Another way would be to define
def function_1(x,y):
return x
But that leaves me with a dangling y parameter.
but that will not work as easily. Is my way the "proper" way of solving my problem or does there exist a better way?
There are couple of approaches here, all of them adding more boiler-plate code.
There is also this PEP which may be interesting to you.
But 'pythonic' way of doing it is not as elegant as usual function overloading due to the fact that functions are just class attributes.
So you can either go with function like that:
def foo(*args):
and then count how many args you've got which will be very broad but very flexible as well.
another approach is the default arguments:
def foo(first, second=None, third=None)
less flexible but easier to predict, and then lastly you can also use:
def foo(anything)
and detect the type of anything in your function acting accordingly.
Your monkey-patching example can work too, but it becomes more complex if you use it with class methods, and does make introspection tricky.
EDIT: Also, for your case you may want to keep the functions separate and write single 'dispatcher' function that will call appropriate function for you depending on the arguments, which is probably best solution considering above.
EDIT2: base on your comments I believe that following approach may work for you
def weigh_dispatcher(*args, **kwargs):
#decide which function to call base on args
if 'somethingspecial' in kwargs:
return weight2(*args, **kwargs)
def weight_prep(arg):
#common part here
def weight1(arg1, arg2):
weitht_prep(arg1)
#rest of the func
def weight2(arg1, arg2, arg3):
weitht_prep(arg1)
#rest of the func
alternatively you can move the common part into the dispatcher
You may also have a function with optional second argument:
def function_1(x, y = None):
if y != None:
return x + y
else:
return x
Here's the sample run:
>>> function_1(3)
3
>>> function_1(3, 4)
7
Or even optional multiple arguments! Check this out:
def function_2(x, *args):
return x + sum(args)
And the sample run:
>>> function_2(3)
3
>>> function_2(3, 4)
7
>>> function_2(3, 4, 5, 6, 7)
25
You may here refer to args as to list:
def function_3(x, *args):
if len(args) < 1:
return x
else:
return x + sum(args)
And the sample run:
>>> function_3(1,2,3,4,5)
15
Let me first acknowledge that what I want to do may be considered anything from silly to evil, but I want to find out if I can do it in Python anyway.
Let's say I have a function decorator that takes keyword arguments defining variables, and I want to access those variables in the wrapped function. I might do something like this:
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
return f(extras, *args, **kwargs)
return wrapped
return wrapper
Now I can do something like:
#more_vars(a='hello', b='world')
def test(deco_vars, x, y):
print(deco_vars['a'], deco_vars['b'])
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
The thing I don't like about this is that when you use this decorator, you have to change the call signature of the function, adding the extra variable in addition to slapping on the decorator. Also, if you look at the help for the function, you see an extra variable that you're not expected to use when calling the function:
help(test)
# Output:
# Help on function test in module __main__:
#
# test(deco_vars, x, y)
This makes it look like the user is expected to call the function with 3 parameters, but obviously that won't work. So you'd have to also add a message to the docstring indicating that the first parameter isn't part of the interface, it's just an implementation detail and should be ignored. That's kind of crappy, though. Is there any way to do this without hanging these variables on something in the global scope? Ideally, I'd like it to look like the following:
#more_vars(a='hello', b='world')
def test(x, y):
print(a, b)
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
help(test)
# Output:
# Help on function test in module __main__:
#
# test(x, y)
I am content with a Python 3 only solution if one exists.
You could do this with some trickery that inserts the variables passed to the decorator into the function's local variables:
import sys
from functools import wraps
from types import FunctionType
def is_python3():
return sys.version_info >= (3, 0)
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
fn_globals = {}
fn_globals.update(globals())
fn_globals.update(extras)
if is_python3():
func_code = '__code__'
else:
func_code = 'func_code'
call_fn = FunctionType(getattr(f, func_code), fn_globals)
return call_fn(*args, **kwargs)
return wrapped
return wrapper
#more_vars(a="hello", b="world")
def test(x, y):
print("locals: {}".format(locals()))
print("x: {}".format(x))
print("y: {}".format(y))
print("a: {}".format(a))
print("b: {}".format(b))
if __name__ == "__main__":
test(1, 2)
Can you do this? Sure! Should you do this? Probably not!
(Code available here.)
EDIT: answer edited for readability. Latest answer is on top, original follows.
If I understand well
you want the new arguments to be defined as keywords in the #more_vars decorator
you want to use them in the decorated function
and you want them to be hidden to the normal users (the exposed signature should still be the normal signature)
Have a look at the #with_partial decorator in my library makefun. It provides this functionality out of the box:
from makefun import with_partial
#with_partial(a='hello', b='world')
def test(a, b, x, y):
"""Here is a doc"""
print(a, b)
print(x, y)
It yields the expected output and the docstring is modified accordingly:
test(1, 2)
help(test)
yields
hello world
1 2
Help on function test in module <...>:
test(x, y)
<This function is equivalent to 'test(x, y, a=hello, b=world)', see original 'test' doc below.>
Here is a doc
To answer the question in your comment, the function creation strategy in makefun is exactly the same than the one in the famous decorator library: compile + exec. No magic here, but decorator has been using this trick for years in real-world applications so it is quite solid. See def _make in the source code.
Note that the makefun library also provides a partial(f, *args, **kwargs) function if you want to create the decorator yourself for some reason (see below for inspiration).
If you wish to do this manually, this is a solution that should work as you expect, it relies on the wraps function provided by makefun, to modify the exposed signature.
from makefun import wraps, remove_signature_parameters
def more_vars(**extras):
def wrapper(f):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
# inject the invisible args again
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
return wrapper
You can test that it works:
#more_vars(a='hello', b='world')
def test(x, y, invisible_args):
a = invisible_args['a']
b = invisible_args['b']
print(a, b)
print(x, y)
test(1, 2)
help(test)
You can even make the decorator definition more compact if you use decopatch to remove the useless level of nesting:
from decopatch import DECORATED
from makefun import wraps, remove_signature_parameters
#function_decorator
def more_vars(f=DECORATED, **extras):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
Finally, if you rather do not want to depend on any external library, the most pythonic way to do it is to create a function factory (but then you cannot have this as a decorator):
def make_test(a, b, name=None):
def test(x, y):
print(a, b)
print(x, y)
if name is not None:
test.__name__ = name
return test
test = make_test(a='hello', b='world')
test2 = make_test(a='hello', b='there', name='test2')
I'm the author of makefun and decopatch by the way ;)
It sounds like your only problem is that help is showing the signature of the raw test as the signature of the wrapped function, and you don't want it to.
The only reason that's happening is that wraps (or, rather, update_wrapper, which wraps calls) explicitly copies this from the wrappee to the wrapper.
You can decide exactly what you do and don't want to copy. If what you want to do differently is simple enough, it's just a matter of filtering stuff out of the default WRAPPER_ASSIGNMENTS and WRAPPER_UPDATES. If you want to change other stuff, you may need to fork update_wrapper and use your own version—but functools is one of those modules that has a link to the source right at the top of the docs, because it's meant to be used as readable sample code.
In your case, it may just be a matter of wraps(f, updated=[]), or you may want to do something fancy, like use inspect.signature to get the signature of f, and modify it to remove the first parameter, and build a wrapper explicitly around that to fool even the inspect module.
I've found a solution to this problem, although the solution is by most standards almost certainly worse than the problem itself. With some clever rewriting of the decorated function's bytecode, you can redirect all references to variables of a given name to a new closure you can dynamically create for the function. This solution only works for the standard CPython, and I have only tested it with 3.7.
import inspect
from dis import opmap, Bytecode
from types import FunctionType, CodeType
def more_vars(**vars):
'''Decorator to inject more variables into a function.'''
def wrapper(f):
code = f.__code__
new_freevars = code.co_freevars + tuple(vars.keys())
new_globals = [var for var in code.co_names if var not in vars.keys()]
new_locals = [var for var in code.co_varnames if var not in vars.keys()]
payload = b''.join(
filtered_bytecode(f, new_freevars, new_globals, new_locals))
new_code = CodeType(code.co_argcount,
code.co_kwonlyargcount,
len(new_locals),
code.co_stacksize,
code.co_flags & ~inspect.CO_NOFREE,
payload,
code.co_consts,
tuple(new_globals),
tuple(new_locals),
code.co_filename,
code.co_name,
code.co_firstlineno,
code.co_lnotab,
code.co_freevars + tuple(vars.keys()),
code.co_cellvars)
closure = tuple(get_cell(v) for (k, v) in vars.items())
return FunctionType(new_code, f.__globals__, f.__name__, f.__defaults__,
(f.__closure__ or ()) + closure)
return wrapper
def get_cell(val=None):
'''Create a closure cell object with initial value.'''
# If you know a better way to do this, I'd like to hear it.
x = val
def closure():
return x # pragma: no cover
return closure.__closure__[0]
def filtered_bytecode(func, freevars, globals, locals):
'''Get the bytecode for a function with adjusted closed variables
Any references to globlas or locals in the bytecode which exist in the
freevars are modified to reference the freevars instead.
'''
opcode_map = {
opmap['LOAD_FAST']: opmap['LOAD_DEREF'],
opmap['STORE_FAST']: opmap['STORE_DEREF'],
opmap['LOAD_GLOBAL']: opmap['LOAD_DEREF'],
opmap['STORE_GLOBAL']: opmap['STORE_DEREF']
}
freevars_map = {var: idx for (idx, var) in enumerate(freevars)}
globals_map = {var: idx for (idx, var) in enumerate(globals)}
locals_map = {var: idx for (idx, var) in enumerate(locals)}
for instruction in Bytecode(func):
if instruction.opcode not in opcode_map:
yield bytes([instruction.opcode, instruction.arg or 0])
elif instruction.argval in freevars_map:
yield bytes([opcode_map[instruction.opcode],
freevars_map[instruction.argval]])
elif 'GLOBAL' in instruction.opname:
yield bytes([instruction.opcode,
globals_map[instruction.argval]])
elif 'FAST' in instruction.opname:
yield bytes([instruction.opcode,
locals_map[instruction.argval]])
This behaves exactly as I wanted:
In [1]: #more_vars(a='hello', b='world')
...: def test(x, y):
...: print(a, b)
...: print(x, y)
...:
In [2]: test(1, 2)
hello world
1 2
In [3]: help(test)
Help on function test in module __main__:
test(x, y)
This is almost certainly not ready for production use. I would be surprised if there weren't edge cases that behave unexpectedly, and possibly even segfault. I'd probably file this under the "educational curiosity" heading.
Is there a way to forward function arguments without hiding the fact that the original call did or did not provide optional arguments?
def func1(a=x):
# do stuff
def func2(b=y):
# pass args to func1 without masking func1 defaults
return func1(?)
A call to func2() should result in func1() being called without arguments or at least with its default arguments, whatever they may be.
The following almost works but fundamentally I don't know if there is a way for func2 to determine if its defaults were invoked on not.
def func2(b=y):
# this comes close but what if func2(y) is called?
if b == y:
return func1()
else:
return func1(b)
The usual way of determining if a parameter is left off is to use None as the default. It's unlikely that you'll be calling a function with None so it's a useful marker.
def func2(b=None):
if b is None:
return func1()
else:
return func1(b)
I suspect the right way to do this is to have your func2 function use a sentinel value as its default argument, so you can recognize it easily. If you get that sentinel, you can set up the arguments you'll pass on to func1 however you want (e.g. not passing any argument). You can use argument unpacking to handle passing a variable number of arguments (such as 0-1).
A common sentinel is None, though if that could be a meaningful value for a caller to pass, you may want to use something else (an instance of object is a common choice). Here's an example:
def func1(a="default value"): # lets assume we don't know what this default is
# do stuff with a
# later, perhaps in a different module
_sentinel = object() # our sentinel object
def func2(b=_sentinel):
if b is _sentinel: # test for the sentinel
b = "some useful value"
a_args = () # arguments to func1 is an empty tuple
else:
a_args = (b,) # pack b into a 1-tuple
# do stuff with b perhaps
func1(*a_args) # call func1 with appropriate arguments (either b or nothing)
Note that this design is relatively silly. Most of the time you'll either call func1 with an argument in all cases, or you'll call it without an argument in all cases. You rarely need to conditionally pass an argument like this.
See this answer:
https://stackoverflow.com/a/2088101/933416
There is no way to get the information you want from the internals. To detect whether defaults were used, you would need to re-implement the internal default argument processing within the function, i.e.:
def func2(*args, **kwargs):
if len(args) == 0 and "b" not in kwargs:
b = y
return func1()
else:
return func1(b)
Now from the first check we guarantee that func2() was called as opposed to func2(y) or func2(b=y). In almost every case, the unique object sentinel is good enough to avoid having to truly guarantee how it was called, but it can be done.
But judging from the fact that you immediately return the result of func1, I see no reason why func2 even has default arguments. In the default call (func2()), that y is never used. So why is it there? Why don't you just use define func2(*a, **k) and pass them directly to func1?
Argument forwarding should be done with variadic arguments:
def func2(*args, **kwargs):
func1(*args, **kwargs)
Everything will just work, although introspection can suffer a bit.
If you need to sometimes not pass on an argument, you can remove an argument whenever:
del kwargs["name"]
An example:
def print_wrapper(*args, extrabig=False, **kwargs):
if extrabig:
args = [arg*2 for arg in args]
kwargs["sep"] = kwargs.get("sep", " ") * 2
print(*args, **kwargs)
print_wrapper(2, 4, 8, end="!!!\n")
#>>> 2 4 8!!!
print_wrapper(2, 4, 8, sep=", ", end="!!!\n")
#>>> 2, 4, 8!!!
print_wrapper(2, 4, 8, extrabig=True, end="!!!\n")
#>>> 4 8 16!!!
If you really don't want to do this (although you'd be wrong), you can use object to generate a unique sentinel.
# Bad! Won't let you print None
def optionally_print_one_thing(thing=None):
if thing is not None:
print(thing)
# Better
_no_argument = object()
def optionally_print_one_thing(thing=_no_argument):
if thing is not _no_argument:
print(thing)
What is your exact use case? func2 should be smart enough to only pass on the appropriate params to func1, and that should rely on the default values of any parameters.
The only time I have ever found it necessary to change how func2 calls func1 is when func1 is a c function with a screwy signature:
def func2(this, that, those=None):
if those is None:
return func1(this, that)
else:
return func1(this, that, those)
I need to keep track of the number of times each function in a collection has been called. If a function is called more than x times within n seconds, my program needs to pause, after which the count for that function is reset.
My functions calls might look something like this:
a(1)
b(1,param2=2, param3=3)
c(1,param3=3)
My best idea is to have a wrapper function keep track of all of the limits. Something like
def wrapper(function, function_params,x,n):
if not hasattr(wrapper, "function_dict"):
wrapper.function_dict = {}
if function not in wrapper.function_dict.keys():
wrapper.function_dict[function] = {
remaining = x, expires = time.time() + n
}
remaining = wrapper.function_dict[function]['remaining']
expires = wrapper.function_dict[function]['expires']
if remaining == 0:
time.sleep(expires - time.time())
wrapper.function_dict[function] = {
remaining = x, expires = time.time() + n
}
results = ????? # call function, this is what I don't know how to do
wrapper.function_dict[function]['remaining'] -= 1
My question is, how to I handle the parameters for the functions? I'm not sure how exactly to account for the fact that there might be a variable number of parameters, and that some might be named. For example, the function definition for c might be:
def c(param1,param2=2, param3=3):
return param1 + param2 + param3
But I might need to call it with only param1 and param3.
Do I have the right general approach? This feels like something I could accomplish with the ** operator, but I'm stuck on how exactly to proceed.
Write a decorator, and use a splat operator to handle arbitrary arguments.
Example:
def pause_wrapper(x, n):
def decorator(f):
config = [x, time.time()+n]
def wrapped(*args, **kwargs):
if config[0] == 0:
time.sleep(config[1] - time.time())
config = [x, time.time() + n]
return f(*args, **kwargs)
return wrapped
return decorator
and usage:
#pause_wrapper(x, n)
def function(a, b, c):
...
The *args and **kwargs are informally called "splat" arguments. A function that takes *args, **kwargs receives all positional parameters in the tuple args and all keyword arguments in the dictionary kwargs. (You can have other arguments besides the splats, in which case the splats soak up all arguments not sent to named arguments).
Passing *args and **kwargs has the opposite effect, passing the contents of args as extra positional parameters, and kwargs as keyword parameters.
Using both allows you to handle any set of arguments, in or out, letting you do transparent wrapping (like this example).
this is basically what decorators were made for
from collections import defaultdict
class counted:
calls = defaultdict(int)
def __init__(self,x,n):
self.x = x
self.n = n
def __call__(self,fn,*args,**kwargs):
results = fn(*args,**kwargs)
calls[fn.__name__] += 1
#do something with the count ...
#counted(3,9)
def functionWhatever(arg1,arg2,arg3,**kwargs):
return "55"
functionWhatever(1,2,3,something=5)