I am trying to implement a callback system in Python that is similar to how JavaScript can have different numbers of parameters in its callbacks. Ideally, I want to achieve this without using *args or **kwargs in the parameters of my callbacks.
My Goal
What I want is something that looks roughly like this:
def callback1(val):
print(val)
def callback2(x, y):
print(x, y)
def callback3(a, b, c):
print(a, b, c)
def foo(callback):
callback(1, 2, 3) # Always has 3 arguments to pass
foo(callback1) # Fails. Should print "1"
foo(callback2) # Fails. Should print "1 2"
foo(callback3) # Ok. Prints "1 2 3"
Perhaps a more verbose way of putting it would be:
# num_params() method isn't real (that I know of), but this is an
# inelegant example of how the callbacks might work if it were
def foo2(callback):
if num_params(callback) == 1:
callback(1)
elif num_params(callback) == 2:
callback(1, 2)
elif num_params(callback) == 3:
callback(1, 2, 3)
What I Don't Want
I don't want to use *args or **kwargs in each callback (unless this isn't possible any other way) like the following:
# This is just SO ugly
def callback1(*args):
print(args[0])
def callback2(*args):
print(args[0], args[1])
def callback3(*args):
print(args[0], args[1], args[2])
JavaScript Equivalent
This is relatively common in JavaScript. For example, one can supply the callback of a .forEach() function with 1, 2, or 3 arguments:
let myArray = [1, 2, 3, 4]
// Valid
myArray.forEach((element) => {
// Do stuff with the element only
});
// Valid
myArray.forEach((element, index) => {
// Do stuff with the element AND the index
});
// Valid
myArray.forEach((element, index, array) => {
// Do stuff with the element, index and the whole array
});
However, despite my best efforts in Google searching, I have no idea how to implement this in Python (or even in JavaScript for that matter, but that's beside the point; I hope this doesn't come back to bite me).
I would very much like to know if this is possible in Python and/or what the proper term is for this coding technique.
What's wrong with args and kwargs? It is the pythonic way to do that. Python is not JavaScript. If you do not like accessing args by indexes like args[0], args[1], etc, you could just define some args like usual, and rest (unused args) - in *args:
def callback1(a, *args):
print(a)
def callback2(a, b, *args):
print(a, b)
def callback3(a, b, c):
print(a, b, c)
Also you can unpack them in the function:
def callback1(*args):
a, *rest = args
print(a)
It makes it more verbose inside, but same definition for all callbacks.
Also it's common to name variables, you are not going to use with _ (underscore) instead of args, rest, etc.:
def callback1(a, *_):
print(a)
def callback1(*args):
a, *_ = args
print(a)
You can define all your callback functions using the same number of arguments, i.e.:
def callback1(val, b=None, c=None):
print(val)
def callback2(x, y, c=None):
print(x, y)
def callback3(a, b, c):
print(a, b, c)
Alternatively you can unpack *args within functions:
def callback1(*args):
val, _, _ = args
print(val)
def callback2(*args):
x, y, _ = args
print(x, y)
def callback3(*args):
a, b, c = args
print(a, b, c)
Finally, you can get creative using functools.partial.
Related
I have a class & method, each with several arguments: my_class(a,b,c).my_method(d,e,f) and I'd like to be able to only change a single argument, while holding the others constant.
Constantly copy-pasting the other constant arguments seems bad, so I'd like to create a new object wrapper_fct where I reference my_class but only provide the one argument I want to change, b, without always having to specify the remaining arguments. How would wrapper_fct() look like?
For example, wrapper_fct(my_class, b1) would return my_class(a,b1,c).my_method(d,e,f), wrapper_fct(my_class, b2) would return my_class(a,b2,c).my_method(d,e,f).
Here's an example in practice:
Loop through just the variable b and evaluate several classes/methods for each new instance of b, and append the results in a list.
I can currently do this in a for loop:
mylist1 = [] # init lists (append results here)
mylist2 = []
mylist2 = []
for b in [1,2,3,4,5]:
mylist1.append( my_class1(a,b,c).my_method(d,e,f) )
mylist2.append( my_class2(a,b,c).my_method(d,e,f) )
mylist3.append( my_class3(a,b,c).my_method(d,e,f) )
...
But it seems better to create a function loop_through_B() and use the wrapper_fct(my_class,b) as specified above. Not sure if it's the ideal solution, but maybe something like:
def loop_through_B(input_class, b_values = [1,2,3,4,5])
mylist = []
for b in b_values:
mylist.append( wrapper_fct(input_class,b) )
return mylist
loop_through_B(my_class1) # would I also have to specify the method here as well?
loop_through_B(my_class2)
loop_through_B(my_class3)
Extra Question: how would I add the ability to vary method arguments, or even multiple class & method arguments?
After #chepner pointed me in the right direction, I think the best solution is to use the lambda function:
wrapper_fct = lambda b: my_class1(a,b,c).my_method(d,e,f)
In this case, I can vary b as much as I want while holding the class arguments a,c, and method arguments d,e,f constant. Note that with lambda functions, I can also vary the method arguments and/or the class arguments. For example:
wrapper_fct_multiple = lambda b, e: my_class1(a,b,c).my_method(d,e,f)
It is also possible to do this with functools.partial, but it's not obvious to me how I would specify both class & method arguments with functools.
Anyway, here is the solution implementation using lambda:
# define the "wrapper function" outside the loop
wrapper_fct = lambda b: my_class1(a,b,c).my_method(d,e,f)
# define the function I want to use to loop through B:
def loop_through_B(class_wrapper, b_values)
mylist = []
for b in b_values:
mylist.append( class_wrapper(b) )
return mylist
# run:
loop_through_B(wrapper_fct, b_values=[1,2,3,4,5])
# Can make additional wrapper_fct2, wrapper_fct3, for my_class2, my_class3 ...
You can pass the method a dictionary of arguments, and change what the method sees by selectively updating it when calling the method.
Here's what I mean:
class MyClass:
def __init__(self, a, b, c):
self.a, self.b, self.c = a, b, c
def my_method(self, kwargs):
return sum((kwargs[key] for key in kwargs.keys()))
def __repr__(self):
classname = type(self).__name__
args = ', '.join((f'{v!r}' for v in (self.a, self.b, self.c)))
return f'{classname}({args})'
instance = MyClass('a','b','c')
print(instance) # -> MyClass('a', 'b', 'c')
kwargs = dict(d=1, e=2, f=3)
print(instance.my_method(kwargs)) # -> 6
print(instance.my_method(dict(kwargs, e=38))) # -> 42
Mock version of the problem
For a function
def f(a,b,c):
return a+b+c
The function
def fix(func, **kwargs):
fa = kwargs.get('a')
fb = kwargs.get('b')
if fa is not None and fb is not None:
def f(*args):
return func(a=fa, b=fb, c=args[0])
elif fa is not None:
def f(*args):
return func(a=fa, b=args[0], c=args[1])
elif fb is not None:
def f(*args):
return func(a=args[0],b=fb, c=args[1])
else:
def f(*args):
return func(args)
return f
allows to obtain a new function by fixing some of the parameters of func.
For example: fix(g, b=3) would give us a function like
def fixed_b_in_g(a,c):
return g(a,3,c)
Question: I would like to see if there is some trick to use fix in such a way that produces a function like
def fix_a_equal_b_in_g(a,c):
return g(a,a,c)
Concrete problem
The function scipy.stats.rv_continuous.fit allows to fit parameters of a distribution to an input sample. It allows to input some keyword arguments (like fix above does) to tell it to keep some of the parameters fixed to values that the user inputs. Internally scipy.stats.rv_continuous.fit has a function, scipy.stats.rv_continuous._reduce_func, that does more or less what dix does (better implemented than my fix for example).
In my case, rather than fixing some parameters to values, I would like to fit to keep two parameters (say a and b) equal to each other, but still free during the fitting.
We can use this function to copy a keyword argument whose name is base_kwarg_name to added_kwarg_name:
def with_copied_kwargs(func, added_kwarg_names_by_base):
def fixed_func(*args, **base_kwargs):
added_kwargs = {
added_kwarg_name: base_kwargs[base_kwarg_name]
for base_kwarg_name, added_kwarg_name in added_kwarg_names_by_base.items()
}
return func(*args, **base_kwargs, **added_kwargs)
return fixed_func
Given:
def add(*, a, b, c):
return a + b + c
then modified_add = with_copied_kwargs(add, {"b": "c"}) is equivalent to:
def modified_add(*, a, b):
return add(a=a, b=b, c=b)
with_copied_kwargs can then be used along with functools.partial to both both copy keyword arguments and provide values incrementally. modified_add = functools.partial(with_copied_kwargs(add, {"b": "c"}), a=1) is equivalent to:
def modified_add(*, b):
return add(a=1, b=b, c=b)
Note that I add * (see PEP 3102) before all parameters in functions I then apply with_copied_kwargs to because the minute people start using positional arguments, things would get messy. So better to restrict it to keyword-only arguments.
Is it possible to define a closure for a function which is already defined?
For example I'd like to have a "raw" function and a function which already has some predefined values set by a surrounding closure.
Here is some code showing what I can do with a closure to add predefined variables to a function definition:
def outer(a, b, c):
def fun(d):
print(a + b + c - d)
return fun
foo = outer(4, 5, 6)
foo(10)
Now I want to have a definition of fun outside of a wrapping closure function, to be able to call fun either with variables from a closure or by passing variables directly. I know that I need to redefine a function to make it usable in a closure, thus I tried using lambda for it:
def fun(a, b, c, d): # raw function
print(a + b + c - d)
def clsr(func): # make a "closure" decorator
def wrap(*args):
return lambda *args: func(*args)
return wrap
foo = clsr(fun)(5, 6, 7) # make a closure with values already defined
foo(10) # raises TypeError: fun() missing 3 required positional arguments: 'a', 'b', and 'c'
fun(5, 6, 7, 10) # prints 8
What I also tried is using wraps from functools, but I was not able to make it work.
But is this even possible? And if yes: Is there any module which already implements decorators for this?
You can just define the wrap on the fly:
def fun(a, b, c, d): # raw function
print(a + b + c - d)
def closed(d): fun(5,6,7,d)
closed(10)
You can use this with lambda, but #juanpa points out you should not if there is no reason to. The above code will result in 8. This method by the way is not Python specific, most languages would support this.
But if you need a closure in a sense that it relies on the wrapper variables, than no, and there is good reason not to. This will create essentially a non-working function, that relies on wrapping. In this case using a class maybe better:
class fun:
def __init__(self,*args): #Can use specific things, not just *args.
self.args = args #Or meaningful names
def __call__(self,a, b, c, d): # raw function
print(a + b + c - d,self.args)
def closed(d):
fun("some",3,"more",['args'])(5,6,7,d)
closed(10)
or using *args/**kwargs directly and passing extra variables through that. Otherwise I am not familiar with a "inner function" construct that only works after wrapping.
Is there an equivalent to R's do.call in python?
do.call(what = 'sum', args = list(1:10)) #[1] 55
do.call(what = 'mean', args = list(1:10)) #[1] 5.5
?do.call
# Description
# do.call constructs and executes a function call from a name or a function and a list of arguments to be passed to it.
There is no built-in for this, but it is easy enough to construct an equivalent.
You can look up any object from the built-ins namespace using the __builtin__ (Python 2) or builtins (Python 3) modules then apply arbitrary arguments to that with *args and **kwargs syntax:
try:
# Python 2
import __builtin__ as builtins
except ImportError:
# Python 3
import builtins
def do_call(what, *args, **kwargs):
return getattr(builtins, what)(*args, **kwargs)
do_call('sum', range(1, 11))
Generally speaking, we don't do this in Python. If you must translate strings into function objects, it is generally preferred to build a custom dictionary:
functions = {
'sum': sum,
'mean': lambda v: sum(v) / len(v),
}
then look up functions from that dictionary instead:
functions['sum'](range(1, 11))
This lets you strictly control what names are available to dynamic code, preventing a user from making a nuisance of themselves by calling built-ins for their destructive or disruptive effects.
do.call is pretty much the equivalent of the splat operator in Python:
def mysum(a, b, c):
return sum([a, b, c])
# normal call:
mysum(1, 2, 3)
# with a list of arguments:
mysum(*[1, 2, 3])
Note that I’ve had to define my own sum function since Python’s sum already expects a list as an argument, so your original code would just be
sum(range(1, 11))
R has another peculiarity: do.call internally performs a function lookup of its first argument. This means that it finds the function even if it’s a character string rather than an actual function. The Python equivalent above doesn’t do this — see Martijn’s answer for a solution to this.
Goes similar to previous answer, but why so complicated?
def do_call(what, args=[], kwargs = {}):
return what(*args, **kwargs)
(Which is more elegant than my previously posted definition:)
def do_call(which, args=None, kwargs = None):
if args is None and kwargs is not None:
return which(**kwargs)
elif args is not None and kwargs is None:
return which(*args)
else:
return which(*args, **kwargs)
Python's sum is different than R's sum (1 argument a list expected vs.
arbitraily many arguments expected in R). So we define our own sum (mysum)
which behaves similarly to R's sum. In a similar way we define mymean.
def mysum(*args):
return sum(args)
def mymean(*args):
return sum(args)/len(args)
Now we can recreate your example in Python - as a reasonable 1:1 translation of the R function call.
do_call(what = mymean, args=[1, 2, 3])
## 2.0
do_call(what = mysum, args=[1, 2, 3])
## 6
For functions with argument names, we use a dict for kwargs, where the parameter
names are keys of the dictionary (as strings) and their values the values.
def myfunc(a, b, c):
return a + b + c
do_call(what = myfunc, kwargs={"a": 1, "b": 2, "c": 3})
## 6
# we can even mix named and unnamed parts
do_call(what = myfunc, args = [1, 2], kwargs={"c": 3})
## 6
How can I write a decorator so that decorated functions can accept (and ignore) arbitrary arguments?
I have some functions like this:
def foo(x):
return x
def foo2(x, y):
if bar(y):
return x
else:
return x + 1
def foo3(x, y, z):
...
foo() can calculate the return value for a given x just based on x, but foo2() needs another parameter, and foo3() needs a third parameter. I have a method elsewhere that, among other things, calls either foo(), foo2(), etc depending on a user-specified argument.
Right now, the method just grabs the appropriate function with getattr(user_arg) and calls it with all of x, y, z. To avoid a TypeError for the wrong number of arguments, the foo functions are all defined with *args like this:
def foo(x, *args):
return x
But I'd like to just have a decorator to avoid including *args in every single foo function definition. Is there a way to do that? Or can you suggest a better way to organize this code?
One way would be to write the method like this:
if user_arg == 'foo':
foo(x)
elif user_arg == 'foo2':
foo2(x, y)
elif user_arg == 'foo3':
foo3(x, y, z)
But I'd really like to avoid this, because there are a lot of foo functions, and because I'd like to make it possible to add new foo functions without also having to add another branch to the method.
Edit: To clarify, it's not the case that I need to call a different foo function based on the number of arguments. It's arbitrary (user-specified) which of the foo functions is called.
def foo3(x, y, z):
return x + y + z
def foo4(x, y, z):
return x + y - z
You can use inspect.getargspec to determine what arguments the decorated function takes:
import functools
import inspect
def ignore_extra_arguments(func):
args, varargs, kwvarargs, defaults = inspect.getargspec(func)
#functools.wraps(func)
def wrapper_func(*wrapper_args, **wrapper_kwargs):
if varargs is None:
# remove extra positional arguments
wrapper_args = wrapper_args[:len(args)]
if kwvarargs is None:
# remove extra keyword arguments
wrapper_kwargs = {k: v for k, v in wrapper_kwargs.iteritems() if k in args}
return func(*wrapper_args, **wrapper_kwargs)
return wrapper_func
This can be optimized a little by lifting the if checks out of wrapper_func, at the expense of writing more code.
You can use named arguments and always call foo() (the first function) in order to dispatch to the "right" function, as follows:
def foo(a, b=None, c=None, d=None):
if b:
return foo2(a, b, c, d)
else:
# do your thing
def foo2(a, b, c=None, d=None):
if c:
return foo3(a, b, c, d)
else:
# do your thing
def foo3(a, b, c, d=None):
...
Another option:
def foo(a, *b):
if b:
return foo2(a, *b)
else:
print "in foo"
# do your thing
def foo2(a, b, *c):
if c:
return foo3(a, b, *c)
else:
print "in foo2"
# do your thing
def foo3(a, b, c, *d):
print "in foo3"
foo(1, 2, 3)