I've got some trouble dealing with defining a user friendly function interface when passing two keyworded arguments with the same key.
Question
What is the best way to make it possible to call a function where two keyworded arguments have the same key and the second keyworded argument has precedence?
If this problem occurs, the first keyworded argument always stems from an unzipped database in a dict, while the second keyworded argument is always passed by giving it "directly" as a keyworded argument.
The database dictionary values must not be overwritten in the outer scopy of the functions, since they may be used multiple times.
edit: To keep up the usability of the function for the user, a backend-implementation is preferred. This means that the user can simply pass arguments to the function without the use of additional modules, while the function itself does all the magic.
Problem
I've got a function, called fun_one here, which receives a multitude of arguments defined directly by the user of my program. This may be length and width of a heat exchanger for example. To ease the use of the function and make the calling code as short as possible, the use of databases is encouraged. These databases contain the data in a dict (or in a pandas Series), in this case called inputs.
To pass the database-dict inputs to the function, it is unzipped with **inputs and thus passed as keyworded arguments.
Now if the user wants to overwrite a specific argument of the database, my understanding of a user-friendly approach would be to just let him pass the preceded argument again, for example with length=23.7, and internally overwrite the argument from the database. But of course (see example code) this raises the error before I can even enter the function where I could try/except:
TypeError: fun_one() got multiple values for keyword argument 'length'
Code example reproducing the error
def fun_one(*args, **kwargs): # short example function
print(kwargs)
inputs = {'length': 15.8, 'width': 1.1, 'some_other_args': np.random.rand(3)}
fun_one(**inputs, length=23.7)
My current solution
My current solution fun_two involves not unzipping the database and passing it to *args. It checks *args for dicts and sets values which are not yet in kwargs to kwargs, as shown in the code example below.
def fun_two(*args, **kwargs): # example function printing kwargs
print(kwargs) # print kwargs before applying changes
for arg in args: # find dicts
if type(arg) is dict:
for key, val in arg.items(): # loop over dict
_ = kwargs.setdefault(key, val) # set val if key not in dict
print(kwargs) # print kwargs after applying changes
inputs = {'length': 15.8, 'width': 1.1, 'some_other_args': np.random.rand(3)}
fun_two(inputs, length=23.7)
But this approach is imho quite obscure for the user and requires looping and checking at the beginning of quite alot functions, since this will apply to numerous functions. (I'll outsource it to a module, so it is one line per function. But it still deviates from my understanding of an easy and clear function definition).
Is there any better (more Pythonic) way to do this? Did I oversee some way to do it in the process of calling the function? Performance does not matter.
Thanks in advance!
Easiest solution is using ChainMap from collections (manual pages). That way you can chose which arguments have precedence. Example:
from collections import ChainMap
def fun_one(*args, **kwargs): # short example function
print(kwargs)
inputs = {'length': 15.8, 'width': 1.1, 'some_other_args': 1}
c = ChainMap({'length': 23.7}, inputs) # we overwrite length here
fun_one(**c)
Outputs:
{'some_other_args': 1, 'width': 1.1, 'length': 23.7}
If we call fun_one just with inputs:
c = ChainMap(inputs)
# or c = inputs
fun_one(**c)
Output will be:
{'width': 1.1, 'length': 15.8, 'some_other_args': 1}
From manual:
A ChainMap groups multiple dicts or other mappings together to create
a single, updateable view. If no maps are specified, a single empty
dictionary is provided so that a new chain always has at least one
mapping.
You can wrap this ChainMap in decorator, one change is that don't call fun_one() with **input, only input:
from collections import ChainMap
def function_with_many_arguments(func):
orig_func = func
def _f(*args, **kwargs):
if args:
c = ChainMap(kwargs, args[0])
return orig_func(**c)
else:
return orig_func(*args, **kwargs)
return _f
#function_with_many_arguments
def fun_one(*args, **kwargs): # short example function
print(kwargs)
inputs = {'length': 15.8, 'width': 1.1, 'some_other_args': 1}
fun_one(inputs, length=23)
Prints:
{'some_other_args': 1, 'length': 23, 'width': 1.1}
As extension to Andrej Kesely's answer (thanks again!), I added a loop of ChainMaps to enable the use of multiple databases in the same function and to be able to use all kinds of positional arguments. The precedence of the multiple databases is first-come-first-served, but this is ok in this case. This is the decorator:
def function_with_many_arguments(func):
orig_func = func
def _f(*args, **kwargs):
if args:
c = ChainMap(kwargs)
for arg in args:
if type(arg) is dict:
c = ChainMap(c, arg)
orig_func(*args, **c)
else:
orig_func(*args, **kwargs)
return _f
And here is my extended example function with some code to test it. I just added all kind of random arguments, not considering any Pythonic way to do it... ;)
#function_with_many_arguments
def fun_one(a, b, *args, name, database=None, **kwargs):
print(name)
print(a, b)
print(kwargs)
inputs = {'length': 15.8, 'width': 1.1, 'some_other_args': np.random.rand(3)}
inputs2 = inputs.copy()
inputs2['width'] = 123
inputs2['weight'] = 3.8
fun_one(4, 8, inputs, database=inputs2, name='abc', length=23.8, weight=55)
Related
In python we can do this:
def myFun1(one = '1', two = '2'):
...
Then we can call the function and pass the arguments by their name:
myFun1(two = 'two', one = 'one')
Also, we can do this:
def myFun2(**kwargs):
print kwargs.get('one', 'nothing here')
myFun2(one='one')
So I was wondering if it is possible to combine both methods like:
def myFun3(name, lname, **other_info):
...
myFun3(lname='Someone', name='myName', city='cityName', otherInfo='blah')
In general what combinations can we do?
Thanks and sorry for my silly question.
The general idea is:
def func(arg1, arg2, ..., kwarg1=default, kwarg2=default, ..., *args, **kwargs):
...
You can use as many of those as you want. The * and ** will 'soak up' any remaining values not otherwise accounted for.
Positional arguments (provided without defaults) can't be given by keyword, and non-default arguments can't follow default arguments.
Note Python 3 also adds the ability to specify keyword-only arguments by having them after *:
def func(arg1, arg2, *args, kwonlyarg=default):
...
You can also use * alone (def func(a1, a2, *, kw=d):) which means that no arguments are captured, but anything after is keyword-only.
So, if you are in 3.x, you could produce the behaviour you want with:
def myFun3(*, name, lname, **other_info):
...
Which would allow calling with name and lname as keyword-only.
Note this is an unusual interface, which may be annoying to the user - I would only use it in very specific use cases.
In 2.x, you would need to manually make this by parsing **kwargs.
You can add your named arguments along with kwargs. If the keys are available in the calling function It will taken to your named argument otherwise it will be taken by the kwargs dictionary.
def add(a=1, b=2,**c):
res = a+b
for items in c:
res = res + c[items]
print(res)
add(2,3)
5
add(b=4, a =3)
7
add(a =1,b=2,c=3,d=4)
10
It's possible at least for Python 2.7. Keyword arguments get assigned to positional parameters by name, so you can do
In [34]: def func(name, lname, **kwargs):
print 'name='+name, 'lname='+lname
print kwargs
....:
In [35]: func(lname='lname_val', name='name_val', city='cityName', otherInfo='blah')
name=name_val lname=lname_val
{'city': 'cityName', 'otherInfo': 'blah'}
Official docs state it that way:
"If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). If the slot is already filled, a TypeError exception is raised. Otherwise, the value of the argument is placed in the slot, filling it (even if the expression is None, it fills the slot)."
https://docs.python.org/2/reference/expressions.html#calls
I know it's a common case for people to give a function an arbitrary number of kwargs with **kwargs, and then access them as a dictionary; however, I want to explicitly specify my functions kwargs, but still be able to access them as a dictionary.
This is because I want my function to only receive specific kwargs, but I need to perform an identical operation with all of them, which I can put into a for loop.
def my_func(kwarg1=None, kwarg2=None, kwarg3=None):
kwargs = {} # After somehow getting all my kwargs into a dictionary
for k in kwargs:
# Do my operation
I do not want my function to receive an arbitrary number of kwargs, but I do want to access my kwargs in a dictionary.
Assuming you have no positional arguments, you could get access to your kwargs via locals if you put it at the top of your function:
def my_func(kwarg1=None, kwarg2=None, kwarg3=None):
# Don't add any more variables before the next line!
kwargs = dict(locals())
for k in kwargs:
# Do my operation
This is hacky (at best) and it's probably better to just spell it out:
kwargs = {'kwarg1': kwarg1, ...}
This is Python3.3+ code that creates the list of keyword argument names automatically. Just for completness. I would prefer any of the simpler solutions.
import inspect
def my_func(*, kwarg1=None, kwarg2=None, kwarg3=None):
local_vars = locals()
kwargs = {k: local_vars[k] for k in KWARGS_my_func}
print(kwargs)
KWARGS_my_func = [p.name for p in inspect.signature(my_func).parameters.values()
if p.kind == inspect.Parameter.KEYWORD_ONLY]
my_func(kwarg2=2)
Simply create a dictionary as normal, retrieving the values of each argument.
def my_func(kwarg1=None, kwarg2=None, kwarg3=None):
kwargs = {'kwarg1':kwarg1, 'kwarg2':kwarg2, 'kwarg3':kwarg3}
for k in kwargs:
# Do my operation
I am wondering if it is possible to list the variables expected by a Python function, prior to calling it, in order to pass the expected variables from a bigger dict containing a lot of variables.
I have searched the net but couldn't find anything. However, the python interpreter can show the list of expected variables, so there surely must be some way to do it in a script?
You can use either the inspect.signature() or inspect.getfullargspec() functions:
import inspect
argspec = inspect.getfullargspec(somefunction)
signature = inspect.signature(somefunction)
inspect.fullargspec returns a named tuple with 7 elements:
A list with the argument names
The name of the catchall *args parameter, if defined (None otherwise)
The name of the catchall **kwargs parameter, if defined (None otherwise)
A tuple with default values for the keyword arguments; they go with the last elements of the arguments; match these by length of the default values tuple.
A list of keyword-only parameter names
A dictionary of default values for the keyword-only parameter names, if any
and a dictionary containing the annotations
With inspect.signature() you get a Signature object, a rich object that models not only the above data as a more structured set of objects but also lets you bind values to parameters the same way a call to the function would.
Which one is better will depend on your use cases.
Demo:
>>> import inspect
>>> def foo(bar, baz, spam='eggs', *monty, python: "kwonly", spanish=42, **inquisition) -> "return annotation":
... pass
...
>>> inspect.getfullargspec(foo)
FullArgSpec(args=['bar', 'baz', 'spam'], varargs='monty', varkw='inquisition', defaults=('eggs',), kwonlyargs=['python', 'spanish'], kwonlydefaults={'spanish': 42}, annotations={'return': 'return annotation', 'python': 'kwonly'})
>>> signature = inspect.signature(foo)
>>> signature
<Signature (bar, baz, spam='eggs', *monty, python: 'kwonly', spanish=42, **inquisition) -> 'return annotation'>
>>> signature.parameters['python'].kind.description
'keyword-only'
>>> signature.bind('Eric', 'Idle', 'John', python='Cleese')
<BoundArguments (bar='Eric', baz='Idle', spam='John', python='Cleese')>
If you have a dictionary named values of possible parameter values, I'd use inspect.signature() and use the Signature.parameters mapping to match names:
posargs = [
values[param.name]
for param in signature.parameters.values()
if param.kind is Parameter.POSITIONAL_ONLY
]
skip_kinds = {Parameter.POSITIONAL_ONLY, Parameter.VAR_POSITIONAL, Parameter.VAR_KEYWORD}
kwargs = {
param.name: values[param.name]
for param in signature.parameters.values()
if param.name in values and param.kind not in skip_kinds
}
The above gives you a list of values for the positional-only parameters, and a dictionary for the rest (excepting any *args or **kwargs parameters).
Just as a side answer, I now use another approach to pass to functions the variables they expect: I pass them all.
What I mean is that I maintain a kind of global/shared dictionnary of variables in my root object (which is the parent of all other objects), eg:
shareddict = {'A': 0, 'B':'somestring'}
Then I simply pass this dict to any method of any other object that is to be called, just like this:
shareddict.update(call_to_func(**shareddict))
As you can see, we unpack all the keys/values in shareddict as keyword arguments to call_to_func(). We also update shareddict with the returned result, we'll see below why.
Now with this technic, I can simply and clearly define in my functions/methods if I need one or several variables from this dict:
my_method1(A=None, *args, **kwargs):
''' This method only computes on A '''
new_A = Do_some_stuff(A)
return {'A': new_A} # Return the new A in a dictionary to update the shared value of A in the shareddict
my_method2(B=None, *args, **kwargs):
''' This method only computes on B '''
new_B = Do_some_stuff(B)
return {'B': new_B} # Return the new B in a dictionary to update the shareddict
my_method3(A=None, B=None, *args, **kwargs):
''' This method swaps A and B, and then create a new variable C '''
return {'A': B, 'B': A, 'C': 'a_new_variable'} # Here we will update both A and B and create the new variable C
As you can notice, all the methods above return a dict of variables, which will update the shareddict, and which will get passed along to other functions.
This technic has several advantages:
Quite simple to implement
Elegant way to maintain a shared list of variables but without using a global variable
Functions and methods clearly show in their definitions what they expect (but of course one caveat is that even mandatory variables will need to be set as a keyword argument with a default value such as None, which usually means that the variable is optional, but here it's not
The methods are inheritable and overloadable
Low memory footprint since the same shareddict is passed all along
The children functions/methods define what they need (bottom-up), instead of the root defining what arguments will be passed to children (top-down)
Very easy to create/update variables
Optionally, it's VERY easy to dump all those variables in a file, eg by using json.dumps(finaldict, sort_keys=True).
Nice and easy:
import inspect #library to import
def foo(bar, baz, spam='eggs', *monty, **python): pass #example function
argspec = inspect.signature(foo)
print(argspec) #print your output
prints: (bar, baz, spam='eggs', *monty, **python)
It also works for methods inside classes (very useful!):
class Complex: #example Class
def __init__(self, realpart, imagpart): #method inside Class
... self.r = realpart
... self.i = imagpart
argspec = inspect.signature(Complex)
print(argspec)
prints: (realpart, imagpart)
In python we can do this:
def myFun1(one = '1', two = '2'):
...
Then we can call the function and pass the arguments by their name:
myFun1(two = 'two', one = 'one')
Also, we can do this:
def myFun2(**kwargs):
print kwargs.get('one', 'nothing here')
myFun2(one='one')
So I was wondering if it is possible to combine both methods like:
def myFun3(name, lname, **other_info):
...
myFun3(lname='Someone', name='myName', city='cityName', otherInfo='blah')
In general what combinations can we do?
Thanks and sorry for my silly question.
The general idea is:
def func(arg1, arg2, ..., kwarg1=default, kwarg2=default, ..., *args, **kwargs):
...
You can use as many of those as you want. The * and ** will 'soak up' any remaining values not otherwise accounted for.
Positional arguments (provided without defaults) can't be given by keyword, and non-default arguments can't follow default arguments.
Note Python 3 also adds the ability to specify keyword-only arguments by having them after *:
def func(arg1, arg2, *args, kwonlyarg=default):
...
You can also use * alone (def func(a1, a2, *, kw=d):) which means that no arguments are captured, but anything after is keyword-only.
So, if you are in 3.x, you could produce the behaviour you want with:
def myFun3(*, name, lname, **other_info):
...
Which would allow calling with name and lname as keyword-only.
Note this is an unusual interface, which may be annoying to the user - I would only use it in very specific use cases.
In 2.x, you would need to manually make this by parsing **kwargs.
You can add your named arguments along with kwargs. If the keys are available in the calling function It will taken to your named argument otherwise it will be taken by the kwargs dictionary.
def add(a=1, b=2,**c):
res = a+b
for items in c:
res = res + c[items]
print(res)
add(2,3)
5
add(b=4, a =3)
7
add(a =1,b=2,c=3,d=4)
10
It's possible at least for Python 2.7. Keyword arguments get assigned to positional parameters by name, so you can do
In [34]: def func(name, lname, **kwargs):
print 'name='+name, 'lname='+lname
print kwargs
....:
In [35]: func(lname='lname_val', name='name_val', city='cityName', otherInfo='blah')
name=name_val lname=lname_val
{'city': 'cityName', 'otherInfo': 'blah'}
Official docs state it that way:
"If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). If the slot is already filled, a TypeError exception is raised. Otherwise, the value of the argument is placed in the slot, filling it (even if the expression is None, it fills the slot)."
https://docs.python.org/2/reference/expressions.html#calls
I have created a class MyClassthat contains a lot of simulation data. The class groups simulation results for different simulations that have a similar structure. The results can be retreived with a MyClass.get(foo) method. It returns a dictionary with simulationID/array pairs, array being the value of foo for each simulation.
Now I want to implement a method in my class to apply any function to all the arrays for foo. It should return a dictionary with simulationID/function(foo) pairs.
For a function that does not need additional arguments, I found the following solution very satisfying (comments always welcome :-) ):
def apply(self, function, variable):
result={}
for k,v in self.get(variable).items():
result[k] = function(v)
return result
However, for a function requiring additional arguments I don't see how to do it in an elegant way. A typical operation would be the integration of foo with bar as x-values like np.trapz(foo, x=bar), where both foo and bar can be retreived with MyClass.get(...)
I was thinking in this direction:
def apply(self, function_call):
"""
function_call should be a string with the complete expression to evaluate
eg: MyClass.apply('np.trapz(QHeat, time)')
"""
result={}
for SID in self.simulations:
result[SID] = eval(function_call, locals=...)
return result
The problem is that I don't know how to pass the locals mapping object. Or maybe I'm looking in a wrong direction. Thanks on beforehand for your help.
Roel
You have two ways. The first is to use functools.partial:
foo = self.get('foo')
bar = self.get('bar')
callable = functools.partial(func, foo, x=bar)
self.apply(callable, variable)
while the second approach is to use the same technique used by partial, you can define a function that accept arbitrary argument list:
def apply(self, function, variable, *args, **kwds):
result={}
for k,v in self.get(variable).items():
result[k] = function(v, *args, **kwds)
return result
Note that in both case the function signature remains unchanged. I don't know which one I'll choose, maybe the first case but I don't know the context on you are working on.
I tried to recreate (the relevant part of) the class structure the way I am guessing it is set up on your side (it's always handy if you can provide a simplified code example for people to play/test).
What I think you are trying to do is translate variable names to variables that are obtained from within the class and then use those variables in a function that was passed in as well. In addition to that since each variable is actually a dictionary of values with a key (SID), you want the result to be a dictionary of results with the function applied to each of the arguments.
class test:
def get(self, name):
if name == "valA":
return {"1":"valA1", "2":"valA2", "3":"valA3"}
elif name == "valB":
return {"1":"valB1", "2":"valB2", "3":"valB3"}
def apply(self, function, **kwargs):
arg_dict = {fun_arg: self.get(sim_args) for fun_arg, sim_args in kwargs.items()}
result = {}
for SID in arg_dict[kwargs.keys()[0]]:
fun_kwargs = {fun_arg: sim_dict[SID] for fun_arg, sim_dict in arg_dict.items()}
result[SID] = function(**fun_kwargs)
return result
def joinstrings(string_a, string_b):
return string_a+string_b
my_test = test()
result = my_test.apply(joinstrings, string_a="valA", string_b="valB")
print result
So the apply method gets an argument dictionary, gets the class specific data for each of the arguments and creates a new argument dictionary with those (arg_dict).
The SID keys are obtained from this arg_dict and for each of those, a function result is calculated and added to the result dictionary.
The result is:
{'1': 'valA1valB1', '3': 'valA3valB3', '2': 'valA2valB2'}
The code can be altered in many ways, but I thought this would be the most readable. It is of course possible to join the dictionaries instead of using the SID's from the first element etc.