I am wondering if it is possible to list the variables expected by a Python function, prior to calling it, in order to pass the expected variables from a bigger dict containing a lot of variables.
I have searched the net but couldn't find anything. However, the python interpreter can show the list of expected variables, so there surely must be some way to do it in a script?
You can use either the inspect.signature() or inspect.getfullargspec() functions:
import inspect
argspec = inspect.getfullargspec(somefunction)
signature = inspect.signature(somefunction)
inspect.fullargspec returns a named tuple with 7 elements:
A list with the argument names
The name of the catchall *args parameter, if defined (None otherwise)
The name of the catchall **kwargs parameter, if defined (None otherwise)
A tuple with default values for the keyword arguments; they go with the last elements of the arguments; match these by length of the default values tuple.
A list of keyword-only parameter names
A dictionary of default values for the keyword-only parameter names, if any
and a dictionary containing the annotations
With inspect.signature() you get a Signature object, a rich object that models not only the above data as a more structured set of objects but also lets you bind values to parameters the same way a call to the function would.
Which one is better will depend on your use cases.
Demo:
>>> import inspect
>>> def foo(bar, baz, spam='eggs', *monty, python: "kwonly", spanish=42, **inquisition) -> "return annotation":
... pass
...
>>> inspect.getfullargspec(foo)
FullArgSpec(args=['bar', 'baz', 'spam'], varargs='monty', varkw='inquisition', defaults=('eggs',), kwonlyargs=['python', 'spanish'], kwonlydefaults={'spanish': 42}, annotations={'return': 'return annotation', 'python': 'kwonly'})
>>> signature = inspect.signature(foo)
>>> signature
<Signature (bar, baz, spam='eggs', *monty, python: 'kwonly', spanish=42, **inquisition) -> 'return annotation'>
>>> signature.parameters['python'].kind.description
'keyword-only'
>>> signature.bind('Eric', 'Idle', 'John', python='Cleese')
<BoundArguments (bar='Eric', baz='Idle', spam='John', python='Cleese')>
If you have a dictionary named values of possible parameter values, I'd use inspect.signature() and use the Signature.parameters mapping to match names:
posargs = [
values[param.name]
for param in signature.parameters.values()
if param.kind is Parameter.POSITIONAL_ONLY
]
skip_kinds = {Parameter.POSITIONAL_ONLY, Parameter.VAR_POSITIONAL, Parameter.VAR_KEYWORD}
kwargs = {
param.name: values[param.name]
for param in signature.parameters.values()
if param.name in values and param.kind not in skip_kinds
}
The above gives you a list of values for the positional-only parameters, and a dictionary for the rest (excepting any *args or **kwargs parameters).
Just as a side answer, I now use another approach to pass to functions the variables they expect: I pass them all.
What I mean is that I maintain a kind of global/shared dictionnary of variables in my root object (which is the parent of all other objects), eg:
shareddict = {'A': 0, 'B':'somestring'}
Then I simply pass this dict to any method of any other object that is to be called, just like this:
shareddict.update(call_to_func(**shareddict))
As you can see, we unpack all the keys/values in shareddict as keyword arguments to call_to_func(). We also update shareddict with the returned result, we'll see below why.
Now with this technic, I can simply and clearly define in my functions/methods if I need one or several variables from this dict:
my_method1(A=None, *args, **kwargs):
''' This method only computes on A '''
new_A = Do_some_stuff(A)
return {'A': new_A} # Return the new A in a dictionary to update the shared value of A in the shareddict
my_method2(B=None, *args, **kwargs):
''' This method only computes on B '''
new_B = Do_some_stuff(B)
return {'B': new_B} # Return the new B in a dictionary to update the shareddict
my_method3(A=None, B=None, *args, **kwargs):
''' This method swaps A and B, and then create a new variable C '''
return {'A': B, 'B': A, 'C': 'a_new_variable'} # Here we will update both A and B and create the new variable C
As you can notice, all the methods above return a dict of variables, which will update the shareddict, and which will get passed along to other functions.
This technic has several advantages:
Quite simple to implement
Elegant way to maintain a shared list of variables but without using a global variable
Functions and methods clearly show in their definitions what they expect (but of course one caveat is that even mandatory variables will need to be set as a keyword argument with a default value such as None, which usually means that the variable is optional, but here it's not
The methods are inheritable and overloadable
Low memory footprint since the same shareddict is passed all along
The children functions/methods define what they need (bottom-up), instead of the root defining what arguments will be passed to children (top-down)
Very easy to create/update variables
Optionally, it's VERY easy to dump all those variables in a file, eg by using json.dumps(finaldict, sort_keys=True).
Nice and easy:
import inspect #library to import
def foo(bar, baz, spam='eggs', *monty, **python): pass #example function
argspec = inspect.signature(foo)
print(argspec) #print your output
prints: (bar, baz, spam='eggs', *monty, **python)
It also works for methods inside classes (very useful!):
class Complex: #example Class
def __init__(self, realpart, imagpart): #method inside Class
... self.r = realpart
... self.i = imagpart
argspec = inspect.signature(Complex)
print(argspec)
prints: (realpart, imagpart)
Related
I am reading the book Python Crash Course, and in the following I have a confusion why a dictionary is defined as user_info={k=v,k1=v1,...} and not user_info={k:v,k1:v1,...}. Here is the code from the book (slightly altered for name of person):
def build_profile(first, last, **user_info):
profile = {}
profile['first_name'] = first.title()
profile['last_name'] = last.title()
for key, value in user_info.items():
profile[key] = value
return profile
build_profile('freddie','mercury',location='uk',field='rock')
In the book, he says the following: The
double asterisks before the parameter **user_info cause Python to create
an empty dictionary called user_info and pack whatever name-value pairs it
receives into this dictionary.
Now, when I type in the values for the dictionary the ordinary way (ie key:value),
python gives me an error.
Could somebody explain why is this so?
Dictionaries are objects with key-value pairs. There's a special literal for creating dicts:
{'key': 'value'}
You could also use the dict constructor to do the same thing:
dict(key='value')
What's shown here are named function arguments. Taking a step back…
These are function definitions:
def foo(bar, baz): ...
This function defines two parameters. You can give these parameters default values:
def foo(bar, baz=42): ...
The parameter baz has the default value 42. When calling the function, you can supply the arguments positionally:
foo('a', 14)
Or, and here we go, you can supply the arguments by name:
foo(bar='a', baz=14)
foo(baz=14, bar='a') # exactly the same as above
This is in fact what the dict constructor does, it's a function call with named arguments.
Now, you can also accept arbitrary named arguments in your function:
def foo(**kwargs): ...
Any named argument you pass in will be accepted and land in the kwargs parameter, which happens to be a dict (because that's the natural data structure for a key-value mapping).
So, you're not constructing a dict with = there. You're calling a function that accepts arbitrary named arguments via **, and those end up being a dict. Function parameters or arguments always involve the = syntax. The only time : is used is in dict literals.
A good way to think of this might be that you are allowing the user to input in as many values as they would want. Say you have a function like this:
def Function(*arguments):
...
Here, the asterisks before the arguments means that the user is allowed to supply as many values as they want into the parameter. It it thus creating a list of values. Similarly, adding two asterisks before also means that the user is allowed to type in as many values as they want, but this time, the values will be organized as a dictionary.
so I have a function:
def Function(**arguments):
print(arguments)
then, if I call it like so:
Function("Key0", 0, "Key1", 1, ...)
The function takes the even values (value 0, 2, ...) as the keys and the odd values (1, 3, ...) as the values. So this code would print:
{"Key0": 0, "Key1": 1, ...}
Now note, this is not exactly how it functions at the lower lever, but it is a good way to think about it.
In python we can do this:
def myFun1(one = '1', two = '2'):
...
Then we can call the function and pass the arguments by their name:
myFun1(two = 'two', one = 'one')
Also, we can do this:
def myFun2(**kwargs):
print kwargs.get('one', 'nothing here')
myFun2(one='one')
So I was wondering if it is possible to combine both methods like:
def myFun3(name, lname, **other_info):
...
myFun3(lname='Someone', name='myName', city='cityName', otherInfo='blah')
In general what combinations can we do?
Thanks and sorry for my silly question.
The general idea is:
def func(arg1, arg2, ..., kwarg1=default, kwarg2=default, ..., *args, **kwargs):
...
You can use as many of those as you want. The * and ** will 'soak up' any remaining values not otherwise accounted for.
Positional arguments (provided without defaults) can't be given by keyword, and non-default arguments can't follow default arguments.
Note Python 3 also adds the ability to specify keyword-only arguments by having them after *:
def func(arg1, arg2, *args, kwonlyarg=default):
...
You can also use * alone (def func(a1, a2, *, kw=d):) which means that no arguments are captured, but anything after is keyword-only.
So, if you are in 3.x, you could produce the behaviour you want with:
def myFun3(*, name, lname, **other_info):
...
Which would allow calling with name and lname as keyword-only.
Note this is an unusual interface, which may be annoying to the user - I would only use it in very specific use cases.
In 2.x, you would need to manually make this by parsing **kwargs.
You can add your named arguments along with kwargs. If the keys are available in the calling function It will taken to your named argument otherwise it will be taken by the kwargs dictionary.
def add(a=1, b=2,**c):
res = a+b
for items in c:
res = res + c[items]
print(res)
add(2,3)
5
add(b=4, a =3)
7
add(a =1,b=2,c=3,d=4)
10
It's possible at least for Python 2.7. Keyword arguments get assigned to positional parameters by name, so you can do
In [34]: def func(name, lname, **kwargs):
print 'name='+name, 'lname='+lname
print kwargs
....:
In [35]: func(lname='lname_val', name='name_val', city='cityName', otherInfo='blah')
name=name_val lname=lname_val
{'city': 'cityName', 'otherInfo': 'blah'}
Official docs state it that way:
"If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). If the slot is already filled, a TypeError exception is raised. Otherwise, the value of the argument is placed in the slot, filling it (even if the expression is None, it fills the slot)."
https://docs.python.org/2/reference/expressions.html#calls
What I want to accomplish:
dct = {'foo':0, 'bar':1, 'baz':2}
def func(**dct):
pass
#function signature is now func(foo=0, bar=1, baz=2)
However, the ** syntax is obviously clashing here between expanding a dict (what I want to do) and declaring a parameter that holds the keyword arguments (what I don't want to do).
Is this possible?
Based on my interpretation of your requirements -- you want to dynamically define a function with a signature that matches the content of adict provided at runtime -- there are two issues here which makes it impractical.
If the arguments are defined at run-time, how can your function reference the variables? Are you planning to build the function body at run-time as well?
dicts are unordered, so you cannot reliably use them to define positional arguments
I suspect this is an XY problem. If you can explain what you're trying to achieve then perhaps we can be of better help.
However, assuming you're trying to assign default keyword arguments using a dict then one way to achieve this would be to use decorators. For example:
def defaultArgs(default_kw):
"decorator to assign default kwargs"
def wrap(f):
def wrapped_f(**kwargs):
kw = {}
kw.update(default_kw) # apply defaults
kw.update(kwargs) # apply from input args
f(**kw) # run actual function with updated kwargs
return wrapped_f
return wrap
You can then use this decorator to assign default keyword arguments to a function that expects only keyword arguments:
defaults = {'foo':0, 'bar':1, 'baz':2}
#defaultArgs(defaults)
def func(**kwargs):
print kwargs # args accessible via the kwargs dict
Results:
func() # prints {'baz': 2, 'foo': 0, 'bar': 1}
func(foo=2) # prints {'baz': 2, 'foo': 2, 'bar': 1}
params = {'bar':1000, 'hello':'world'}
func(**params) # prints {'baz': 2, 'foo': 0, 'bar': 1000, 'hello': 'world'}
Note that you will not be able to use positional arguments:
func(1, 2, 3) # raises TypeError
what you want i believe is eval() link
an answer i gave on a similar question: https://stackoverflow.com/a/11865190/1561176
I'm really not sure what you plan on accomplishing here. The following works (sort of):
def func(**dct):
pass
dct = {'foo':0, 'bar':1, 'baz':2}
func(**dct)
How do you plan on using foo, bar or baz in the function if they're created dynamically? If you give some more details on what you're actually trying to accomplish, we might be able to be a little more helpful.
I have created a class MyClassthat contains a lot of simulation data. The class groups simulation results for different simulations that have a similar structure. The results can be retreived with a MyClass.get(foo) method. It returns a dictionary with simulationID/array pairs, array being the value of foo for each simulation.
Now I want to implement a method in my class to apply any function to all the arrays for foo. It should return a dictionary with simulationID/function(foo) pairs.
For a function that does not need additional arguments, I found the following solution very satisfying (comments always welcome :-) ):
def apply(self, function, variable):
result={}
for k,v in self.get(variable).items():
result[k] = function(v)
return result
However, for a function requiring additional arguments I don't see how to do it in an elegant way. A typical operation would be the integration of foo with bar as x-values like np.trapz(foo, x=bar), where both foo and bar can be retreived with MyClass.get(...)
I was thinking in this direction:
def apply(self, function_call):
"""
function_call should be a string with the complete expression to evaluate
eg: MyClass.apply('np.trapz(QHeat, time)')
"""
result={}
for SID in self.simulations:
result[SID] = eval(function_call, locals=...)
return result
The problem is that I don't know how to pass the locals mapping object. Or maybe I'm looking in a wrong direction. Thanks on beforehand for your help.
Roel
You have two ways. The first is to use functools.partial:
foo = self.get('foo')
bar = self.get('bar')
callable = functools.partial(func, foo, x=bar)
self.apply(callable, variable)
while the second approach is to use the same technique used by partial, you can define a function that accept arbitrary argument list:
def apply(self, function, variable, *args, **kwds):
result={}
for k,v in self.get(variable).items():
result[k] = function(v, *args, **kwds)
return result
Note that in both case the function signature remains unchanged. I don't know which one I'll choose, maybe the first case but I don't know the context on you are working on.
I tried to recreate (the relevant part of) the class structure the way I am guessing it is set up on your side (it's always handy if you can provide a simplified code example for people to play/test).
What I think you are trying to do is translate variable names to variables that are obtained from within the class and then use those variables in a function that was passed in as well. In addition to that since each variable is actually a dictionary of values with a key (SID), you want the result to be a dictionary of results with the function applied to each of the arguments.
class test:
def get(self, name):
if name == "valA":
return {"1":"valA1", "2":"valA2", "3":"valA3"}
elif name == "valB":
return {"1":"valB1", "2":"valB2", "3":"valB3"}
def apply(self, function, **kwargs):
arg_dict = {fun_arg: self.get(sim_args) for fun_arg, sim_args in kwargs.items()}
result = {}
for SID in arg_dict[kwargs.keys()[0]]:
fun_kwargs = {fun_arg: sim_dict[SID] for fun_arg, sim_dict in arg_dict.items()}
result[SID] = function(**fun_kwargs)
return result
def joinstrings(string_a, string_b):
return string_a+string_b
my_test = test()
result = my_test.apply(joinstrings, string_a="valA", string_b="valB")
print result
So the apply method gets an argument dictionary, gets the class specific data for each of the arguments and creates a new argument dictionary with those (arg_dict).
The SID keys are obtained from this arg_dict and for each of those, a function result is calculated and added to the result dictionary.
The result is:
{'1': 'valA1valB1', '3': 'valA3valB3', '2': 'valA2valB2'}
The code can be altered in many ways, but I thought this would be the most readable. It is of course possible to join the dictionaries instead of using the SID's from the first element etc.
I want to create a python dictionary that returns me the key value for the keys are missing from the dictionary.
Usage example:
dic = smart_dict()
dic['a'] = 'one a'
print(dic['a'])
# >>> one a
print(dic['b'])
# >>> b
dicts have a __missing__ hook for this:
class smart_dict(dict):
def __missing__(self, key):
return key
Could simplify it as (since self is never used):
class smart_dict(dict):
#staticmethod
def __missing__(key):
return key
Why don't you just use
dic.get('b', 'b')
Sure, you can subclass dict as others point out, but I find it handy to remind myself every once in a while that get can have a default value!
If you want to have a go at the defaultdict, try this:
dic = defaultdict()
dic.__missing__ = lambda key: key
dic['b'] # should set dic['b'] to 'b' and return 'b'
except... well: AttributeError: ^collections.defaultdict^object attribute '__missing__' is read-only, so you will have to subclass:
from collections import defaultdict
class KeyDict(defaultdict):
def __missing__(self, key):
return key
d = KeyDict()
print d['b'] #prints 'b'
print d.keys() #prints []
Congratulations. You too have discovered the uselessness of the
standard collections.defaultdict type. If that execrable midden heap of code smell
offends your delicate sensibilities as much as it did mine, this is your lucky
StackOverflow day.
Thanks to the forbidden wonder of the 3-parameter
variant of the type()
builtin, crafting a non-useless default dictionary type is both fun and profitable.
What's Wrong with dict.__missing__()?
Absolutely nothing, assuming you like excess boilerplate and the shocking silliness of collections.defaultdict – which should behave as expected but really doesn't. To be fair, Jochen
Ritzel's accepted
solution of subclassing dict and
implementing the optional __missing__()
method is a fantastic
workaround for small-scale use cases only requiring a single default dictionary.
But boilerplate of this sort scales poorly. If you find yourself instantiating
multiple default dictionaries, each with their own slightly different logic for
generating missing key-value pairs, an industrial-strength alternative
automating boilerplate is warranted.
Or at least nice. Because why not fix what's broken?
Introducing DefaultDict
In less than ten lines of pure Python (excluding docstrings, comments, and
whitespace), we now define a DefaultDict type initialized with a user-defined
callable generating default values for missing keys. Whereas the callable passed
to the standard collections.defaultdict type uselessly accepts no
parameters, the callable passed to our DefaultDict type usefully accepts the
following two parameters:
The current instance of this dictionary.
The current missing key to generate a default value for.
Given this type, solving sorin's
question reduces to a single line of Python:
>>> dic = DefaultDict(lambda self, missing_key: missing_key)
>>> dic['a'] = 'one a'
>>> print(dic['a'])
one a
>>> print(dic['b'])
b
Sanity. At last.
Code or It Didn't Happen
def DefaultDict(keygen):
'''
Sane **default dictionary** (i.e., dictionary implicitly mapping a missing
key to the value returned by a caller-defined callable passed both this
dictionary and that key).
The standard :class:`collections.defaultdict` class is sadly insane,
requiring the caller-defined callable accept *no* arguments. This
non-standard alternative requires this callable accept two arguments:
#. The current instance of this dictionary.
#. The current missing key to generate a default value for.
Parameters
----------
keygen : CallableTypes
Callable (e.g., function, lambda, method) called to generate the default
value for a "missing" (i.e., undefined) key on the first attempt to
access that key, passed first this dictionary and then this key and
returning this value. This callable should have a signature resembling:
``def keygen(self: DefaultDict, missing_key: object) -> object``.
Equivalently, this callable should have the exact same signature as that
of the optional :meth:`dict.__missing__` method.
Returns
----------
MappingType
Empty default dictionary creating missing keys via this callable.
'''
# Global variable modified below.
global _DEFAULT_DICT_ID
# Unique classname suffixed by this identifier.
default_dict_class_name = 'DefaultDict' + str(_DEFAULT_DICT_ID)
# Increment this identifier to preserve uniqueness.
_DEFAULT_DICT_ID += 1
# Dynamically generated default dictionary class specific to this callable.
default_dict_class = type(
default_dict_class_name, (dict,), {'__missing__': keygen,})
# Instantiate and return the first and only instance of this class.
return default_dict_class()
_DEFAULT_DICT_ID = 0
'''
Unique arbitrary identifier with which to uniquify the classname of the next
:func:`DefaultDict`-derived type.
'''
The key ...get it, key? to this arcane wizardry is the call to
the 3-parameter variant
of the type() builtin:
type(default_dict_class_name, (dict,), {'__missing__': keygen,})
This single line dynamically generates a new dict subclass aliasing the
optional __missing__ method to the caller-defined callable. Note the distinct
lack of boilerplate, reducing DefaultDict usage to a single line of Python.
Automation for the egregious win.
The first respondent mentioned defaultdict,
but you can define __missing__ for any subclass of dict:
>>> class Dict(dict):
def __missing__(self, key):
return key
>>> d = Dict(a=1, b=2)
>>> d['a']
1
>>> d['z']
'z'
Also, I like the second respondent's approach:
>>> d = dict(a=1, b=2)
>>> d.get('z', 'z')
'z'
I agree this should be easy to do, and also easy to set up with different defaults or functions that transform a missing value somehow.
Inspired by Cecil Curry's answer, I asked myself: why not have the default-generator (either a constant or a callable) as a member of the class, instead of generating different classes all the time? Let me demonstrate:
# default behaviour: return missing keys unchanged
dic = FlexDict()
dic['a'] = 'one a'
print(dic['a'])
# 'one a'
print(dic['b'])
# 'b'
# regardless of default: easy initialisation with existing dictionary
existing_dic = {'a' : 'one a'}
dic = FlexDict(existing_dic)
print(dic['a'])
# 'one a'
print(dic['b'])
# 'b'
# using constant as default for missing values
dic = FlexDict(existing_dic, default = 10)
print(dic['a'])
# 'one a'
print(dic['b'])
# 10
# use callable as default for missing values
dic = FlexDict(existing_dic, default = lambda missing_key: missing_key * 2)
print(dic['a'])
# 'one a'
print(dic['b'])
# 'bb'
print(dic[2])
# 4
How does it work? Not so difficult:
class FlexDict(dict):
'''Subclass of dictionary which returns a default for missing keys.
This default can either be a constant, or a callable accepting the missing key.
If "default" is not given (or None), each missing key will be returned unchanged.'''
def __init__(self, content = None, default = None):
if content is None:
super().__init__()
else:
super().__init__(content)
if default is None:
default = lambda missing_key: missing_key
self.default = default # sets self._default
#property
def default(self):
return self._default
#default.setter
def default(self, val):
if callable(val):
self._default = val
else: # constant value
self._default = lambda missing_key: val
def __missing__(self, x):
return self.default(x)
Of course, one can debate whether one wants to allow changing the default-function after initialisation, but that just means removing #default.setter and absorbing its logic into __init__.
Enabling introspection into the current (constant) default value could be added with two extra lines.
Subclass dict's __getitem__ method. For example, How to properly subclass dict and override __getitem__ & __setitem__