I'm trying to figure out how to pass optional arguments from optparse. The problem I'm having is if an optparse option is not specified, it defaults to a None type, but if I pass the None type into a function, it yells at me instead of using the default (Which is understandable and valid).
conn = psycopg2.connect(database=options.db, hostname=options.hostname, port=options.port)
The question is, how do I use the function's defaults for optional arguments but still pass in user inputs if there is an input without having a huge number of if statements.
Define a function remove_none_values that filters a dictionary for none-valued arguments.
def remove_none_values(d):
return dict((k,v) for (k,v) in d.iteritems() if not v is None)
kwargs = {
'database': options.db,
'hostname': options.hostname,
...
}
conn = psycopg2.connect(**remove_none_values(kwargs))
Or, define a function wrapper that removes none values before passing the data on to the original function.
def ignore_none_valued_kwargs(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
newkwargs = dict((k,v) for (k,v) in d.iteritems() if not v is None)
return f(*args, **kwargs)
return wrapper
my_connect = ignore_none_valued_kwargs(psycopg2)
conn = my_connect(database=options.db, hostname=options.hostname, port=options.port)
The opo module of my thebops package (pip install thebops, https://bitbucket.org/therp/thebops) contains an add_optval_option function.
This uses an additional keyword argument empty which specifies the value to use if the option is used without a value. If one of the option strings is found in the commandline, this value is injected into the argument list.
This is still hackish, but at least it is made a simple-to-use function ...
It works well under the following circumstances:
The argument vector does already exist when the option is created. This is usually true.
All programs I found which sport arguments with optional values require the given value to be attached as --option=value or -ovalue rather than --option value or -o value.
Maybe I'll tweak thebops.optparse to support the empty argument as well; but I'd like to have a test suite first to prevent regressions, preferably the original Optik / optparse tests.
This is the code:
from sys import argv
def add_optval_option(pog, *args, **kwargs):
"""
Add an option which can be specified without a value;
in this case, the value (if given) must be contained
in the same argument as seen by the shell,
i.e.:
--option=VALUE, --option will work;
--option VALUE will *not* work
Arguments:
pog -- parser or group
empty -- the value to use when used without a value
Note:
If you specify a short option string as well, the syntax given by the
help will be wrong; -oVALUE will be supported, -o VALUE will not!
Thus it might be wise to create a separate option for the short
option strings (in a "hidden" group which isn't added to the parser after
being populated) and just mention it in the help string.
"""
if 'empty' in kwargs:
empty_val = kwargs.pop('empty')
# in this case it's a good idea to have a <default> value; this can be
# given by another option with the same <dest>, though
for i in range(1, len(argv)):
a = argv[i]
if a == '--':
break
if a in args:
argv.insert(i+1, empty_val)
break
pog.add_option(*args, **kwargs)
Related
For this function
def eat_dog(name, should_digest=True):
print "ate dog named %s. Digested, too? %" % (name, str(should_digest))
I want to, external to the function, read its arguments and any default values attached. So for this specific example, I want to know that name has no default value (i.e. that it is a required argument) and that True is the default value for should_digest.
I'm aware of inspect.getargspec(), which does give me information about arguments and default values, but I see no connection between the two:
ArgSpec(args=['name', 'should_digest'], varargs=None, keywords=None, defaults=(True,))
From this output how can I tell that True (in the defaults tuple) is the default value for should_digest?
Additionally, I'm aware of the "ask for forgiveness" model of approaching a problem, but unfortunately output from that error won't tell me the name of the missing argument:
>>> eat_dog()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: eat_dog() takes at least 1 argument (0 given)
To give context (why I want to do this), I'm exposing functions in a module over a JSON API. If the caller omits certain function arguments, I want to return a specific error that names the specific function argument that was omitted. If a client omits an argument, but there's a default provided in the function signature, I want to use that default.
Python3.x
In a python3.x world, you should probably use a Signature object:
import inspect
def get_default_args(func):
signature = inspect.signature(func)
return {
k: v.default
for k, v in signature.parameters.items()
if v.default is not inspect.Parameter.empty
}
Python2.x (old answer)
The args/defaults can be combined as:
import inspect
a = inspect.getargspec(eat_dog)
zip(a.args[-len(a.defaults):],a.defaults)
Here a.args[-len(a.defaults):] are the arguments with defaults values and obviously a.defaults are the corresponding default values.
You could even pass the output of zip to the dict constructor and create a mapping suitable for keyword unpacking.
looking at the docs, this solution will only work on python2.6 or newer since I assume that inspect.getargspec returns a named tuple. Earlier versions returned a regular tuple, but it would be very easy to modify accordingly. Here's a version which works with older (and newer) versions:
import inspect
def get_default_args(func):
"""
returns a dictionary of arg_name:default_values for the input function
"""
args, varargs, keywords, defaults = inspect.getargspec(func)
return dict(zip(args[-len(defaults):], defaults))
Come to think of it:
return dict(zip(reversed(args), reversed(defaults)))
would also work and may be more intuitive to some people.
Depending on exactly what you need, you might not need the inspect module since you can check the __defaults__ attribute of the function:
>>> eat_dog.__defaults__
(True,)
>>> eat_dog.__code__.co_argcount
2
>>> eat_dog.__code__.co_varnames
('name', 'should_digest')
>>>
>>> eat_dog.__kwdefaults__
>>> eat_dog.__code__.co_kwonlyargcount
0
You can use inspect module with its getargspec function:
inspect.getargspec(func)
Get the names and default values of a Python function’s arguments. A tuple of four things is returned: (args, varargs, keywords, defaults). args is a list of the argument names (it may contain nested lists). varargs and keywords are the names of the * and ** arguments or None. defaults is a tuple of default argument values or None if there are no default arguments; if this tuple has n elements, they correspond to the last n elements listed in args.
See mgilson's answer for exact code on how to retrieve argument names and their default values.
To those looking for a version to grab a specific default parameter with mgilson's answer.
value = signature(my_func).parameters['param_name'].default
Here's a full working version, done in Python 3.8.2
from inspect import signature
def my_func(a, b, c, param_name='apple'):
pass
value = signature(my_func).parameters['param_name'].default
print(value == 'apple') # True
to take care of keyword-only args (and because defaults and kwonlydefaults can be None):
spec = inspect.getfullargspec(func)
defaults = dict(zip(spec.args[::-1], (spec.defaults or ())[::-1]))
defaults.update(spec.kwonlydefaults or {})
You can get this via some of the __dunder__ vars as mentioned by other posts. Putting that into a simple helper function can get you a dictionary of default values.
.__code__.co_varnames: A tuple of all input variables
.__defaults__: A tuple of the default values
It is worth noting that this tuple only incudes the default provided variables which must always be positioned last in the function arguments
You can use these two items to match the last n variables in the .__code__.co_varnames with all the items in the .__defaults__
EDIT Thanks to #griloHBG - Added if statement to prevent exceptions when no defaults are specified.
def my_fn(a, b=2, c='a'):
pass
def get_defaults(fn):
if fn.__defaults__==None:
return {}
return dict(zip(
fn.__code__.co_varnames[-len(fn.__defaults__):],
fn.__defaults__
))
print(get_defaults(my_fn))
Should give:
{'b': 2, 'c': 'a'}
In python, all the arguments with default value come after the arguments without default value. So the mapping should start from the end till you exhaust the default value list. Hence the logic:
dict(zip(reversed(args), reversed(defaults)))
gives the correctly mapped defaults.
I am using the Click library but I can't seem to find a behavior similar to dest from argparse.
For example, I have
#click.option('--format', type=click.Choice(['t', 'j']))
def plug(format):
pass
Notice that I am using a flag with --format that gets translated into a built-in Python construct format which is not ideal.
Is there a way to change the argument passed into the click function for options?
While Click doesn't have dest-equivalent of argparse, it has certain argument-naming behavior which can be exploited. Specifically, for parameters with multiple possible names, it will prefer non-dashed to dashed names, and as secondary preference will prioritize longer names over shorter names.
URL: http://click.pocoo.org/dev/parameters/#parameter-names
So if you declare your option as...
#click.option('--format', 'not-format', type=click.Choice(['t', 'j']))
...then Click will prioritize non-dashed variant ('not-format') and call your function with not_format=... argument.
Of course it also means that this alternative spelling can also be used in command line. If that is not desired, then I guess you could add a decorator to rename keyword arguments:
import functools
def rename_kwargs(**replacements):
def actual_decorator(func):
#functools.wraps(func)
def decorated_func(*args, **kwargs):
for internal_arg, external_arg in replacements.iteritems():
if external_arg in kwargs:
kwargs[internal_arg] = kwargs.pop(external_arg)
return func(*args, **kwargs)
return decorated_func
return actual_decorator
Testing code:
if __name__ == '__main__':
#rename_kwargs(different_arg='format')
def tester(different_arg):
print different_arg
tester(format='test value')
Test output:
$ python test_decor.py
test value
In your case, it would look like:
#click.option('--format', type=click.Choice(['t', 'j']))
#replace_kwargs(not_format='format')
def plug(not_format):
pass
Renaming an option to a differently named function argument is possible by decorating the function with
#click.option('--format', '-f', 'format_arg_name')
def plug(format_arg_name):
print(format_arg_name)
then it will remap the option named format and make it available as the format_arg_name parameter.
format_arg_name will not be available as a command line option, but --format and -f are.
I have written a python module mymod.py that can be used also as a standalone program from command line.
In mymod.py I have defined a few functions (where default values are set using keywords)
and an if __name__=="__main__" block to use the module as a standalone program.
I want the possibility to override some of the default options, therefore in the main program I import argparse and use it to parse the options. I use a dictionary to store
the default values, so that if some day I need to change the default values I can easily
do it modifying its value in one place only.
It works, but I find that the code is not "clean" and thought that probably I am not doing it in the proper pythonic way.
This is a toy example to show what I do:
#!/usr/bin/env python
#mymod.py
__default_options__={
"f1_x":10,
"f2_x":10
}
def f1(x=__default_options__["f1_x"]):
return x**2
def f2(x=__default_options__["f2_x"]):
return x**4
# this function is the "core" function which uses f1 and f2
# to produce the main task of the program
def f(x=__default_options__["f1_x"],y=__default_options__["f2_x"]):
return f1(x)+f2(y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application")
parser.add_argument("--f1-x",help="the parameter passed to f1",
default=__default_options__["f1_x"], type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
default=__default_options__["f2_x"], type = float, dest = "y")
options= parser.parse_args()
print f(options.x,options.y)
Passing the default values like I do it is a bit cumbersome and probably against the spirit both of Python and argparse.
How can this code be improved to be more pythonic and use argparse at its best?
You can use the `ArgumentParser.set_defaults method, in the following way
default_options={
"x":10,
"y":10
}
def f1(**kwargs):
x=kwargs.get('x', defalut_options['x'])
return x**2
def f2(**kwargs):
y=kwargs.get('y', defalut_options['y'])
return x**4
def f(**kwargs):
x=kwargs.get('x', defalut_options['x'])
y=kwargs.get('y', defalut_options['y'])
return f1(x=x, y=y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application", formatter_class=argparse.ArgumentDefaultsHelpFormatter )
parser.add_argument("--f1-x",help="the parameter passed to f1",
type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
type = float, dest = "y")
parser.set_defaults(**default_options)
options= parser.parse_args()
print f(options.x,options.y)
It took me a while to make it work, because I didn't notice that you are using dest in add_argument (I never use it). If this keyword is not provided, argparse set the default dest to the long name of the argument (in this case f1_x and f2_x, as it substitutes - with _). To go to the point: if you want to provide a dictionary of defaults, the keys needs to match dest if provided. Besides, take care that parser.set_defaults just add arguments to the parser, so if you have some entry not in your parser, it will be added to the Namespace.
--Edited to add generic kwargs to the functions--
As #Francesco wrote in a comment, your defaults dictionary won't work as you probably intended: The functions will retain the defaults they had while loading the module, regardless of later changes to the dictionary. Here's how to make them track the current value of the dictionary:
_default_options = {
"f1_x":10,
"f2_x":10
}
def f1(x=None):
if x == None:
x = _default_options["f1_x"]
...
You can then modify _default_options via ArgumentParser, or in any other way, and f1() will use it if called with no arguments.
This requires that None could never be a meaningful value for x; if that's not the case, choose a suitable impossible value.
I'm using Python's optparse to do what it does best, but I can't figure out how to make the option callback trigger on the default argument value if no other is specified via command-line; is this even possible? This would make my code much cleaner.
I can't use argparse unfortunately, as the platform I'm running on has an outdated Python version.
Edit:
To provide more detail, I'm adding an option with a callback and a default value
parser.add_option(
"-f",
"--format",
type = "string",
action = "callback",
callback = format_callback,
default = "a,b,c,d")
The callback function is defined as follows:
def format_callback(option, opt, value, parser):
# some processing
parser.values.V = processed_value
Basically I'm processing the "--format" value and putting the result into the parser. This works fine, when "--format" is specified directly via command-line, but I'd like the callback to be triggered on the default "a,b,c,d" value as well.
It is simply not possible.
The optparse.OptionParser implementation of parse_args starts with:
def parse_args(self, args=None, values=None):
"""
parse_args(args : [string] = sys.argv[1:],
values : Values = None)
-> (values : Values, args : [string])
Parse the command-line options found in 'args' (default:
sys.argv[1:]). Any errors result in a call to 'error()', which
by default prints the usage message to stderr and calls
sys.exit() with an error message. On success returns a pair
(values, args) where 'values' is an Values instance (with all
your option values) and 'args' is the list of arguments left
over after parsing options.
"""
rargs = self._get_args(args)
if values is None:
values = self.get_default_values()
Default values are set before processing any arguments. Actual values then overwrite defaults as options are parsed; the option callbacks are called when a corresponding argument is found.
So callbacks simply cannot be invoked for defaults. The design of the optparse module makes this very hard to change.
You can inject the default when calling parse_args
options, args = parser.parse_args(args=["--option=default"] + sys.argv[1:])
Since flags passed later in the argument list override those passed earlier, this should work. It's possible you may need to modify your callback function to expect this depending on what it is doing.
I have the following function:
def foo(**kwargs):
if not kwargs:
# No keyword arguments? It's all right. Set defaults here...
elif ('start_index' or 'end_index') in kwargs:
# Do something here...
else:
# Catch unexpected keyword arguments
raise TypeError("%r are invalid keyword arguments" % (kwargs.keys())
Question:
I want to make sure that the only valid keyword arguments are start_index or end_index. Anything else will raise an error, even if mixed with the valid ones. What's the cookbook recipe to make sure that only start_index or end_index are accepted? Yes, I'm looking for a cookbook recipe but I'm not sure how to search for it. I'm not sure if using an if-elif-else structure is the correct way to do it either.
Why do you need **kwargs here? Just
def foo(start_index=None, end_index=None):
and Python will perform all validation for you.
For the sake of completeness, Here's an alternative that still uses **kwargs.
def foo(**kwargs):
start_index = kwargs.pop('start_index', STARTINDEX_DEFAULT)
end_index = kwargs.pop('end_index', ENDINDEX_DEFAULT)
if kwargs:
# Catch unexpected keyword arguments
raise TypeError("%r are invalid keyword arguments" % (kwargs.keys())
# Do something here...
But, you shouldn't want to use this when you don't absolutely need it, use regular parameters with default values (as in Roman Bodnarchuk's answer).
Cases when you might need this is when you also want to use *args, and need a way to distinguish the keyword arguments from arbitrarily man positional arguments. using **kwargs this way forces the keyword arguments to be passed as keywords; A positional argument can never find its way into **kwargs.
Another reason is so that you can really distinguish between a default and an explicit parameter which happens to be the default. None is often used as a default value for arguments to indicate "the argument doesn't apply", but sometimes you actually need to interpret the None as something other than a default. Checking for the presence or absence of a key in the **kwargs dict can accurately distinguish between these cases. (An alternative is to create an instance of a subclass of object whos sole purpose is to be the default value of a specific argument to that specific function)
If you really want to use **kwargs, I'd write that like:
def foo(**kwargs):
# Define default values for all keys
args = {'start_index': 0, 'end_index': -1}
# Get the keys passed in that aren't in args
extraargs = set(kwargs) - set(args)
if extraargs:
raise TypeError("Invalid arguments: %s" % list(extraargs))
# Overwrite the default values with the passed-in values
args.update(kwargs)
# Now, do stuff with the values in args
But all of that is a complicated, slow way to duplicate built-in functionality. Don't do that unless you really need to.
In any case getting keys from a dict is as easy as using .get e.g.
kwargs.get('WIDTH',500)
this way if it doesn't find WIDTH as a key you get 500.