Right now, I have a script that can accept command line arguments using argparse. For example, like this:
#foo.py
def function_with_args(optional_args=None):
parser = argparse.ArgumentParser()
# add some arguments
args = parser.parse_args(optional_args)
# do something with args
However, I'd like to be able to use this function with a dictionary instead, for example with something like this:
def function_using_dict(**kwargs):
# define parser and add some arguments
args = parser.parse_dict_args(kwargs)
# everything else is the same
Note that I have a lot of arguments with default values in argparse which I'd like to use, so the following wouldn't work:
def function_no_default_args(**kwargs):
args = kwargs # not using default values that we add to the parser!
argparse.Namespace is a relatively simple object subclass, with most of its code devoted to displaying the attributes (as print(args) shows). Internally parse_args uses get_attr and set_attr to access the namespace, minimizing the assumptions about attributes names.
When using subparsers, the subparser starts with a 'blank' namespace, and uses the following code to copy its values to the main namespace.
# In case this subparser defines new defaults, we parse them
# in a new namespace object and then update the original
# namespace for the relevant parts.
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)
Originally the main namespace was passed to the subparser, eg. parser.parse_known_args(arg_strings, namespace), but the current version lets the subparser defaults take priority.
Handling defaults is a bit complicated. If you don't have any required arguments then
args = parser.parse_args([])
will set all the defaults. Or you could look at the start of parse.parse_known_args to see how defaults are inserted into the namespace at the start of parsing. Just beware that there's an added step at the end of parsing that runs remaining defaults through their respective type functions.
If you are trying to convert the result of parse_args into a dict, you can probably just do this:
kwargs = vars(args)
After your comment, I thought about it. Going to go with your existing function.
#foo.py
def function_with_args_and_default_kwargs(optional_args=None, **kwargs):
parser = argparse.ArgumentParser()
# add some arguments
# add the other arguments
for k, v in kwargs.items():
parser.add_argument('--' + k, default=v)
args = parser.parse_args(optional_args)
# do something with args
Related
I have such script:
import argparse
parser = argparse.ArgumentParser(
description='Text file conversion.'
)
parser.add_argument("inputfile", help="file to process", type=str)
parser.add_argument("-o", "--out", default="output.txt",
help="output name")
parser.add_argument("-t", "--type", default="detailed",
help="Type of processing")
args = parser.parse_args()
for arg in args:
print(arg)
But it doesnt work. I get error:
TypeError: 'Namespace' object is not iterable
How to iterate over arguments and their value?
Add vars if you want to iterate over a Namespace object:
for arg in vars(args):
print arg, getattr(args, arg)
Namespace objects aren't iterable, the standard docs suggest doing the following if you want a dictionary:
>>> vars(args)
{'foo': 'BAR'}
So
for key, value in vars(args).items():
# do stuff
To be honest, I am not sure why you want to iterate over the arguments. That somewhat defeats the purpose of having an argument parser.
After
args = parser.parse_args()
to display the arguments, use:
print args # or print(args) in python3
The args object (of type argparse.Namespace) isn't iterable (i.e. not a list), but it has a .__str__ method, which displays the values nicely.
args.out and args.type give the values of the 2 arguments you defined. This works most of the time. getattr(args, key) the most general way of accessing the values, but usually isn't needed.
vars(args)
turns the namespace into a dictionary, which you can access with all the dictionary methods. This is spelled out in the docs.
ref: the Namespace paragraph of the docs - https://docs.python.org/2/library/argparse.html#the-namespace-object
I'm using args.__dict__, which lets you access the underlying dict structure.
Then, its a simple key-value iteration:
for k in args.__dict__:
print k, args.__dict__[k]
Parsing the _actions from your parser seems like a decent idea. Instead of running parse_args() and then trying to pick stuff out of your Namespace.
import argparse
parser = argparse.ArgumentParser(
description='Text file conversion.')
parser.add_argument("inputfile", help="file to process", type=str)
parser.add_argument("-o", "--out", default="output.txt",
help="output name")
parser.add_argument("-t", "--type", default="detailed",
help="Type of processing")
options = parser._actions
for k in options:
print(getattr(k, 'dest'), getattr(k, 'default'))
You can modify the 'dest' part to be 'choices' for example if you need the preset defaults for a parameter in another script (by returning the options in a function for example).
Using the builtin dict() class to iterate over the args is straightforward and works great:
args = parser.parse_args()
args_dict = dict(vars(args))
for i, ii in args_dict.items():
print(i, ii)
From builtins.py:
def __init__(self, seq=None, **kwargs): # known special case of dict.__init__
"""
dict() -> new empty dictionary
dict(mapping) -> new dictionary initialized from a mapping object's
(key, value) pairs
dict(iterable) -> new dictionary initialized as if via:
d = {}
for k, v in iterable:
d[k] = v
dict(**kwargs) -> new dictionary initialized with the name=value pairs
in the keyword argument list. For example: dict(one=1, two=2)
# (copied from class doc)
"""
pass
ArgumentParser.parse_args returns a Namespace object instead of an iterable arrays.
For your reference, https://docs.python.org/3/library/argparse.html#parsing-arguments
ArgumentParser parses arguments through the parse_args() method. This will inspect the command line, convert each argument to the appropriate type and then invoke the appropriate action.
And it is not supposed to be used like that. Consider your use case, in the doc, it says argparse will figure out how to parse those out of sys.argv., which means you don't have to iterate over those arguments.
I realize that the question is rather general but I didn't know exactly how to ask it for what I am doing, but here goes.
I want to create a tool that allows option in the following format which also uses custom actions:
tool.py {start|stop|restart|configure}
Each of the above commands are mutually exclusive and some can have separate unique options. All will call a custom action (subclassed argparse.Action).
tool.py start
The above will do nothing because no arguments (via "add_argument()") was defined.
I though about making a subparser, but doing so doesn't work initially unless you set default arguments, via "set_defaults()". However, doing this and setting:
class CustomAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
print('Args: %r %r %r' % (namespace, values, option_string))
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser(help="Basic daemon.")
subparsers = parser.add_subparsers()
start_parser = subparsers.add_parser("start")
start_parser.set_defaults(start=True, action=CustomAction)
doesn't seem to kick off the custom action as expected. Below is the output I get:
$ custom_parser.py start
Namespace(action=<class '__main__.BasicAction'>, start=True)
I can see that the values are being assigned, but NOT called.
I basically want to have exclusive parent options that can be specified without child argument but still allow exclusive sub-arguments like so, if desired:
tool.py configure {interval|recipients}
Any ideas?
You can use subparsers coupled with default functions
def start_something():
do_starting_actions()
def stop_something():
do_terminal_actions()
def parse_args():
parser = ArgumentParser()
subparsers = parser.add_subparsers()
start = subparsers.add_parser("start")
start.set_defaults(func=start_something)
stop = subparsers.add_parser("stop")
stop.set_defaults(func=stop_something)
# ...
return parser.parse_args()
def main():
args = parse_args()
args.func()
Then you can call the parser from the command line
mymodule.py start
If you wanted to extend the subparser you could do it like:
start = subparsers.add_parser("start")
start.add_argument("--foo")
I have written a python module mymod.py that can be used also as a standalone program from command line.
In mymod.py I have defined a few functions (where default values are set using keywords)
and an if __name__=="__main__" block to use the module as a standalone program.
I want the possibility to override some of the default options, therefore in the main program I import argparse and use it to parse the options. I use a dictionary to store
the default values, so that if some day I need to change the default values I can easily
do it modifying its value in one place only.
It works, but I find that the code is not "clean" and thought that probably I am not doing it in the proper pythonic way.
This is a toy example to show what I do:
#!/usr/bin/env python
#mymod.py
__default_options__={
"f1_x":10,
"f2_x":10
}
def f1(x=__default_options__["f1_x"]):
return x**2
def f2(x=__default_options__["f2_x"]):
return x**4
# this function is the "core" function which uses f1 and f2
# to produce the main task of the program
def f(x=__default_options__["f1_x"],y=__default_options__["f2_x"]):
return f1(x)+f2(y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application")
parser.add_argument("--f1-x",help="the parameter passed to f1",
default=__default_options__["f1_x"], type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
default=__default_options__["f2_x"], type = float, dest = "y")
options= parser.parse_args()
print f(options.x,options.y)
Passing the default values like I do it is a bit cumbersome and probably against the spirit both of Python and argparse.
How can this code be improved to be more pythonic and use argparse at its best?
You can use the `ArgumentParser.set_defaults method, in the following way
default_options={
"x":10,
"y":10
}
def f1(**kwargs):
x=kwargs.get('x', defalut_options['x'])
return x**2
def f2(**kwargs):
y=kwargs.get('y', defalut_options['y'])
return x**4
def f(**kwargs):
x=kwargs.get('x', defalut_options['x'])
y=kwargs.get('y', defalut_options['y'])
return f1(x=x, y=y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application", formatter_class=argparse.ArgumentDefaultsHelpFormatter )
parser.add_argument("--f1-x",help="the parameter passed to f1",
type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
type = float, dest = "y")
parser.set_defaults(**default_options)
options= parser.parse_args()
print f(options.x,options.y)
It took me a while to make it work, because I didn't notice that you are using dest in add_argument (I never use it). If this keyword is not provided, argparse set the default dest to the long name of the argument (in this case f1_x and f2_x, as it substitutes - with _). To go to the point: if you want to provide a dictionary of defaults, the keys needs to match dest if provided. Besides, take care that parser.set_defaults just add arguments to the parser, so if you have some entry not in your parser, it will be added to the Namespace.
--Edited to add generic kwargs to the functions--
As #Francesco wrote in a comment, your defaults dictionary won't work as you probably intended: The functions will retain the defaults they had while loading the module, regardless of later changes to the dictionary. Here's how to make them track the current value of the dictionary:
_default_options = {
"f1_x":10,
"f2_x":10
}
def f1(x=None):
if x == None:
x = _default_options["f1_x"]
...
You can then modify _default_options via ArgumentParser, or in any other way, and f1() will use it if called with no arguments.
This requires that None could never be a meaningful value for x; if that's not the case, choose a suitable impossible value.
I'm using Python's optparse to do what it does best, but I can't figure out how to make the option callback trigger on the default argument value if no other is specified via command-line; is this even possible? This would make my code much cleaner.
I can't use argparse unfortunately, as the platform I'm running on has an outdated Python version.
Edit:
To provide more detail, I'm adding an option with a callback and a default value
parser.add_option(
"-f",
"--format",
type = "string",
action = "callback",
callback = format_callback,
default = "a,b,c,d")
The callback function is defined as follows:
def format_callback(option, opt, value, parser):
# some processing
parser.values.V = processed_value
Basically I'm processing the "--format" value and putting the result into the parser. This works fine, when "--format" is specified directly via command-line, but I'd like the callback to be triggered on the default "a,b,c,d" value as well.
It is simply not possible.
The optparse.OptionParser implementation of parse_args starts with:
def parse_args(self, args=None, values=None):
"""
parse_args(args : [string] = sys.argv[1:],
values : Values = None)
-> (values : Values, args : [string])
Parse the command-line options found in 'args' (default:
sys.argv[1:]). Any errors result in a call to 'error()', which
by default prints the usage message to stderr and calls
sys.exit() with an error message. On success returns a pair
(values, args) where 'values' is an Values instance (with all
your option values) and 'args' is the list of arguments left
over after parsing options.
"""
rargs = self._get_args(args)
if values is None:
values = self.get_default_values()
Default values are set before processing any arguments. Actual values then overwrite defaults as options are parsed; the option callbacks are called when a corresponding argument is found.
So callbacks simply cannot be invoked for defaults. The design of the optparse module makes this very hard to change.
You can inject the default when calling parse_args
options, args = parser.parse_args(args=["--option=default"] + sys.argv[1:])
Since flags passed later in the argument list override those passed earlier, this should work. It's possible you may need to modify your callback function to expect this depending on what it is doing.
I'm trying to figure out how to pass optional arguments from optparse. The problem I'm having is if an optparse option is not specified, it defaults to a None type, but if I pass the None type into a function, it yells at me instead of using the default (Which is understandable and valid).
conn = psycopg2.connect(database=options.db, hostname=options.hostname, port=options.port)
The question is, how do I use the function's defaults for optional arguments but still pass in user inputs if there is an input without having a huge number of if statements.
Define a function remove_none_values that filters a dictionary for none-valued arguments.
def remove_none_values(d):
return dict((k,v) for (k,v) in d.iteritems() if not v is None)
kwargs = {
'database': options.db,
'hostname': options.hostname,
...
}
conn = psycopg2.connect(**remove_none_values(kwargs))
Or, define a function wrapper that removes none values before passing the data on to the original function.
def ignore_none_valued_kwargs(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
newkwargs = dict((k,v) for (k,v) in d.iteritems() if not v is None)
return f(*args, **kwargs)
return wrapper
my_connect = ignore_none_valued_kwargs(psycopg2)
conn = my_connect(database=options.db, hostname=options.hostname, port=options.port)
The opo module of my thebops package (pip install thebops, https://bitbucket.org/therp/thebops) contains an add_optval_option function.
This uses an additional keyword argument empty which specifies the value to use if the option is used without a value. If one of the option strings is found in the commandline, this value is injected into the argument list.
This is still hackish, but at least it is made a simple-to-use function ...
It works well under the following circumstances:
The argument vector does already exist when the option is created. This is usually true.
All programs I found which sport arguments with optional values require the given value to be attached as --option=value or -ovalue rather than --option value or -o value.
Maybe I'll tweak thebops.optparse to support the empty argument as well; but I'd like to have a test suite first to prevent regressions, preferably the original Optik / optparse tests.
This is the code:
from sys import argv
def add_optval_option(pog, *args, **kwargs):
"""
Add an option which can be specified without a value;
in this case, the value (if given) must be contained
in the same argument as seen by the shell,
i.e.:
--option=VALUE, --option will work;
--option VALUE will *not* work
Arguments:
pog -- parser or group
empty -- the value to use when used without a value
Note:
If you specify a short option string as well, the syntax given by the
help will be wrong; -oVALUE will be supported, -o VALUE will not!
Thus it might be wise to create a separate option for the short
option strings (in a "hidden" group which isn't added to the parser after
being populated) and just mention it in the help string.
"""
if 'empty' in kwargs:
empty_val = kwargs.pop('empty')
# in this case it's a good idea to have a <default> value; this can be
# given by another option with the same <dest>, though
for i in range(1, len(argv)):
a = argv[i]
if a == '--':
break
if a in args:
argv.insert(i+1, empty_val)
break
pog.add_option(*args, **kwargs)