Develop Reference

Python `argparse`: Is there a clean way to add a flag that sets multiple flags (e.g. `--all`" is equivalent to `--x --y`)

I've got some Python argparse command-line processing code that initially looked like this:
import argparse
ap = argparse.ArgumentParser()
ap.add_argument("--x", help = "Set `x`.", action = "store_true", default = False)
ap.add_argument("--y", help = "Set `y`.", action = "store_true", default = False)
ap.add_argument(
"--all", help = "Equivalent to `--x --y`.",
action = "store_true", default = False
)
args = ap.parse_args()
if args.all:
args.x = True
args.y = True
print "args.x", args.x
print "args.y", args.y
The basic idea: I have some boolean flags that toggle on a particular setting (--x, --y, etc), and I want to add a convenience option that toggles multiple settings on - e.g. --all is equivalent to --x --y.
I wanted to avoid having any command-line processing logic that was not contained within the ArgumentParser and done in parse_args, so I came up with this solution using custom argparse.Actions:
import argparse
def store_const_multiple(const, *destinations):
"""Returns an `argparse.Action` class that sets multiple argument
destinations (`destinations`) to `const`."""
class store_const_multiple_action(argparse.Action):
def __init__(self, *args, **kwargs):
super(store_const_multiple_action, self).__init__(
metavar = None, nargs = 0, const = const, *args, **kwargs
)
def __call__(self, parser, namespace, values, option_string = None):
for destination in destinations:
setattr(namespace, destination, const)
return store_const_multiple_action
def store_true_multiple(*destinations):
"""Returns an `argparse.Action` class that sets multiple argument
destinations (`destinations`) to `True`."""
return store_const_multiple(True, *destinations)
ap = argparse.ArgumentParser()
ap.add_argument("--x", help = "Set `x`.", action = "store_true", default = False)
ap.add_argument("--y", help = "Set `y`.", action = "store_true", default = False)
ap.add_argument(
"--all", help = "Equivalent to `--x --y`.",
action = store_true_multiple("x", "y")
)
args = ap.parse_args()
print "args.x", args.x
print "args.y", args.y
Is there any clean way of achieving what I want with argparse without either (0) doing some processing after parse_args() (first example) or (2) writing a custom argparse.Action (second example)?

This is late, and not exactly what you wanted, but here is something you could try:
import argparse
myflags = ['x', 'y', 'z']
parser = argparse.ArgumentParser()
parser.add_argument('--flags', nargs="+", choices=myflags)
parser.add_argument('--all-flags', action='store_const', const=myflags, dest='flags')
args = parser.parse_args()
print(args)
Then calling python myscript.py --flags x y outputs this
Namespace(flags=['x', 'y'])
And calling python myscript.py --all-flags outputs this
Namespace(flags=['x', 'y', 'z'])
But you'll have to check your flags via 'x' in args.flags instead of simply args.x

While using Python's add_argument in argparse, how can I throw an exception if a particular deprecated flag is called?

Basically imagine that I have argparser that has multiple arguments.
I have a particular function definition that looks like this:
def add_to_parser(self, parser):
group = parser.add_argument_group('')
group.add_argument( '--deprecateThis', action='throw exception', help='Stop using this. this is deprecated')
Whether I can try and create that action to throw an exception and stop the code or if I can wrap it to check for the deprecateThis flag and then throw an exception, I'd like to know how to do it and which is best! Thanks.

Here's what I came up with:
You can register custom actions for your arguments, I registered one to print out a deprecation warning and remove the item from the resulting namespace:
class DeprecateAction(argparse.Action):
def __init__(self, *args, **kwargs):
self.call_count = 0
if 'help' in kwargs:
kwargs['help'] = f'[DEPRECATED] {kwargs["help"]}'
super().__init__(*args, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
if self.call_count == 0:
sys.stderr.write(f"The option `{option_string}` is deprecated. It will be ignored.\n")
sys.stderr.write(self.help + '\n')
delattr(namespace, self.dest)
self.call_count += 1
if __name__ == "__main__":
my_parser = ArgumentParser('this is the description')
my_parser.register('action', 'ignore', DeprecateAction)
my_parser.add_argument(
'-f', '--foo',
help="This argument is deprecated",
action='ignore')
args = my_parser.parse_args()
# print(args.foo) # <- would throw an exception

Conditional passing of arguments to methods in python

I have many possible arguments from argparse that I want to pass to a function. If the variable hasn't been set, I want the method to use its default variable. However, handling which arguments have been set and which haven't is tedious:
import argparse
def my_func(a = 1, b = 2):
return a+b
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Get the numeric values.')
parser.add_argument('-a', type=int)
parser.add_argument('-b', type=int)
args = parser.parse_args()
if not args.a is None and not args.b is None:
result = my_func(a = args.a, b = args.b)
elif not args.a is None and args.b is None:
result = my_func(a = args.a)
elif not args.b is None and args.a is None:
result = my_func(b = args.b)
else:
result = my_func()
It seems like I should be able to do something like this:
result = my_func(a = args.a if not args.a is None, b = args.b if not args.b is None)
But this gives a syntax error on the comma.
I could set default values in the argparser, but I want to use the defaults set in the method definition.

Use a dictionary with the kwargs unpacking syntax.
args = parser.parse_args()
result = my_func(**vars(args))
Edit
Use the SUPPRESS argument to ArgumentParser to remove empty values:
parser = argparse.ArgumentParser(description='Get the numeric values.',
argument_default=argparse.SUPPRESS)

The first solution that comes to me seems kind of hacky...but here it is.
Use inspect to write a function that looks at the arguments of a function and only passes it those arguments from args which it accepts and are not None. My guess is that this would be widely considered bad practice...
import inspect
def call_function(fn, args):
argspec = inspect.getargspec(fn)
arglist = {}
for arg in argspec.args:
if arg in args.__dict__.keys() and args.__dict__[arg] is not None:
arglist[arg] = args.__dict__[arg]
return fn(**arglist)
Here it is in action:
import argparse
def my_func(a=1, c=2):
return a,c
a=None
b=2
c=3
args=argparse.Namespace(a=a,b=b,c=c)
call_function(my_func, args)
>> (1, 3)
This solution is quite under-tested and might need work to make it more robust, but the idea is there and should work in simple cases.

argparse argument order

I have a little problem.
I use argparse to parse my arguments, and it's working very well.
To have the args, I do :
p_args = parser.parse_args(argv)
args = dict(p_args._get_kwargs())
But the problem with p_args is that I don't know how to get these arguments ordered by their position in the command line, because it's a dict.
So is there any possibility to have the arguments in a tuple/list/ordered dict by their order in the command line?

To keep arguments ordered, I use a custom action like this:
import argparse
class CustomAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if not 'ordered_args' in namespace:
setattr(namespace, 'ordered_args', [])
previous = namespace.ordered_args
previous.append((self.dest, values))
setattr(namespace, 'ordered_args', previous)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', action=CustomAction)
parser.add_argument('--test2', action=CustomAction)
To use it, for example:
>>> parser.parse_args(['--test2', '2', '--test1', '1'])
Namespace(ordered_args=[('test2', '2'), ('test1', '1')], test1=None, test2=None)

If you need to know the order in which the arguments appear in your parser, you can set up the parser like this:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('--optional1')
parser.add_argument('positionals', nargs='+')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
Here's a quick example of running this code:
$ python s.py --optional1 X --optional2 Y 1 2 3 4 5
['1', '2', '3', '4', '5']
Note that args.positionals is a list with the positional arguments in order. See the argparse documentation for more information.

This is a bit fragile since it relies on understanding the internals of argparse.ArgumentParser, but in lieu of rewriting argparse.ArgumentParser.parse_known_args, here's what I use:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_arg_order"] = []
self.__dict__["_arg_order_first_time_through"] = True
argparse.Namespace.__init__(self, **kwargs)
def __setattr__(self, name, value):
#print("Setting %s -> %s" % (name, value))
self.__dict__[name] = value
if name in self._arg_order and hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
self.__dict__["_arg_order"].append(name)
def _finalize(self):
if hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
def _latest_of(self, k1, k2):
try:
print self._arg_order
if self._arg_order.index(k1) > self._arg_order.index(k2):
return k1
except ValueError:
if k1 in self._arg_order:
return k1
return k2
This works through the knowledge that argparse.ArgumentParser.parse_known_args runs through the entire option list once setting default values for each argument. Meaning that user specified arguments begin the first time __setattr__ hits an argument that it's seen before.
Usage:
options, extra_args = parser.parse_known_args(sys.argv, namespace=OrderedNamespace())
You can check options._arg_order for the order of user specified command line args, or use options._latest_of("arg1", "arg2") to see which of --arg1 or --arg2 was specified later on the command line (which, for my purposes was what I needed: seeing which of two options would be the overriding one).
UPDATE: had to add _finalize method to handle pathological case of sys.argv() not containing any arguments in the list)

There is module especially made to handle this :
https://github.com/claylabs/ordered-keyword-args
without using orderedkwargs module
def multiple_kwarguments(first , **lotsofothers):
print first
for i,other in lotsofothers:
print other
return True
multiple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
output:
first
second
fifth
fourth
third
On using orderedkwargs module
from orderedkwargs import ordered kwargs
#orderedkwargs
def mutliple_kwarguments(first , *lotsofothers):
print first
for i, other in lotsofothers:
print other
return True
mutliple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
Output:
first
second
third
fourth
fifth
Note: Single asterik is required while using this module with decorator above the function.

I needed this because, for logging purposes, I liked to print the arguments after they were parsed. The problem was that the arguments are not printed in order, which was really annoying.
The custom action class just flat out did not work for me. I had other arguments which used a different action such as 'store_true' and default arguments also don't work since the custom action class is not called if the argument is not given in the command line. What worked for me was creating a wrapper class like this:
import collections
from argparse import ArgumentParser
class SortedArgumentParser():
def __init__(self, *args, **kwargs):
self.ap = ArgumentParser(*args, **kwargs)
self.args_dict = collections.OrderedDict()
def add_argument(self, *args, **kwargs):
self.ap.add_argument(*args, **kwargs)
# Also store dest kwarg
self.args_dict[kwargs['dest']] = None
def parse_args(self):
# Returns a sorted dictionary
unsorted_dict = self.ap.parse_args().__dict__
for unsorted_entry in unsorted_dict:
self.args_dict[unsorted_entry] = unsorted_dict[unsorted_entry]
return self.args_dict
The pros are that the add_argument method should have the exact same functionality as the original ArgumentParser. The cons are that if you want other methods you will have to write wrapped for all of them. Luckily for me all I ever used was add_argument and parse_args, so this served my purposes pretty well. You would also need to do more work if you wanted to use parent ArgumentParsers.

This is my simple solution based on the existing ones:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_order"] = [None]
super().__init__(**kwargs)
def __setattr__(self, attr, value):
super().__setattr__(attr, value)
if attr in self._order:
self.__dict__["_order"].clear()
self.__dict__["_order"].append(attr)
def ordered(self):
if self._order and self._order[0] is None:
self._order.clear()
return ((attr, getattr(self, attr)) for attr in self._order)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', default=1)
parser.add_argument('--test2')
parser.add_argument('-s', '--slong', action='store_false')
parser.add_argument('--test3', default=3)
args = parser.parse_args(['--test2', '2', '--test1', '1', '-s'], namespace=OrderedNamespace())
print(args)
print(args.test1)
for a, v in args.ordered():
print(a, v)
Output:
OrderedNamespace(_order=['test2', 'test1', 'slong'], slong=False, test1='1', test2='2', test3=3)
1
test2 2
test1 1
slong False
It allows actions in add_argument(), which is harder for customized action class solution.

Default sub-command, or handling no sub-command with argparse

How can I have a default sub-command, or handle the case where no sub-command is given using argparse?
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
a.parse_args()
Here I'd like a command to be selected, or the arguments to be handled based only on the next highest level of parser (in this case the top-level parser).
joiner#X:~/src> python3 default_subcommand.py
usage: default_subcommand.py [-h] {hi} ...
default_subcommand.py: error: too few arguments

On Python 3.2 (and 2.7) you will get that error, but not on 3.3 and 3.4 (no response). Therefore on 3.3/3.4 you could test for parsed_args to be an empty Namespace.
A more general solution is to add a method set_default_subparser() (taken from the ruamel.std.argparse package) and call that method just before parse_args():
import argparse
import sys
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if not subparser_found:
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
def do_hi():
print('inside hi')
a = argparse.ArgumentParser()
b = a.add_subparsers()
sp = b.add_parser('hi')
sp.set_defaults(func=do_hi)
a.set_default_subparser('hi')
parsed_args = a.parse_args()
if hasattr(parsed_args, 'func'):
parsed_args.func()
This will work with 2.6 (if argparse is installed from PyPI), 2.7, 3.2, 3.3, 3.4. And allows you to do both
python3 default_subcommand.py
and
python3 default_subcommand.py hi
with the same effect.
Allowing to chose a new subparser for default, instead of one of the existing ones.
The first version of the code allows setting one of the previously-defined subparsers as a default one. The following modification allows adding a new default subparser, which could then be used to specifically process the case when no subparser was selected by user (different lines marked in the code)
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
existing_default = False # check if default parser previously defined
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if sp_name == name: # check existance of default parser
existing_default = True
if not subparser_found:
# If the default subparser is not among the existing ones,
# create a new parser.
# As this is called just before 'parse_args', the default
# parser created here will not pollute the help output.
if not existing_default:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
x.add_parser(name)
break # this works OK, but should I check further?
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
a = argparse.ArgumentParser()
b = a.add_subparsers(dest ='cmd')
sp = b.add_parser('hi')
sp2 = b.add_parser('hai')
a.set_default_subparser('hey')
parsed_args = a.parse_args()
print(parsed_args)
The "default" option will still not show up in the help:
python test_parser.py -h
usage: test_parser.py [-h] {hi,hai} ...
positional arguments:
{hi,hai}
optional arguments:
-h, --help show this help message and exit
However, it is now possible to differentiate between and separately handle calling one of the provided subparsers, and calling the default subparser when no argument was provided:
$ python test_parser.py hi
Namespace(cmd='hi')
$ python test_parser.py
Namespace(cmd='hey')

It seems I've stumbled on the solution eventually myself.
If the command is optional, then this makes the command an option. In my original parser configuration, I had a package command that could take a range of possible steps, or it would perform all steps if none was given. This makes the step a choice:
parser = argparse.ArgumentParser()
command_parser = subparsers.add_parser('command')
command_parser.add_argument('--step', choices=['prepare', 'configure', 'compile', 'stage', 'package'])
...other command parsers
parsed_args = parser.parse_args()
if parsed_args.step is None:
do all the steps...

Here's a nicer way of adding a set_default_subparser method:
class DefaultSubcommandArgParse(argparse.ArgumentParser):
__default_subparser = None
def set_default_subparser(self, name):
self.__default_subparser = name
def _parse_known_args(self, arg_strings, *args, **kwargs):
in_args = set(arg_strings)
d_sp = self.__default_subparser
if d_sp is not None and not {'-h', '--help'}.intersection(in_args):
for x in self._subparsers._actions:
subparser_found = (
isinstance(x, argparse._SubParsersAction) and
in_args.intersection(x._name_parser_map.keys())
)
if subparser_found:
break
else:
# insert default in first position, this implies no
# global options without a sub_parsers specified
arg_strings = [d_sp] + arg_strings
return super(DefaultSubcommandArgParse, self)._parse_known_args(
arg_strings, *args, **kwargs
)

Maybe what you're looking for is the dest argument of add_subparsers:
(Warning: works in Python 3.4, but not in 2.7)
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='cmd')
parser_hi = subparsers.add_parser('hi')
parser.parse_args([]) # Namespace(cmd=None)
Now you can just use the value of cmd:
if cmd in [None, 'hi']:
print('command "hi"')

You can duplicate the default action of a specific subparser on the main parser, effectively making it the default.
import argparse
p = argparse.ArgumentParser()
sp = p.add_subparsers()
a = sp.add_parser('a')
a.set_defaults(func=do_a)
b = sp.add_parser('b')
b.set_defaults(func=do_b)
p.set_defaults(func=do_b)
args = p.parse_args()
if args.func:
args.func()
else:
parser.print_help()
Does not work with add_subparsers(required=True), which is why the if args.func is down there.

In my case I found it easiest to explicitly provide the subcommand name to parse_args() when argv was empty.
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='commands')
runParser = subparsers.add_parser('run', help='[DEFAULT ACTION]')
altParser = subparsers.add_parser('alt', help='Alternate command')
altParser.add_argument('alt_val', type=str, help='Value required for alt command.')
# Here's my shortcut: If `argv` only contains the script name,
# manually inject our "default" command.
args = parser.parse_args(['run'] if len(sys.argv) == 1 else None)
print args
Example runs:
$ ./test.py
Namespace()
$ ./test.py alt blah
Namespace(alt_val='blah')
$ ./test.py blah
usage: test.py [-h] {run,alt} ...
test.py: error: invalid choice: 'blah' (choose from 'run', 'alt')

In python 2.7, you can override the error behaviour in a subclass (a shame there isn't a nicer way to differentiate the error):
import argparse
class ExceptionArgParser(argparse.ArgumentParser):
def error(self, message):
if "invalid choice" in message:
# throw exception (of your choice) to catch
raise RuntimeError(message)
else:
# restore normal behaviour
super(ExceptionArgParser, self).error(message)
parser = ExceptionArgParser()
subparsers = parser.add_subparsers(title='Modes', dest='mode')
default_parser = subparsers.add_parser('default')
default_parser.add_argument('a', nargs="+")
other_parser = subparsers.add_parser('other')
other_parser.add_argument('b', nargs="+")
try:
args = parser.parse_args()
except RuntimeError:
args = default_parser.parse_args()
# force the mode into namespace
setattr(args, 'mode', 'default')
print args

Here's another solution using a helper function to build a list of known subcommands:
import argparse
def parse_args(argv):
parser = argparse.ArgumentParser()
commands = []
subparsers = parser.add_subparsers(dest='command')
def add_command(name, *args, **kwargs):
commands.append(name)
return subparsers.add_parser(name, *args, **kwargs)
hi = add_command("hi")
hi.add_argument('--name')
add_command("hola")
# check for default command
if not argv or argv[0] not in commands:
argv.insert(0, "hi")
return parser.parse_args(argv)
assert parse_args([]).command == 'hi'
assert parse_args(['hi']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).name == 'John'
assert parse_args(['--name', 'John']).command == 'hi'
assert parse_args(['hola']).command == 'hola'

You can add an argument with a default value that will be used when nothing is set I believe.
See this: http://docs.python.org/dev/library/argparse.html#default
Edit:
Sorry, I read your question a bit fast.
I do not think you would have a direct way of doing what you want via argparse. But you could check the length of sys.argv and if its length is 1 (only script name) then you could manually pass the default parameters for parsing, doing something like this:
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
if len(sys.argv) == 1:
a.parse_args(['hi'])
else:
a.parse_args()
I think that should do what you want, but I agree it would be nice to have this out of the box.

For later reference:
...
b = a.add_subparsers(dest='cmd')
b.set_defaults(cmd='hey') # <-- this makes hey as default
b.add_parser('hi')
so, these two will be same:
python main.py hey
python main.py