Develop Reference

Resolve argparse alias back to the original command

I'm using a subparser/subcommand that has an alias.
I'm using the dest option for the subparser to store the name of the subcommand so I can get it later.
Currently if the subcommand's name is reallyLongName and the alias is r (say) then the dest option stores either reallyLongName or r exactly - whatever I typed in gets stored. This is annoying because I now have to check for the name of the command or any of its aliases in order to identify the command.
Is there a way to get argparse to store the subcommand's name in the dest field in some sort of single, canonical text string?
For example, given the following code:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='command', help='sub-command help')
parser_ag = subparsers.add_parser( 'mySubcommand',
aliases=['m'],
help='Subcommand help')
print(parser.parse_args('mySubcommand'.split()))
print(parser.parse_args('m'.split()))
the following output is produced:
Namespace(command='mySubcommand')
Namespace(command='m')
Desired result: command has a single, canonical value for both, for example:
Namespace(command='mySubcommand')
Namespace(command='mySubcommand')

There was a Python bug/issue requesting this - saving the 'base' name, rather than the alias. You can't change that without changing argparse.py code. I think the change would limited to the Action subclass that handles subparsers. https://bugs.python.org/issue36664
But I point out that there's simpler way of handling this. Just use set_defaults as documented near the end of the https://docs.python.org/3/library/argparse.html#sub-commands section. There
parser_foo.set_defaults(func=foo)
is used to set a subparser specific function, but it could just as well be used to set the 'base' name.
parser_foo.set_defaults(name='theIncrediblyLongAlias')

This was surprisingly difficult to dig out. When you add a subparser, it gets stored in the parents ._actions attribute. From there it is just digging through attributes to get what you need. Below I create dictionaries to reference the subparser arguments by the dest name, and then added a function that lets us remap the inputted arguments to the primary argument name.
from collections import defaultdict
def get_subparser_aliases(parser, dest):
out = defaultdict(list)
prog_str = parser.prog
dest_dict = {a.dest: a for a in parser._actions}
try:
choices = dest_dict.get(dest).choices
except AttributeError:
raise AttributeError(f'The parser "{parser}" has no subparser with a `dest` of "{dest}"')
for k, v in choices.items():
clean_v = v.prog.replace(prog_str, '', 1).strip()
out[clean_v].append(k)
return dict(out)
def remap_args(args, mapping, dest):
setattr(args, dest, mapping.get(getattr(args, dest)))
return args
Using your example, we can remap the parse args using:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='command', help='sub-command help')
parser_ag = subparsers.add_parser('mySubcommand',
aliases=['m'],
help='Subcommand help')
args = parser.parse_args('m'.split())
mapping = get_subparser_aliases(parser, 'command')
remap_args(args, mapping, 'command')
print(args)
# prints:
Namespace(command='mySubcommand')
Here is an example of it at work with multiple subparser levels.. We have a parser with an optional argument and a subparser. The subparser has 3 possible arguments, the last of which invoke another subparser (a sub-subparser), with 2 possible arguments.
You can examine either the top level parser or the first level subparser to see alias mappings.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--someoption', '-s', action='store_true')
subparser1 = parser.add_subparsers(help='sub-command help', dest='sub1')
parser_r = subparser1.add_parser('reallyLongName', aliases=['r'])
parser_r.add_argument('foo', type=int, help='foo help')
parser_s = subparser1.add_parser('otherReallyLong', aliases=['L'])
parser_s.add_argument('bar', choices='abc', help='bar help')
parser_z = subparser1.add_parser('otherOptions', aliases=['oo'])
subparser2 = parser_z.add_subparsers(help='sub-sub-command help', dest='sub2')
parser_x = subparser2.add_parser('xxx', aliases=['x'])
parser_x.add_argument('fizz', type=float, help='fizz help')
parser_y = subparser2.add_parser('yyy', aliases=['y'])
parser_y.add_argument('blip', help='blip help')
get_subparser_aliases(parser, 'sub1')
# returns:
{'reallyLongName': ['reallyLongName', 'r'],
'otherReallyLong': ['otherReallyLong', 'L'],
'otherOptions': ['otherOptions', 'oo']}
get_subparser_aliases(parser_z, 'sub2')
# returns:
{'xxx': ['xxx', 'x'], 'yyy': ['yyy', 'y']}
Using this with the function above, we can remap the collected args to their longer names.
args = parser.parse_args('-s oo x 1.23'.split())
print(args)
# prints:
Namespace(fizz=1.23, someoption=True, sub1='oo', sub2='x')
for p, dest in zip((parser, parser_z), ('sub1', 'sub2')):
mapping = get_subparser_aliases(p, dest)
remap_args(args, mapping, dest)
print(args)
# prints:
Namespace(fizz=1.23, someoption=True, sub1='otherOptions', sub2='xxx')

Add top level argparse arguments after subparser args

How can you allow for top-level program arguments to be added after using a subcommand from a subparser?
I have a program that includes several subparsers to allow for subcommands, changing the behavior of the program. Here is an example of how its set up:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import argparse
def task_a():
print('did task_a')
def task_c():
print('did task_c')
def task_d():
print('did task_d')
def run_foo(args):
a_arg = args.a
c_arg = args.c
if a_arg:
task_a()
if c_arg:
task_c()
def run_bar(args):
a_arg = args.a
d_arg = args.d
if a_arg:
task_a()
if d_arg:
task_d()
def parse():
'''
Run the program
arg parsing goes here, if program was run as a script
'''
# create the top-level parser
parser = argparse.ArgumentParser()
# add top-level args
parser.add_argument("-a", default = False, action = "store_true", dest = 'a')
# add subparsers
subparsers = parser.add_subparsers(title='subcommands', description='valid subcommands', help='additional help', dest='subparsers')
# create the parser for the "foo" command
parser_foo = subparsers.add_parser('foo')
parser_foo.set_defaults(func = run_foo)
parser_foo.add_argument("-c", default = False, action = "store_true", dest = 'c')
# create the parser for the "bar" downstream command
parser_bar = subparsers.add_parser('bar')
parser_bar.set_defaults(func = run_bar)
parser_bar.add_argument("-d", default = False, action = "store_true", dest = 'd')
# parse the args and run the default parser function
args = parser.parse_args()
args.func(args)
if __name__ == "__main__":
parse()
When I run the program I can call a subcommand with its args like this:
$ ./subparser_order.py bar -d
did task_d
$ ./subparser_order.py foo -c
did task_c
But if I want to include the args from the top level, I have to call it like this:
$ ./subparser_order.py -a foo -c
did task_a
did task_c
However, I think this is confusing, especially if there are many top-level args and many subcommand args; the subcommand foo is sandwiched in the middle here and harder to discern.
I would rather be able to call the program like subparser_order.py foo -c -a, but this does not work:
$ ./subparser_order.py foo -c -a
usage: subparser_order.py [-h] [-a] {foo,bar} ...
subparser_order.py: error: unrecognized arguments: -a
In fact, you cannot call the top-level args at all after specifying a subcommand:
$ ./subparser_order.py foo -a
usage: subparser_order.py [-h] [-a] {foo,bar} ...
subparser_order.py: error: unrecognized arguments: -a
Is there a solution that will allow for the top-level args to be included after the subcommand?

There is actually a way to do it. You can use parse_known_args, take the namespace and unparsed arguments and pass these back to a parse_args call. It will combine and override in the 2nd pass and any left over arguments from there on will still throw parser errors.
Simple example, here is the setup:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_true')
sp = parser.add_subparsers(dest='subargs')
sp_1 = sp.add_parser('foo')
sp_1.add_argument('-b', action='store_true')
print(parser.parse_args())
In the proper order for argparse to work:
- $ python3 argparse_multipass.py
Namespace(a=False, subargs=None)
- $ python3 argparse_multipass.py -a
Namespace(a=True, subargs=None)
- $ python3 argparse_multipass.py -a foo
Namespace(a=True, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo
Namespace(a=False, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo -b
Namespace(a=False, b=True, subargs='foo')
- $ python3 argparse_multipass.py -a foo -b
Namespace(a=True, b=True, subargs='foo')
Now, you can't parse arguments after a subparser kicks in:
- $ python3 argparse_multipass.py foo -b -a
usage: argparse_multipass.py [-h] [-a] {foo} ...
argparse_multipass.py: error: unrecognized arguments: -a
However, you can do a multi-pass to get your arguments back. Here is the updated code:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_true')
sp = parser.add_subparsers(dest='subargs')
sp_1 = sp.add_parser('foo')
sp_1.add_argument('-b', action='store_true')
args = parser.parse_known_args()
print('Pass 1: ', args)
args = parser.parse_args(args[1], args[0])
print('Pass 2: ', args)
And the results for it:
- $ python3 argparse_multipass.py
Pass 1: (Namespace(a=False, subargs=None), [])
Pass 2: Namespace(a=False, subargs=None)
- $ python3 argparse_multipass.py -a
Pass 1: (Namespace(a=True, subargs=None), [])
Pass 2: Namespace(a=True, subargs=None)
- $ python3 argparse_multipass.py -a foo
Pass 1: (Namespace(a=True, b=False, subargs='foo'), [])
Pass 2: Namespace(a=True, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo
Pass 1: (Namespace(a=False, b=False, subargs='foo'), [])
Pass 2: Namespace(a=False, b=False, subargs='foo')
- $ python3 argparse_multipass.py foo -b
Pass 1: (Namespace(a=False, b=True, subargs='foo'), [])
Pass 2: Namespace(a=False, b=True, subargs='foo')
- $ python3 argparse_multipass.py -a foo -b
Pass 1: (Namespace(a=True, b=True, subargs='foo'), [])
Pass 2: Namespace(a=True, b=True, subargs='foo')
- $ python3 argparse_multipass.py foo -b -a
Pass 1: (Namespace(a=False, b=True, subargs='foo'), ['-a'])
Pass 2: Namespace(a=True, b=True, subargs='foo')
This will maintain original functionality but allow continued parsing for when subparsers kick in. Additionally you could make disordered parsing out of the thing entirely if you do something like this:
ns, ua = parser.parse_known_args()
while len(ua):
ns, ua = parser.parse_known_args(ua, ns)
It will keep parsing arguments in case they are out of order until it has completed parsing all of them. Keep in mind this one will keep going if there is an unknown argument that stays in there. Mind want to add something like this:
pua = None
ns, ua = parser.parse_known_args()
while len(ua) and ua != pua:
ns, ua = parser.parse_known_args(ua, ns)
pua = ua
ns, ua = parser.parse_args(ua, ns)
Just keep a previously unparsed arguments object and compare it, when it breaks do a final parse_args call to make the parser run its own errors path.
It's not the most elegant solution but I ran into the exact same problem where my arguments on the main parser were used as optional flags additionally on top of what was specified in a sub parser.
Keep the following in mind though: This code will make it so a person can specify multiple subparsers and their options in a run, the code that these arguments invoke should be able to deal with that.

Once the top level parser encounters 'foo' it delegates parsing to parser_foo. That modifies the args namespace, and returns. The top level parser does not resume parsing. It just handles any errors returned by the subparser.
In [143]: parser=argparse.ArgumentParser()
In [144]: parser.add_argument('-a', action='store_true');
In [145]: sp = parser.add_subparsers(dest='cmd')
In [146]: sp1 = sp.add_parser('foo')
In [147]: sp1.add_argument('-c', action='store_true');
In [148]: parser.parse_args('-a foo -c'.split())
Out[148]: Namespace(a=True, c=True, cmd='foo')
In [149]: parser.parse_args('foo -c'.split())
Out[149]: Namespace(a=False, c=True, cmd='foo')
In [150]: parser.parse_args('foo -c -a'.split())
usage: ipython3 [-h] [-a] {foo} ...
ipython3: error: unrecognized arguments: -a
You can keep it from choking on the unrecognized argument, but it won't resume parsing:
In [151]: parser.parse_known_args('foo -c -a'.split())
Out[151]: (Namespace(a=False, c=True, cmd='foo'), ['-a'])
You could also add an argument with the same flag/dest to the subparser.
In [153]: sp1.add_argument('-a', action='store_true')
In [154]: parser.parse_args('foo -c -a'.split())
Out[154]: Namespace(a=True, c=True, cmd='foo')
but the default for the sub entry overrides the toplevel value (there has been bug/issue discussion over this behavior).
In [155]: parser.parse_args('-a foo -c'.split())
Out[155]: Namespace(a=False, c=True, cmd='foo')
It might be possible to parse that extra string with a two stage parser, or with a custom _SubParsersAction class. But with the argparse as it is, there isn't an easy way around this behavior.

argparse: Default value for nargs='*'?

I tried to use this statement :
parser = argparse.ArgumentParser()
parser.add_argument('-m', '--music', nargs='*', default=False, const=True)
args = parser.parse_args()
print(args.music)
But got this error:
`builtins.ValueError: nargs must be '?' to supply const`
what i want to do is :
if -m is in args list but whithout any value, args.music will give me True
if -m is in args list and have 'N' values, args.music will give me a list of all values
if -m is not in args list, args,music will return False
the second and lastOne worked but, when i try to use const i got an error

You could use a custom action:
import argparse
class EmptyIsTrue(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if len(values) == 0:
values = True
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser()
parser.add_argument('-m', '--music', nargs='*', default=False, action=EmptyIsTrue)
print(parser.parse_args([]))
# Namespace(music=False)
print(parser.parse_args(['-m']))
# Namespace(music=True)
print(parser.parse_args('-m 1 2'.split()))
# Namespace(music=['1', '2'])
If you have only one argument to handle this way, then
arg.music = True if len(arg.music) == 0 else arg.music
is simpler. If you have many such arguments, then defining a custom action could reduce the repetition, and help ensure all those arguments are treated the same way.

What about :
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-m', '--music', nargs='*', default=False)
args = parser.parse_args()
if vars(args).get('music', False) is not False:
if not args.music:
args.music = True
print args.music
Output:
tmp:/>python arg.py
False
tmp:/>python arg.py -m
True
tmp:/>python arg.py -m 1 2 3
['1', '2', '3']
tmp:/>

The following hack after the argparse section solves your problem:
import argparse
# Same as your code above
parser = argparse.ArgumentParser()
parser.add_argument('-m', '--music', nargs='*', default=False)
args = parser.parse_args()
# Modifies args.music: [] -> True
args.music = True if args.music==[] else args.music
print(args.music)
Tested in the command line, it gives:
$ python /tmp/blah.py -m
True
$ python /tmp/blah.py -m 1 -m 2
['2']
$ python /tmp/blah.py -m 1 2 3
['1', '2', '3']
$ python /tmp/blah.py
False

Setting options from environment variables when using argparse

I have a script which has certain options that can either be passed on the command line, or from environment variables. The CLI should take precedence if both are present, and an error occur if neither are set.
I could check that the option is assigned after parsing, but I prefer to let argparse to do the heavy lifting and be responsible for displaying the usage statement if parsing fails.
I have come up with a couple of alternative approaches to this (which I will post below as answers so they can be discussed separately) but they feel pretty kludgey to me and I think that I am missing something.
Is there an accepted "best" way of doing this?
(Edit to make the desired behaviour clear when both the CLI option and environment variable are unset)

You can set the default= of the argument to a .get() of os.environ with the environment variable you want to grab.
You can also pass a 2nd argument in the .get() call, which is the default value if .get() doesn't find an environment variable by that name (by default .get() returns None in that case).
import argparse
import os
parser = argparse.ArgumentParser(description='test')
parser.add_argument('--url', default=os.environ.get('URL'))
args = parser.parse_args()
if not args.url:
exit(parser.print_usage())

I use this pattern frequently enough that I have packaged a simple action class to handle it:
import argparse
import os
class EnvDefault(argparse.Action):
def __init__(self, envvar, required=True, default=None, **kwargs):
if not default and envvar:
if envvar in os.environ:
default = os.environ[envvar]
if required and default:
required = False
super(EnvDefault, self).__init__(default=default, required=required,
**kwargs)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values)
I can then call this from my code with:
import argparse
from envdefault import EnvDefault
parser=argparse.ArgumentParser()
parser.add_argument(
"-u", "--url", action=EnvDefault, envvar='URL',
help="Specify the URL to process (can also be specified using URL environment variable)")
args=parser.parse_args()

I usually have to do this for multiple arguments (authentication and API keys).. this is simple and straight forward. Uses **kwargs.
def environ_or_required(key):
return (
{'default': os.environ.get(key)} if os.environ.get(key)
else {'required': True}
)
parser.add_argument('--thing', **environ_or_required('THING'))

ConfigArgParse adds support for environment variables to argparse, so you can do things like:
p = configargparse.ArgParser()
p.add('-m', '--moo', help='Path of cow', env_var='MOO_PATH')
options = p.parse_args()

One option is to check whether the environment variable is set, and to modify the calls to add_argument accordingly
e.g.
import argparse
import os
parser=argparse.ArgumentParser()
if 'CVSWEB_URL' in os.environ:
cvsopt = { 'default': os.environ['CVSWEB_URL'] }
else:
cvsopt = { 'required': True }
parser.add_argument(
"-u", "--cvsurl", help="Specify url (overrides CVSWEB_URL environment variable)",
**cvsopt)
args=parser.parse_args()

The topic is quite old, but I had similar problem and I thought I would share my solution with you. Unfortunately custom action solution suggested by #Russell Heilling doesn't work for me for couple of reasons:
It prevents me from using predefined actions (like store_true)
I would rather like it to fallback to default when envvar is not in os.environ (that could be easily fixed)
I would like to have this behaviour for all of my arguments without specifying action or envvar (which should always be action.dest.upper())
Here's my solution (in Python 3):
class CustomArgumentParser(argparse.ArgumentParser):
class _CustomHelpFormatter(argparse.ArgumentDefaultsHelpFormatter):
def _get_help_string(self, action):
help = super()._get_help_string(action)
if action.dest != 'help':
help += ' [env: {}]'.format(action.dest.upper())
return help
def __init__(self, *, formatter_class=_CustomHelpFormatter, **kwargs):
super().__init__(formatter_class=formatter_class, **kwargs)
def _add_action(self, action):
action.default = os.environ.get(action.dest.upper(), action.default)
return super()._add_action(action)

There is an example use-case for ChainMap where you merge together defaults, environment variables and command line arguments.
import os, argparse
defaults = {'color': 'red', 'user': 'guest'}
parser = argparse.ArgumentParser()
parser.add_argument('-u', '--user')
parser.add_argument('-c', '--color')
namespace = parser.parse_args()
command_line_args = {k:v for k, v in vars(namespace).items() if v}
combined = ChainMap(command_line_args, os.environ, defaults)
Came to me from a great talk about beautiful and idiomatic python.
However, I'm not sure how to go about the difference of lower- and uppercase dictionary keys. In the case where both -u foobar is passed as an argument and environment is set to USER=bazbaz, the combined dictionary will look like {'user': 'foobar', 'USER': 'bazbaz'}.

Thought I'd post my solution as the original question/answer gave me a lot of help.
My problem is a little different to Russell's. I'm using OptionParser and instead of an environmental variable for each argument I have just one which simulates the command line.
i.e.
MY_ENVIRONMENT_ARGS = --arg1 "Maltese" --arg2 "Falcon" -r "1930" -h
Solution:
def set_defaults_from_environment(oparser):
if 'MY_ENVIRONMENT_ARGS' in os.environ:
environmental_args = os.environ[ 'MY_ENVIRONMENT_ARGS' ].split()
opts, _ = oparser.parse_args( environmental_args )
oparser.defaults = opts.__dict__
oparser = optparse.OptionParser()
oparser.add_option('-a', '--arg1', action='store', default="Consider")
oparser.add_option('-b', '--arg2', action='store', default="Phlebas")
oparser.add_option('-r', '--release', action='store', default='1987')
oparser.add_option('-h', '--hardback', action='store_true', default=False)
set_defaults_from_environment(oparser)
options, _ = oparser.parse_args(sys.argv[1:])
Here I don't throw an error if an argument is not found. But if I wish to I could just do something like
for key in options.__dict__:
if options.__dict__[key] is None:
# raise error/log problem/print to console/etc

You can use OptionParser()
from optparse import OptionParser
def argument_parser(self, parser):
parser.add_option('--foo', dest="foo", help="foo", default=os.environ.get('foo', None))
parser.add_option('--bar', dest="bar", help="bar", default=os.environ.get('bar', None))
return(parser.parse_args())
parser = OptionParser()
(options, args) = argument_parser(parser)
foo = options.foo
bar = options.bar
print("foo: {}".format(foo))
print("bar: {}".format(bar))
shell:
export foo=1
export bar=2
python3 script.py

The Click library handles this explicitly:
import click
#click.command()
#click.argument('src', envvar='SRC', type=click.File('r'))
def echo(src):
"""Print value of SRC environment variable."""
click.echo(src.read())
And from the command line:
$ export SRC=hello.txt
$ echo
Hello World!
https://click.palletsprojects.com/en/master/arguments/#environment-variables
You can install it with
pip install click

Here's a relatively simple (looks longer because it's well-commented) yet complete solution that avoids kludging default by using the namespace argument of parse_args. By default it parses environment variables no differently than command-line arguments though that can easily be changed.
import shlex
# Notes:
# * Based on https://github.com/python/cpython/blob/
# 15bde92e47e824369ee71e30b07f1624396f5cdc/
# Lib/argparse.py
# * Haven't looked into handling "required" for mutually exclusive groups
# * Probably should make new attributes private even though it's ugly.
class EnvArgParser(argparse.ArgumentParser):
# env_k: The keyword to "add_argument" as well as the attribute stored
# on matching actions.
# env_f: The keyword to "add_argument". Defaults to "env_var_parse" if
# not provided.
# env_i: Basic container type to identify unfilled arguments.
env_k = "env_var"
env_f = "env_var_parse"
env_i = type("env_i", (object,), {})
def add_argument(self, *args, **kwargs):
map_f = (lambda m,k,f=None,d=False:
(k, k in m, m.pop(k,f) if d else m.get(k,f)))
env_k = map_f(kwargs, self.env_k, d=True, f="")
env_f = map_f(kwargs, self.env_f, d=True, f=self.env_var_parse)
if env_k[1] and not isinstance(env_k[2], str):
raise ValueError(f"Parameter '{env_k[0]}' must be a string.")
if env_f[1] and not env_k[1]:
raise ValueError(f"Parameter '{env_f[0]}' requires '{env_k[0]}'.")
if env_f[1] and not callable(env_f[2]):
raise ValueError(f"Parameter '{env_f[0]}' must be callable.")
action = super().add_argument(*args, **kwargs)
if env_k[1] and not action.option_strings:
raise ValueError(f"Positional parameters may not specify '{env_k[0]}'.")
# We can get the environment now:
# * We need to know now if the keys exist anyway
# * os.environ is static
env_v = map_f(os.environ, env_k[2], f="")
# Examples:
# env_k:
# ("env_var", True, "FOO_KEY")
# env_v:
# ("FOO_KEY", False, "")
# ("FOO_KEY", True, "FOO_VALUE")
#
# env_k:
# ("env_var", False, "")
# env_v:
# ("" , False, "")
# ("", True, "RIDICULOUS_VALUE")
# Add the identifier to all valid environment variable actions for
# later access by i.e. the help formatter.
if env_k[1]:
if env_v[1] and action.required:
action.required = False
i = self.env_i()
i.a = action
i.k = env_k[2]
i.f = env_f[2]
i.v = env_v[2]
i.p = env_v[1]
setattr(action, env_k[0], i)
return action
# Overriding "_parse_known_args" is better than "parse_known_args":
# * The namespace will already have been created.
# * This method runs in an exception handler.
def _parse_known_args(self, arg_strings, namespace):
"""precedence: cmd args > env var > preexisting namespace > defaults"""
for action in self._actions:
if action.dest is argparse.SUPPRESS:
continue
try:
i = getattr(action, self.env_k)
except AttributeError:
continue
if not i.p:
continue
setattr(namespace, action.dest, i)
namespace, arg_extras = super()._parse_known_args(arg_strings, namespace)
for k,v in vars(namespace).copy().items():
# Setting "env_i" on the action is more effective than using an
# empty unique object() and mapping namespace attributes back to
# actions.
if isinstance(v, self.env_i):
fv = v.f(v.a, v.k, v.v, arg_extras)
if fv is argparse.SUPPRESS:
delattr(namespace, k)
else:
# "_parse_known_args::take_action" checks for action
# conflicts. For simplicity we don't.
v.a(self, namespace, fv, v.k)
return (namespace, arg_extras)
def env_var_parse(self, a, k, v, e):
# Use shlex, yaml, whatever.
v = shlex.split(v)
# From "_parse_known_args::consume_optional".
n = self._match_argument(a, "A"*len(v))
# From the main loop of "_parse_known_args". Treat additional
# environment variable arguments just like additional command-line
# arguments (which will eventually raise an exception).
e.extend(v[n:])
return self._get_values(a, v[:n])
# Derived from "ArgumentDefaultsHelpFormatter".
class EnvArgHelpFormatter(argparse.HelpFormatter):
"""Help message formatter which adds environment variable keys to
argument help.
"""
env_k = EnvArgParser.env_k
# This is supposed to return a %-style format string for "_expand_help".
# Since %-style strings don't support attribute access we instead expand
# "env_k" ourselves.
def _get_help_string(self, a):
h = super()._get_help_string(a)
try:
i = getattr(a, self.env_k)
except AttributeError:
return h
s = f" ({self.env_k}: {i.k})"
if s not in h:
h += s
return h
# An example mix-in.
class DefEnvArgHelpFormatter\
( EnvArgHelpFormatter
, argparse.ArgumentDefaultsHelpFormatter
):
pass
Example program:
parser = EnvArgParser\
( prog="Test Program"
, formatter_class=DefEnvArgHelpFormatter
)
parser.add_argument\
( '--bar'
, required=True
, env_var="BAR"
, type=int
, nargs="+"
, default=22
, help="Help message for bar."
)
parser.add_argument\
( 'baz'
, type=int
)
args = parser.parse_args()
print(args)
Example program output:
$ BAR="1 2 3 '45 ' 6 7" ./envargparse.py 123
Namespace(bar=[1, 2, 3, 45, 6, 7], baz=123)
$ ./envargparse.py -h
usage: Test Program [-h] --bar BAR [BAR ...] baz
positional arguments:
baz
optional arguments:
-h, --help show this help message and exit
--bar BAR [BAR ...] Help message for bar. (default: 22) (env_var: BAR)

Another option:
parser = argparse.ArgumentParser()
env = os.environ
def add_argument(key, *args, **kwargs):
if key in env:
kwargs['default'] = env[key]
parser.add_argument(*args, **kwargs)
add_argument('--type', type=str)
Or this one, using os.getenv for setting default value:
parser = argparse.ArgumentParser()
parser.add_argument('--type', type=int, default=os.getenv('type',100))

How to parse multiple nested sub-commands using python argparse?

I am implementing a command line program which has interface like this:
cmd [GLOBAL_OPTIONS] {command [COMMAND_OPTS]} [{command [COMMAND_OPTS]} ...]
I have gone through the argparse documentation. I can implement GLOBAL_OPTIONS as optional argument using add_argument in argparse. And the {command [COMMAND_OPTS]} using Sub-commands.
From the documentation it seems I can have only one sub-command. But as you can see I have to implement one or more sub-commands. What is the best way to parse such command line arguments useing argparse?

I came up with the same qustion, and it seems i have got a better answer.
The solution is we shall not simply nest subparser with another subparser, but we can add subparser following with a parser following another subparser.
Code tell you how:
parent_parser = argparse.ArgumentParser(add_help=False)
parent_parser.add_argument('--user', '-u',
default=getpass.getuser(),
help='username')
parent_parser.add_argument('--debug', default=False, required=False,
action='store_true', dest="debug", help='debug flag')
main_parser = argparse.ArgumentParser()
service_subparsers = main_parser.add_subparsers(title="service",
dest="service_command")
service_parser = service_subparsers.add_parser("first", help="first",
parents=[parent_parser])
action_subparser = service_parser.add_subparsers(title="action",
dest="action_command")
action_parser = action_subparser.add_parser("second", help="second",
parents=[parent_parser])
args = main_parser.parse_args()

#mgilson has a nice answer to this question. But problem with splitting sys.argv myself is that i lose all the nice help message Argparse generates for the user. So i ended up doing this:
import argparse
## This function takes the 'extra' attribute from global namespace and re-parses it to create separate namespaces for all other chained commands.
def parse_extra (parser, namespace):
namespaces = []
extra = namespace.extra
while extra:
n = parser.parse_args(extra)
extra = n.extra
namespaces.append(n)
return namespaces
argparser=argparse.ArgumentParser()
subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')
parser_a = subparsers.add_parser('command_a', help = "command_a help")
## Setup options for parser_a
## Add nargs="*" for zero or more other commands
argparser.add_argument('extra', nargs = "*", help = 'Other commands')
## Do similar stuff for other sub-parsers
Now after first parse all chained commands are stored in extra. I reparse it while it is not empty to get all the chained commands and create separate namespaces for them. And i get nicer usage string that argparse generates.

parse_known_args returns a Namespace and a list of unknown strings. This is similar to the extra in the checked answer.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
sub = parser.add_subparsers()
for i in range(1,4):
sp = sub.add_parser('cmd%i'%i)
sp.add_argument('--foo%i'%i) # optionals have to be distinct
rest = '--foo 0 cmd2 --foo2 2 cmd3 --foo3 3 cmd1 --foo1 1'.split() # or sys.argv
args = argparse.Namespace()
while rest:
args,rest = parser.parse_known_args(rest,namespace=args)
print args, rest
produces:
Namespace(foo='0', foo2='2') ['cmd3', '--foo3', '3', 'cmd1', '--foo1', '1']
Namespace(foo='0', foo2='2', foo3='3') ['cmd1', '--foo1', '1']
Namespace(foo='0', foo1='1', foo2='2', foo3='3') []
An alternative loop would give each subparser its own namespace. This allows overlap in positionals names.
argslist = []
while rest:
args,rest = parser.parse_known_args(rest)
argslist.append(args)

The solution provide by #Vikas fails for subcommand-specific optional arguments, but the approach is valid. Here is an improved version:
import argparse
# create the top-level parser
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='foo help')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')
# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')
# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
parser_b.add_argument('--baz', choices='XYZ', help='baz help')
# parse some argument lists
argv = ['--foo', 'command_a', '12', 'command_b', '--baz', 'Z']
while argv:
print(argv)
options, argv = parser.parse_known_args(argv)
print(options)
if not options.subparser_name:
break
This uses parse_known_args instead of parse_args. parse_args aborts as soon as a argument unknown to the current subparser is encountered, parse_known_args returns them as a second value in the returned tuple. In this approach, the remaining arguments are fed again to the parser. So for each command, a new Namespace is created.
Note that in this basic example, all global options are added to the first options Namespace only, not to the subsequent Namespaces.
This approach works fine for most situations, but has three important limitations:
It is not possible to use the same optional argument for different subcommands, like myprog.py command_a --foo=bar command_b --foo=bar.
It is not possible to use any variable length positional arguments with subcommands (nargs='?' or nargs='+' or nargs='*').
Any known argument is parsed, without 'breaking' at the new command. E.g. in PROG --foo command_b command_a --baz Z 12 with the above code, --baz Z will be consumed by command_b, not by command_a.
These limitations are a direct limitation of argparse. Here is a simple example that shows the limitations of argparse -even when using a single subcommand-:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('spam', nargs='?')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')
# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')
# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
options = parser.parse_args('command_a 42'.split())
print(options)
This will raise the error: argument subparser_name: invalid choice: '42' (choose from 'command_a', 'command_b').
The cause is that the internal method argparse.ArgParser._parse_known_args() it is too greedy and assumes that command_a is the value of the optional spam argument. In particular, when 'splitting' up optional and positional arguments, _parse_known_args() does not look at the names of the arugments (like command_a or command_b), but merely where they occur in the argument list. It also assumes that any subcommand will consume all remaining arguments.
This limitation of argparse also prevents a proper implementation of multi-command subparsers. This unfortunately means that a proper implementation requires a full rewrite of the argparse.ArgParser._parse_known_args() method, which is 200+ lines of code.
Given these limitation, it may be an options to simply revert to a single multiple-choice argument instead of subcommands:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--bar', type=int, help='bar help')
parser.add_argument('commands', nargs='*', metavar='COMMAND',
choices=['command_a', 'command_b'])
options = parser.parse_args('--bar 2 command_a command_b'.split())
print(options)
#options = parser.parse_args(['--help'])
It is even possible to list the different commands in the usage information, see my answer https://stackoverflow.com/a/49999185/428542

You can always split up the command-line yourself (split sys.argv on your command names), and then only pass the portion corresponding to the particular command to parse_args -- You can even use the same Namespace using the namespace keyword if you want.
Grouping the commandline is easy with itertools.groupby:
import sys
import itertools
import argparse
mycommands=['cmd1','cmd2','cmd3']
def groupargs(arg,currentarg=[None]):
if(arg in mycommands):currentarg[0]=arg
return currentarg[0]
commandlines=[list(args) for cmd,args in intertools.groupby(sys.argv,groupargs)]
#setup parser here...
parser=argparse.ArgumentParser()
#...
namespace=argparse.Namespace()
for cmdline in commandlines:
parser.parse_args(cmdline,namespace=namespace)
#Now do something with namespace...
untested

Improving on the answer by #mgilson, I wrote a small parsing method which splits argv into parts and puts values of arguments of commands into hierarchy of namespaces:
import sys
import argparse
def parse_args(parser, commands):
# Divide argv by commands
split_argv = [[]]
for c in sys.argv[1:]:
if c in commands.choices:
split_argv.append([c])
else:
split_argv[-1].append(c)
# Initialize namespace
args = argparse.Namespace()
for c in commands.choices:
setattr(args, c, None)
# Parse each command
parser.parse_args(split_argv[0], namespace=args) # Without command
for argv in split_argv[1:]: # Commands
n = argparse.Namespace()
setattr(args, argv[0], n)
parser.parse_args(argv, namespace=n)
return args
parser = argparse.ArgumentParser()
commands = parser.add_subparsers(title='sub-commands')
cmd1_parser = commands.add_parser('cmd1')
cmd1_parser.add_argument('--foo')
cmd2_parser = commands.add_parser('cmd2')
cmd2_parser.add_argument('--foo')
cmd2_parser = commands.add_parser('cmd3')
cmd2_parser.add_argument('--foo')
args = parse_args(parser, commands)
print(args)
It behaves properly, providing nice argparse help:
For ./test.py --help:
usage: test.py [-h] {cmd1,cmd2,cmd3} ...
optional arguments:
-h, --help show this help message and exit
sub-commands:
{cmd1,cmd2,cmd3}
For ./test.py cmd1 --help:
usage: test.py cmd1 [-h] [--foo FOO]
optional arguments:
-h, --help show this help message and exit
--foo FOO
And creates a hierarchy of namespaces containing the argument values:
./test.py cmd1 --foo 3 cmd3 --foo 4
Namespace(cmd1=Namespace(foo='3'), cmd2=None, cmd3=Namespace(foo='4'))

You could try arghandler. This is an extension to argparse with explicit support for subcommands.

Built a full Python 2/3 example with subparsers, parse_known_args and parse_args (running on IDEone):
from __future__ import print_function
from argparse import ArgumentParser
from random import randint
def main():
parser = get_parser()
input_sum_cmd = ['sum_cmd', '--sum']
input_min_cmd = ['min_cmd', '--min']
args, rest = parser.parse_known_args(
# `sum`
input_sum_cmd +
['-a', str(randint(21, 30)),
'-b', str(randint(51, 80))] +
# `min`
input_min_cmd +
['-y', str(float(randint(64, 79))),
'-z', str(float(randint(91, 120)) + .5)]
)
print('args:\t ', args,
'\nrest:\t ', rest, '\n', sep='')
sum_cmd_result = args.sm((args.a, args.b))
print(
'a:\t\t {:02d}\n'.format(args.a),
'b:\t\t {:02d}\n'.format(args.b),
'sum_cmd: {:02d}\n'.format(sum_cmd_result), sep='')
assert rest[0] == 'min_cmd'
args = parser.parse_args(rest)
min_cmd_result = args.mn((args.y, args.z))
print(
'y:\t\t {:05.2f}\n'.format(args.y),
'z:\t\t {:05.2f}\n'.format(args.z),
'min_cmd: {:05.2f}'.format(min_cmd_result), sep='')
def get_parser():
# create the top-level parser
parser = ArgumentParser(prog='PROG')
subparsers = parser.add_subparsers(help='sub-command help')
# create the parser for the "sum" command
parser_a = subparsers.add_parser('sum_cmd', help='sum some integers')
parser_a.add_argument('-a', type=int,
help='an integer for the accumulator')
parser_a.add_argument('-b', type=int,
help='an integer for the accumulator')
parser_a.add_argument('--sum', dest='sm', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')
# create the parser for the "min" command
parser_b = subparsers.add_parser('min_cmd', help='min some integers')
parser_b.add_argument('-y', type=float,
help='an float for the accumulator')
parser_b.add_argument('-z', type=float,
help='an float for the accumulator')
parser_b.add_argument('--min', dest='mn', action='store_const',
const=min, default=0,
help='smallest integer (default: 0)')
return parser
if __name__ == '__main__':
main()

I had more or less the same requirements: Being able to set global arguments and being able to chain commands and execute them in order of command line.
I ended up with the following code. I did use some parts of the code from this and other threads.
# argtest.py
import sys
import argparse
def init_args():
def parse_args_into_namespaces(parser, commands):
'''
Split all command arguments (without prefix, like --) in
own namespaces. Each command accepts extra options for
configuration.
Example: `add 2 mul 5 --repeat 3` could be used to a sequencial
addition of 2, then multiply with 5 repeated 3 times.
'''
class OrderNamespace(argparse.Namespace):
'''
Add `command_order` attribute - a list of command
in order on the command line. This allows sequencial
processing of arguments.
'''
globals = None
def __init__(self, **kwargs):
self.command_order = []
super(OrderNamespace, self).__init__(**kwargs)
def __setattr__(self, attr, value):
attr = attr.replace('-', '_')
if value and attr not in self.command_order:
self.command_order.append(attr)
super(OrderNamespace, self).__setattr__(attr, value)
# Divide argv by commands
split_argv = [[]]
for c in sys.argv[1:]:
if c in commands.choices:
split_argv.append([c])
else:
split_argv[-1].append(c)
# Globals arguments without commands
args = OrderNamespace()
cmd, args_raw = 'globals', split_argv.pop(0)
args_parsed = parser.parse_args(args_raw, namespace=OrderNamespace())
setattr(args, cmd, args_parsed)
# Split all commands to separate namespace
pos = 0
while len(split_argv):
pos += 1
cmd, *args_raw = split_argv.pop(0)
assert cmd[0].isalpha(), 'Command must start with a letter.'
args_parsed = commands.choices[cmd].parse_args(args_raw, namespace=OrderNamespace())
setattr(args, f'{cmd}~{pos}', args_parsed)
return args
#
# Supported commands and options
#
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--print', action='store_true')
commands = parser.add_subparsers(title='Operation chain')
cmd1_parser = commands.add_parser('add', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
cmd1_parser.add_argument('add', help='Add this number.', type=float)
cmd1_parser.add_argument('-r', '--repeat', help='Repeat this operation N times.',
default=1, type=int)
cmd2_parser = commands.add_parser('mult', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
cmd2_parser.add_argument('mult', help='Multiply with this number.', type=float)
cmd2_parser.add_argument('-r', '--repeat', help='Repeat this operation N times.',
default=1, type=int)
args = parse_args_into_namespaces(parser, commands)
return args
#
# DEMO
#
args = init_args()
# print('Parsed arguments:')
# for cmd in args.command_order:
# namespace = getattr(args, cmd)
# for option_name in namespace.command_order:
# option_value = getattr(namespace, option_name)
# print((cmd, option_name, option_value))
print('Execution:')
result = 0
for cmd in args.command_order:
namespace = getattr(args, cmd)
cmd_name, cmd_position = cmd.split('~') if cmd.find('~') > -1 else (cmd, 0)
if cmd_name == 'globals':
pass
elif cmd_name == 'add':
for r in range(namespace.repeat):
if args.globals.print:
print(f'+ {namespace.add}')
result = result + namespace.add
elif cmd_name == 'mult':
for r in range(namespace.repeat):
if args.globals.print:
print(f'* {namespace.mult}')
result = result * namespace.mult
else:
raise NotImplementedError(f'Namespace `{cmd}` is not implemented.')
print(10*'-')
print(result)
Below an example:
$ python argstest.py --print add 1 -r 2 mult 5 add 3 mult -r 5 5
Execution:
+ 1.0
+ 1.0
* 5.0
+ 3.0
* 5.0
* 5.0
* 5.0
* 5.0
* 5.0
----------
40625.0

Another package which supports parallel parsers is "declarative_parser".
import argparse
from declarative_parser import Parser, Argument
supported_formats = ['png', 'jpeg', 'gif']
class InputParser(Parser):
path = Argument(type=argparse.FileType('rb'), optional=False)
format = Argument(default='png', choices=supported_formats)
class OutputParser(Parser):
format = Argument(default='jpeg', choices=supported_formats)
class ImageConverter(Parser):
description = 'This app converts images'
verbose = Argument(action='store_true')
input = InputParser()
output = OutputParser()
parser = ImageConverter()
commands = '--verbose input image.jpeg --format jpeg output --format gif'.split()
namespace = parser.parse_args(commands)
and namespace becomes:
Namespace(
input=Namespace(format='jpeg', path=<_io.BufferedReader name='image.jpeg'>),
output=Namespace(format='gif'),
verbose=True
)
Disclaimer: I am the author. Requires Python 3.6. To install use:
pip3 install declarative_parser
Here is the documentation and here is the repo on GitHub.

In order to parse the sub commands, I used the following (referred from argparse.py code). It parses the sub parser arguments and retains the help for both. Nothing additional passed there.
args, _ = parser.parse_known_args()

you can use the package optparse
import optparse
parser = optparse.OptionParser()
parser.add_option("-f", dest="filename", help="corpus filename")
parser.add_option("--alpha", dest="alpha", type="float", help="parameter alpha", default=0.5)
(options, args) = parser.parse_args()
fname = options.filename
alpha = options.alpha