Develop Reference

How to make optional subparser in python3?

I want to input args that not configured in argparse:
parser = argparse.ArgumentParser(prog='PROG')
subparsers = parser.add_subparsers(help='sub-command help', dest="character", required=False)
subparsers.required = False
base_subparser = argparse.ArgumentParser(add_help=False)
# define common shared arguments
base_subparser.add_argument('--disable', choices=['false', 'true'])
base_subparser.add_argument('--foo', choices=['false', 'true'])
# create the parser for the "a" command
parser_a = subparsers.add_parser('a', help='a help', parents=[base_subparser])
parser_a.add_argument('--bar', choices='ABC', help='bar help')
# create the parser for the "b" command
parser_b = subparsers.add_parser('b', help='b help', parents=[base_subparser])
parser_b.add_argument('--baz', choices='XYZ', help='baz help')
argcomplete.autocomplete(parser)
args = parser.parse_known_args()
print(args)
I use the parse_known_args() which use a list to store the args not configured in argparse. However, when I use ./prog.py key = val, it shows argument character: invalid choice: 'key=val' (choose from 'a', 'b'). So I have to choose 'a' or 'b', how can I input the args not configured in argparse without choose one of the subparsers.

The error you see is the same as produced by a '?' positional with choices:
In [25]: import argparse
In [26]: parser = argparse.ArgumentParser()
In [27]: parser.add_argument('foo', nargs='?', choices=['a','b'])
'foo' is optional:
In [28]: parser.parse_known_args([])
Out[28]: (Namespace(foo=None), [])
In [29]: parser.parse_known_args(['a'])
Out[29]: (Namespace(foo='a'), [])
but any string is parsed as a possible 'foo' value:
In [30]: parser.parse_known_args(['c'])
usage: ipykernel_launcher.py [-h] [{a,b}]
ipykernel_launcher.py: error: argument foo: invalid choice: 'c' (choose from 'a', 'b')
providing a proper choice first, allows it to treat 'c' as an extra:
In [31]: parser.parse_known_args(['a','c'])
Out[31]: (Namespace(foo='a'), ['c'])
Or if the string looks like a optional's flag:
In [32]: parser.parse_known_args(['-c'])
Out[32]: (Namespace(foo=None), ['-c'])
Another possibility is to go ahead and name a subparser, possibly a dummy one, and provide the extra. The subparser will be the one that actually puts that string in the 'unknowns' category.
In [40]: parser = argparse.ArgumentParser()
In [41]: subp = parser.add_subparsers(dest='cmd')
In [44]: p1 = subp.add_parser('a')
In [45]: parser.parse_known_args(['a','c'])
Out[45]: (Namespace(cmd='a'), ['c'])
In [46]: parser.parse_known_args([]) # not-required is the default
Out[46]: (Namespace(cmd=None), [])
Keep in mind that the main parser does not "know" anything about subparsers, except is a positional. It's doing its normal allocating strings to actions. But once it calls a subparser, that parser has full control over the parsing. Once it's done it passes the namespace back to the main, but the main doesn't do any more parsing - it just wraps things up and exits (with results or error).
Since subp is a positional with a special Action subclass, _SubParsersAction, I was thinking it might be possible to create a flagged argument with that class
parser.add_argument('--foo', action=argparse._SubParsersAction)
but there's more going on in add_subparsers, so it isn't a trivial addition. This is a purely speculative idea.

Correct way to get allowed arguments from ArgumentParser

Question: What is the intended / official way of accessing possible arguments from an existing argparse.ArgumentParser object?
Example: Let's assume the following context:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', '-f', type=str)
Here I'd like to get the following list of allowed arguments:
['-h', '--foo', '--help', '-f']
I found the following workaround which does the trick for me
parser._option_string_actions.keys()
But I'm not happy with it, as it involves accessing a _-member that is not officially documented. Whats the correct alternative for this task?

I don't think there is a "better" way to achieve what you want.
If you really don't want to use the _option_string_actions attribute, you could process the parser.format_usage() to retrieve the options, but doing this, you will get only the short options names.
If you want both short and long options names, you could process the parser.format_help() instead.
This process can be done with a very simple regular expression: -+\w+
import re
OPTION_RE = re.compile(r"-+\w+")
PARSER_HELP = """usage: test_args_2.py [-h] [--foo FOO] [--bar BAR]
optional arguments:
-h, --help show this help message and exit
--foo FOO, -f FOO a random options
--bar BAR, -b BAR a more random option
"""
options = set(OPTION_RE.findall(PARSER_HELP))
print(options)
# set(['-f', '-b', '--bar', '-h', '--help', '--foo'])
Or you could first make a dictionnary which contains the argument parser configuration and then build the argmuent parser from it. Such a dictionnary could have the option names as key and the option configuration as value. Doing this, you can access the options list via the dictionnary keys flattened with itertools.chain:
import argparse
import itertools
parser_config = {
('--foo', '-f'): {"help": "a random options", "type": str},
('--bar', '-b'): {"help": "a more random option", "type": int, "default": 0}
}
parser = argparse.ArgumentParser()
for option, config in parser_config.items():
parser.add_argument(*option, **config)
print(parser.format_help())
# usage: test_args_2.py [-h] [--foo FOO] [--bar BAR]
#
# optional arguments:
# -h, --help show this help message and exit
# --foo FOO, -f FOO a random options
# --bar BAR, -b BAR a more random option
print(list(itertools.chain(*parser_config.keys())))
# ['--foo', '-f', '--bar', '-b']
This last way is what I would do, if I was reluctant to use _option_string_actions.

This started as a joke answer, but I've learned something since - so I'll post it.
Assume, we know the maximum length of an option allowed. Here is a nice answer to the question in this situation:
from itertools import combinations
def parsable(option):
try:
return len(parser.parse_known_args(option.split())[1]) != 2
except:
return False
def test(tester, option):
return any([tester(str(option) + ' ' + str(v)) for v in ['0', '0.0']])
def allowed_options(parser, max_len=3, min_len=1):
acceptable = []
for l in range(min_len, max_len + 1):
for option in combinations([c for c in [chr(i) for i in range(33, 127)] if c != '-'], l):
option = ''.join(option)
acceptable += [p + option for p in ['-', '--'] if test(parsable, p + option)]
return acceptable
Of course this is very pedantic as the question doesn't require any specific runtime. So I'll ignore that here. I'll also disregard, that the above version produces a mess of output because one can get rid of it easily.
But more importantly, this method detected the following interesting argparse "features":
In in the OP example, argparse would also allow --fo. This has to be a bug.
But further, in the OP example again, argparse would also allow -fo (ie. setting foo to o without space or anything). This is documented and intended, but I didn't know it.
Because of this, a correct solution is a bit longer and would look something like this (only parsable changes, I'll omit the other methods):
def parsable(option):
try:
default = vars(parser.parse_known_args(['--' + '0' * 200])[0])
parsed, remaining = parser.parse_known_args(option.split())
if len(remaining) == 2:
return False
parsed = vars(parsed)
for k in parsed.keys():
try:
if k in default and default[k] != parsed[k] and float(parsed[k]) != 0.0:
return False # Filter '-fx' cases where '-f' is the argument and 'x' the value.
except:
return False
return True
except:
return False
Summary: Besides all the restrictions (runtime and fixed maximum option length), this is the only answer that correctly respects the real parser behavior - however buggy it may even be. So here you are, a perfect answer that is absolutely useless.

I have to agree with Tryph's answer.
Not pretty, but you can retrieve them from parser.format_help():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', '-f', type=str)
goal = parser._option_string_actions.keys()
def get_allowed_arguments(parser):
lines = parser.format_help().split('\n')
line_index = 0
number_of_lines = len(lines)
found_optional_arguments = False
# skip the first lines until the section 'optional arguments'
while line_index < number_of_lines:
if lines[line_index] == 'optional arguments:':
found_optional_arguments = True
line_index += 1
break
line_index += 1
result_list = []
if found_optional_arguments:
while line_index < number_of_lines:
arg_list = get_arguments_from_line(lines[line_index])
if len(arg_list) == 0:
break
result_list += arg_list
line_index += 1
return result_list
def get_arguments_from_line(line):
if line[:2] != ' ':
return []
arg_list = []
i = 2
N = len(line)
inside_arg = False
arg_start = 2
while i < N:
if line[i] == '-' and not inside_arg:
arg_start = i
inside_arg = True
elif line[i] in [',',' '] and inside_arg:
arg_list.append(line[arg_start:i+1])
inside_arg = False
i += 1
return arg_list
answer = get_allowed_arguments(parser)
There's probably a regular expressions alternative to the above mess...

First a note on the argparse docs - it's basically a how-to-use document, not a formal API. The standard for what argparse does is the code itself, the unit tests (test/test_argparse.py), and a paralyzing concern for backward compatibility.
There's no 'official' way of accessing allowed arguments, because users usually don't need to know that (other than reading the help/usage).
But let me illustrate with a simple parser in an iteractive session:
In [247]: parser=argparse.ArgumentParser()
In [248]: a = parser.add_argument('pos')
In [249]: b = parser.add_argument('-f','--foo')
add_argument returns the Action object that it created. This isn't documented, but obvious to any one who has created a parser interactively.
The parser object has a repr method, that displays major parameters. But it has many more attributes, which you can see with vars(parser), or parser.<tab> in Ipython.
In [250]: parser
Out[250]: ArgumentParser(prog='ipython3', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)
The Actions too have repr; the Action subclass is determined by the action parameter.
In [251]: a
Out[251]: _StoreAction(option_strings=[], dest='pos', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
In [252]: b
Out[252]: _StoreAction(option_strings=['-f', '--foo'], dest='foo', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
vars(a) etc can be used to see all attributes.
A key parser attribute is _actions, a list of all defined Actions. This is the basis for all parsing. Note it includes the help action that was created automatically. Look at option_strings; that determines whether the Action is positional or optional.
In [253]: parser._actions
Out[253]:
[_HelpAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, help='show this help message and exit', metavar=None),
_StoreAction(option_strings=[], dest='pos',....),
_StoreAction(option_strings=['-f', '--foo'], dest='foo', ...)]
_option_string_actions is a dictionary, mapping from option_strings to Actions (the same objects that appear in _actions). References to those Action objects appear all over the place in argparse code.
In [255]: parser._option_string_actions
Out[255]:
{'--foo': _StoreAction(option_strings=['-f', '--foo'],....),
'--help': _HelpAction(option_strings=['-h', '--help'],...),
'-f': _StoreAction(option_strings=['-f', '--foo'], dest='foo',...),
'-h': _HelpAction(option_strings=['-h', '--help'], ....)}
In [256]: list(parser._option_string_actions.keys())
Out[256]: ['-f', '--help', '-h', '--foo']
Note that there is a key for each - string, long or short; but there's nothing for pos, the positional has an empty option_strings parameter.
If that list of keys is what you want, use it, and don't worry about the _. It does not have a 'public' alias.
I can understand parsing the help to discover the same; but that's a lot of work to just avoid using a 'private' attribute. If you worry about the undocumented attribute being changed, you should also worry about the help format being changed. That isn't part of the docs either.
help layout is controlled by parser.format_help. The usage is created from information in self._actions. Help lines from information in
for action_group in self._action_groups:
formatter.add_arguments(action_group._group_actions)
(you don't want to get into action groups do you?).
There is another way of getting the option_strings - collect them from the _actions:
In [258]: [a.option_strings for a in parser._actions]
Out[258]: [['-h', '--help'], [], ['-f', '--foo']]
===================
Delving in to code details a bit:
parser.add_argument creates an Action, and then passes it to parser._add_action. This is the method the populates both .actions and action.option_strings.
self._actions.append(action)
for option_string in action.option_strings:
self._option_string_actions[option_string] = action

ArgumentParser: Optional argument with optional value

If I have an optional argument with optional argument value, is there a way to validate if the argument is set when the value is not given?
For instance:
parser = argparse.ArgumentParser()
parser.add_argument('--abc', nargs='?')
args = parser.parse_args()
Would correctly give me:
optional arguments:
--abc [ABC]
How do I distinguish between 1 and 2 below?
'' => args.abc is None
'--abc' => args.abc is still None
'--abc something' => args.abc is something
...
Update:
Found a trick to solve this problem: you can use "nargs='*'" instead of "nargs='?'". This way #1 would return None, and #2 would return an empty list. The downside is this will allow multiple values for the arguments to be accepted too; so you'd need to add a check for it if appropriate.
Alternatively you can also set a default value for the argument; see answer from chepner and Anand S Kumar.

With nargs='?', you can supply both a default and const.
In [791]: parser=argparse.ArgumentParser()
In [792]: parser.add_argument('--abc', nargs='?', default='default', const='const')
If the argument is not given it uses the default:
In [793]: parser.parse_args([])
Out[793]: Namespace(abc='default')
If given, but without an argument string, it uses the const:
In [794]: parser.parse_args(['--abc'])
Out[794]: Namespace(abc='const')
Otherwise it uses the argument string:
In [795]: parser.parse_args(['--abc','test'])
Out[795]: Namespace(abc='test')
In [796]: parser.print_help()
usage: ipython3 [-h] [--abc [ABC]]
optional arguments:
-h, --help show this help message and exit
--abc [ABC]

Use a different default value for the option. Compare
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--abc', nargs='?', default="default")
>>> parser.parse_args()
Namespace(abc='default')
>>> parser.parse_args(['--abc'])
Namespace(abc=None)
>>> parser.parse_args(['--abc', 'value'])
Namespace(abc='value')
I'm not sure how you would provide a different value for when --abc is used without an argument, short of using a custom action instead of the nargs argument.

Not sure if this is the standard way, but you can set default argument to something , and then that value would be used in case --abc is not in the argument list.
Example code -
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--abc', nargs='?', default="-1")
args = parser.parse_args()
print(args)
Result -
>python a.py
Namespace(abc='-1')
>python a.py --abc
Namespace(abc=None)
>python a.py --abc something
Namespace(abc='something')

I'm using this to have a command-line arg for multiprocessing. Specifying --multi uses all cores and given an arg specifies a number of cores, e.g., --multi 4 for four cores.
parser.add_argument("-mp", "--multi", type=int, nargs="*", help=multi_text)
Parsing logic is then:
if (args.multi == None):
num_cores = 1
elif (args.multi == []):
num_cores = multiprocessing.cpu_count()
elif (len(args.multi) == 1):
num_cores = args.multi[0]
else:
print("Invalid specification of core usage.")
sys.exit(1)

generic command line generator in python

Is there a generic command line generator in python? I mean something that is like argparse but has the opposite functionality. argparse lets you define various arguments and then parses a given command line string into values of those arguments. I need something that lets you define various arguments like argparse, but given a dict of argument, value pairs will generate a command line string.
Example :
gencmdline = CmdlineGenerator()
gencmdline.add_argument('-f', '--foo')
gencmdline.add_argument('bar')
gencmdline.add_argument('phi')
gencmdline.gen_cmdline(phi='hello', bar=1, foo=2) returns:
"1 hello -f 2"
gencmdline.gen_cmdline(phi='hello', bar=1) returns:
"1 hello"
gencmdline.gen_cmdline(phi='hello', foo=2) raise exception because positional argument bar is not specified.

I'm assuming you want to use something like call and pass a command some set of keywords.
from subprocess import call
def get_call_array(command=command,**kwargs):
callarray = [command]
for k, v in self.kwargs.items():
callarray.append("--" + k)
if v:
callarray.append(str(v))
return callarray
call(get_call_array("my_prog",height=5,width=10,trials=100,verbose=None))
#calls: my_prog --height 5 --width 10 --trials 100 --verbose
Of course, if you have a dictionary, of all your parameters, it's even easier:
def get_call_array_from_dict(command,options_dict):
callarray=[command]
for k,v in options_dict.items():
callarray.append("--" + str(k))
if v:
callarray.append(str(v))
return callarray

There probably is enough information in the Actions generated by a regular parser definition to do this job.
To take part of your example:
In [122]: parser=argparse.ArgumentParser()
In [123]: arglist = []
In [124]: arg1 = parser.add_argument('-f','--foo')
In [126]: arglist.append(arg1)
In [128]: arg2=parser.add_argument('bar')
In [129]: arglist.append(arg2)
In [131]: arg3=parser.add_argument('phi')
In [132]: arglist.append(arg3)
In [133]: arglist
Out[133]:
[_StoreAction(option_strings=['-f', '--foo'], dest='foo', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None),
_StoreAction(option_strings=[], dest='bar', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None),
_StoreAction(option_strings=[], dest='phi', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)]
add_argument creates an Action object (subclass actually based on action type), with attributes that are either defaults or ones you specify with parameters.
In this example I am collecting them both as variables, arg1 and elements of a list. There are other attributes that aren't shown in this repr. You can explore those in an interactive interpreter.
From your dictionary, dict(phi='hello', bar=1, foo=2) and this list you can deduce that the argument with dest='foo' has an option string (-f or --foo), and everything else is default (nargs=None is the default 1 argument). dest='bar' is postional with an empty option_strings. You'll have to be careful and note that bar occurs before phi.
parser._actions
is the same list of Actions, with the addition of -h.

Reading default arguments with argparse

When using something like this with argparse:
parser.add_argument("-option1", type=int, nargs=1, default=1, choices=xrange(1, 10), help="option 1 message")
If the argument is not passed, args.option1 is an integer. If an argument is passed, args.option is a list.
Which means I have to use either args.option1 or args.option1[0]. Is it possible to avoid this?

If you set nargs to an integer, the argument will always be a list when specified. You could use '?' instead to have it set to a single value or the default:
parser.add_argument(
"-option1", type=int, nargs='?', default=1,
choices=xrange(1, 10), help="option 1 message")
From the nargs documentation:
N (an integer). N arguments from the command line will be gathered together into a list.
[...]
'?'. One argument will be consumed from the command line if possible, and produced as a single item. If no command-line argument is present, the value from default will be produced.
Emphasis mine. You probably want to set const as well though, in case someone uses just -option1 on the command line with no value following it; without const the value is then set to None. Just set it to the same value as the default:
parser.add_argument(
"-option1", type=int, nargs='?', default=1, const=1,
choices=xrange(1, 10), help="option 1 message")
Another option is to not set nargs:
parser.add_argument(
"-option1", type=int, default=1,
choices=xrange(1, 10), help="option 1 message")
This usually means almost the same as nargs='?':
If the nargs keyword argument is not provided, the number of arguments consumed is determined by the action. Generally this means a single command-line argument will be consumed and a single item (not a list) will be produced.
with the difference being that an empty -option1 argument on the command line is not permitted; no const needs to be set.
Demo:
>>> from argparse import ArgumentParser
>>> parser = ArgumentParser()
>>> parser.add_argument(
... "-option1", type=int, nargs='?', default=1, const=1,
... choices=xrange(1, 10), help="option 1 message")
_StoreAction(option_strings=['-option1'], dest='option1', nargs='?', const=1, default=1, type=<type 'int'>, choices=xrange(1, 10), help='option 1 message', metavar=None)
>>> parser.parse_args(['-option1', '4']) # option with argument
Namespace(option1=4)
>>> parser.parse_args(['-option1']) # option with *no* argument
Namespace(option1=1)
>>> parser.parse_args([]) # option not specified
Namespace(option1=1)
>>> parser = ArgumentParser()
>>> parser.add_argument(
... "-option1", type=int, default=1,
... choices=xrange(1, 10), help="option 1 message")
_StoreAction(option_strings=['-option1'], dest='option1', nargs=None, const=None, default=1, type=<type 'int'>, choices=xrange(1, 10), help='option 1 message', metavar=None)
>>> parser.parse_args(['-option1', '4']) # option with argument
Namespace(option1=4)
>>> parser.parse_args([]) # option not specified
Namespace(option1=1)
>>> parser.parse_args(['-option1']) # option with *no* argument
usage: [-h] [-option1 {1,2,3,4,5,6,7,8,9}]
: error: argument -option1: expected one argument

You get a list because you set nargs=1 in the arguments to add_argument. This is also outlined in the docs:
Note that nargs=1 produces a list of one item. This is different from the default, in which the item is produced by itself.
If you remove nargs=1, it should produce the item on its own as desired (from the argument if given, otherwise from default). Should you want to make the number optional (i.e. also allow -option1 without argument), use nargs='?' and set const as the default value for that case.