Develop Reference

Python argparse with subparsers and optional positional arguments

I would like to have a program with subparsers that handles specific arguments while also keep some positional and optional arguments to the previous parsers (In fact what I really want is only one option, I mean, a valid subparser OR a valid local argument).
Example of something I wish to have: Program [{sectionName [{a,b}]}] [{c,d}]. Being c/d incompatible if sectionName was provided and viceversa.
However, the best I could achieve is this test.py [-h] {sectionName} ... [{c,d}]. This means, argparse don't allow me to use the positional arguments c or d without specifying a valid sectionName.
Here is the code:
import argparse
mainparser = argparse.ArgumentParser()
# Subparser
subparser = mainparser.add_subparsers(title="section", required=False)
subparser_parser = subparser.add_parser("sectionName")
subparser_parser.add_argument("attribute", choices=['a', 'b'], nargs='?')
# Main parser positional and optional attributes
mainparser.add_argument("attribute", choices=['c', 'd'], nargs='?')
mainparser.parse_args()
I'm getting crazy with this. Any help would be much appreciated!
Edit: I'm using Python 3.8

The subparser object is actually a positional Action, one that takes choices - in this case {'sectionName'}. positinal arguments are filled in the order that they are defined, using the nargs pattern to allocate strings.
Once the main parser gets the 'sectionName' it passes the parsing to subparser_parser. Than handles the rest of the input, such as the {'a','b'} positional. Anything it can't handle is put on the 'unrecognized' list, and control returns main for final processing. main does not do any further argument processing. Thus your attribute argument is ignored.
You could put define a attribute positional before the add_subparsers, but I wouldn't try to make it nargs='?'.
So it's best to define all main arguments before the subparsers, and to use optionals. This will give the cleanest and most reliable parsing.

'argparse' with optional positional arguments that start with dash

We're trying to build a wrapper script over a command line tool we're using. We would like to set some tool arguments based on options in our wrapper scripts. We would also like to have the possibility to pass native arguments to the command line tool directly as they are written on the command line.
Here is what we came up with:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-f', '--foo', action='store_true')
parser.add_argument('-b', '--bar', action='store_true')
parser.add_argument('native_arg', nargs='*')
args = parser.parse_args()
print (args)
positional is mandatory. Based on the options -f and -b we would add some extra options to our tool call. Anything that is left afterwards (if anything) should be treated as a native tool argument and given to the tool directly. Calling our script with -h produces the following usage:
usage: test.py [-h] [-f] [-b] positional [native_arg [native_arg ...]]
The trick is that these native arguments are themselves options for the tool and contain leading dashes, for example -native0 and -native1. We already know about the trick with the double dash to stop argparse from looking for more options. The following call:
./test.py pos -- -native0 -native1
produces the expected parsed arguments:
Namespace(bar=False, foo=False, native_arg=['-native0', '-native1'], positional='pos')
Trying to add an option after the first positional argument doesn't work, though. More specifically, the following call:
./test.py pos --foo -- -native0 -native1
produces the following output:
usage: [...shortened...]
test.py: error: unrecognized arguments: -- -native0 -native1
Putting the optional arguments before the positionals:
./test.py --foo pos -- -native0 -native1
seems to work, as the following is printed:
Namespace(bar=False, foo=True, native_arg=['-native0', '-native1'], positional='pos')
Even stranger, changing the value of nargs for native_arg to '+' works in all the above situations (with the caveat, of course, that at least one native_arg is expected).
Are we doing something wrong in our Python code or is this some kind of argparse bug?

argparse does have a hard time when you mix non-required positional arguments with optional arguments (see https://stackoverflow.com/a/47208725/1399279 for details into the bug report). Rather than suggesting a way to solve this issue, I am going to present an alternative approach.
You should check out the parse_known_args method, which was created for the situation you describe (i.e. passing options to a wrapped tool).
In [1]: import argparse
In [2]: parser = argparse.ArgumentParser()
In [3]: parser.add_argument('positional')
In [4]: parser.add_argument('-f', '--foo', action='store_true')
In [5]: parser.add_argument('-b', '--bar', action='store_true')
In [6]: parser.parse_known_args(['pos', '--foo', '-native0', '-native1'])
Out[6]: (Namespace(bar=False, foo=True, positional='pos'), ['-native0', '-native1'])
Unlike parse_args, the output of parse_known_args is a two-element tuple. The first element is the Namespace instance you would expect to get from parse_args, and it contains all the attributes defined by calls to add_argument. The second element is a list of all the arguments not known to the parser.
I personally prefer this method because the user does not need to remember any tricks about how to call your program, or which option order does not result in errors.

This is a known issue (https://bugs.python.org/issue15112, argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional)
The parsing alternates handling positionals and optionals. When dealing with positionals it tries to handle as many as the input strings require. But an ? or * positional is satisfied with [], an empty list of strings. + on the other hand requires at least one string
./test.py pos --foo -- -native0 -native1
The parser gives 'pos' to positional, and [] to native-arg. Then it gives '--foo' to its optional. There aren't anymore positionals left to hand the remaining strings, so it raises the error.
The allocation of input strings is done with a stylized form of regex string matching. Imagine matching a pattern that looks like AA?.
To correct this, parser would have to look ahead, and delay handling native-arg. We've suggested patches but they aren't in production.
#SethMMorton's suggestion of using parse_known_args is a good one.
Earlier parsers (e.g. Optparse) handle all the flagged arguments, but return the rest, the positionals, as a undifferentiated list. It's up to the user to split that list. argparse has added the ability to name and parse positionals, but the algorithm works best with fixed nargs, and gets flaky with too many variable nargs.

argparse subparser implied by other parameters

The usually way to define a subparser is to do
master_parser = argparse.ArgumentParser()
subparsers = master_parser.add_subparsers()
parser = subparsers.add_parser('sub')
parser.add_argument('--subopt')
and the subparser would be called with
command sub --subopt
I am implementing a package that calls a number of converters. If I use the usual subparser approach, I would have to do
convert ext1_to_ext2 file.ext1 file.ext2 --args
which is both repetitive and error prone because users might call
convert ext1_to_ext3 file.ext1 file.ext2 --args
I would much prefer that the subparser is automatically determined from the master parser so users can use command
convert file.ext1 file.ext2 EXTRA
and argparse would determine subparser ext1_to_ext2 from file.ext1 and file.ext2 and call the subparser ext1_to_ext2 to parse EXTRA. Of course EXTRA here is subparser specific.
I tried to use groups of parameters for each converter (add_argument_group) but the parameters in argument groups cannot overlap and I got a messy list of combined arguments from all parsers, so using subparser seems to be the way to go.
I tried to use parse_known_args with two positional arguments, determine and use the appropriate subparser to parse the remaining args, but it is difficult to provide users a list of converters and their arguments from help message.
Is there a way to do this?

Inferring the subparser to use is tricky, since it requires reimplementing a lot of the logic used by argparse itself while you are examining each of the following arguments.
A simpler approach is to take the subparser command, which subsquently allows you to "typecheck" the following arguments to ensure they use the correct argument. For example
# This allows a file name ending with any of the given extensions,
# or allows "file.1" in place of "file.1.jpg"
def jpeg_file(s):
for ext in ("jpg", "jpeg"):
if s.endswith(ext) or os.path.exists("%s.%s" % (s, ext)):
return s
raise argparse.ArgumentTypeError()
def tiff_file(s):
# similar to jpeg_file
master_parser = argparse.ArgumentParser()
subparsers = master_parser.add_subparsers()
jpg_to_tiff_parser = subparsers.add_parser('sub')
jpg_to_tiff_parser = parse.add_argument('jpg', type=jpg_file)
jpg_to_tiff_parser = parse.add_argument('tiff', type=tiff_file)

This gives you a little more control in my opinion. It's along the way towards what your asking for. Just add file extension checking for your needs.
#prog.py
topParser=argparse.ArgumentParser()
subParsers = topParser.add_subparsers(
title='SubCommands',
description='Use "prog.py <subCommand> (-h | --help)" to learn more about each subcommand',
help='I can do stuff')
subParser1 = subParsers.add_parser('use1', help="Use1 does one thing")
subParser2 = subParsers.add_parser('use2', help='Use2 does another thing')
subParser1.add_argument(
'-f','--first-arg-for-use1',
help="A text file",
required=True
)
subParser1.add_argument(
'-s','--second-arg-for-use1',
help="An encoding",
required=True
)
subParser2.add_argument(
'-f','--first-arg-for-use2',
help="An image format",
required=True
)
args = topParser.parse_args()
print(args)
If nothing else, it shows how to handle the help text for the different layers.

Argparse: Optional arguments, distinct for different positional arguments

I want to have positional arguments with an optional argument. Smth like my_command foo --version=0.1 baz bar --version=0.2. That should parse into a list of args [foo, bar, baz] with version set for 2 of them.
Without optional argument it's trivial to set nargs=* or nargs=+, but I'm struggling with providing an optional argument for positional ones. Is it even possible with argparse?

Multiple invocation of the same subcommand in a single command line
This tries to parse something like
$ python test.py executeBuild --name foobar1 executeBuild --name foobar2 ....
Both proposed solutions call a parser multiple times. Each call handles a cmd --name value pair. One splits sys.argv before hand, the other collects unparsed strings with a argparse.REMAINDER argument.
Normally optionals can occur in any order. They are identified solely by that - flag value. Positionals have to occur in a particular order, but optionals may occur BETWEEN different positionals. Note also that in the usage display, optionals are listed first, followed by positionals.
usage: PROG [-h] [--version Version] [--other OTHER] FOO BAR BAZ
subparsers are the only way to link a positional argument with one or more optionals. But normally a parser is allowed to have only one subparsers argument.
Without subparsers, append is the only way to collect data from repeated uses of an optional:
parser.add_argument('--version',action='append')
parser.add_argument('foo')
parser.add_argument('bar')
parser.add_argument('baz')
would handle your input string, producing a namespace like:
namespace(version=['0.1','0.2'],foo='foo',bar='bar',baz='baz')
But there's no way of identifying baz as the one that is 'missing' a version value.
Regarding groups - argument groups just affect the help display. They have nothing to do with parsing.
How would you explain to your users (or yourself 6 mths from now) how to use this interface? What would the usage and help look like? It might be simpler to change the design to something that is easier to implement and to explain.
Here's a script which handles your sample input.
import argparse
usage = 'PROG [cmd [--version VERSION]]*'
parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('cmd')
parser.add_argument('-v','--version')
parser.add_argument('rest', nargs=argparse.PARSER)
parser.print_usage()
myargv = 'foo --version=0.1 baz bar --version=0.2'.split()
# myargv = sys.argv[1:] # in production
myargv += ['quit'] # end loop flag
args = argparse.Namespace(rest=myargv)
collect = argparse.Namespace(cmd=[])
while True:
args = parser.parse_args(args.rest)
collect.cmd += [(args.cmd, args.version)]
print(args)
if args.rest[0]=='quit':
break
print collect
It repeatedly parses a positional and optional, collecting the rest in a argparse.PARSER argument. This is like + in that it requires at least one string, but it collects ones that look like optionals as well. I needed to add a quit string so it wouldn't throw an error when there wasn't anything to fill this PARSER argument.
producing:
usage: PROG [cmd [--version VERSION]]*
Namespace(cmd='foo', rest=['baz', 'bar', '--version=0.2', 'quit'], version='0.1')
Namespace(cmd='baz', rest=['bar', '--version=0.2', 'quit'], version=None)
Namespace(cmd='bar', rest=['quit'], version='0.2')
Namespace(cmd=[('foo', '0.1'), ('baz', None), ('bar', '0.2')])
The positional argument that handles subparsers also uses this nargs value. That's how it recognizes and collects a cmd string plus everything else.
So it is possible to parse an argument string such as you want. But I'm not sure the code complexity is worth it. The code is probably fragile as well, tailored to this particular set of arguments, and just a few variants.

Add argparse arguments from external modules

I'm trying to write a Python program that could be extended by third parties. The program will be run from the command line with whatever arguments are supplied.
In order to allow third parties to create their own modules, I've created the following (simplified) base class:
class MyBaseClass(object):
def __init__(self):
self.description = ''
self.command = ''
def get_args(self):
# code that I can't figure out to specify argparse arguments here
# args = []
# arg.append(.....)
return args
Any arguments that they supply via get_args() will be added to a subparser for that particular module. I want them to be able to specify any type of argument.
I'm not sure of the best way to declare and then get the arguments from the subclassed modules into my main program. I successfully find all subclasses of MyBaseClass and loop through them to create the subparsers, but I cannot find a clean way to add the individual arguments to the subparser.
Here is the current code from the main program:
for module in find_modules():
m = module()
subparser_dict[module.__name__] = subparsers.add_parser(m.command, help=m.help)
for arg in m.get_args():
subparser_dict[module.__name__].add_argument(...)
How can I best specify the arguments in the external modules via get_args() or similar and then add them to the subparser? One of my failed attempts looked like the following, which doesn't work because it tries to pass every possible option to add_argument() whether it has a value or is None:
subparser_dict[module.__name__].add_argument(arg['long-arg'],
action=arg['action'],
nargs=arg['nargs'],
const=arg['const'],
default=arg['default'],
type=arg['type'],
choices=arg['choices'],
required=arg['required'],
help=arg['help'],
metavar=arg['metavar'],
dest=arg['dest'],
)

Without trying to fully understand your module structure, I think you want to be able to provide the arguments to a add_argument call as objects that you can import.
You could, for example, provide a list of positional arguments, and dictionary of keyword arguments:
args=['-f','--foo']
kwargs={'type':int, 'nargs':'*', 'help':'this is a help line'}
parser=argparse.ArgumentParser()
parser.add_argument(*args, **kwargs)
parser.print_help()
producing
usage: ipython [-h] [-f [FOO [FOO ...]]]
optional arguments:
-h, --help show this help message and exit
-f [FOO [FOO ...]], --foo [FOO [FOO ...]]
this is a help line
In argparse.py, the add_argument method (of a super class of ArgumentParser), has this general signature
def add_argument(self, *args, **kwargs):
The code of this method manipulates these arguments, adds the args to the kwargs, adds default values, and eventually passes kwargs to the appropriate Action class, returning the new action. (It also 'registers' the action with the parser or subparser). It's the __init__ of the Action subclasses that lists the arguments and their default values.

I would just return an ArgumentParser instance from your get_args method. Then you can create a new ArgumentParser to join all other argument parsers using the parents argument: https://docs.python.org/3/library/argparse.html#parents.