Develop Reference

Python 3.7 ArgumentParser.add_subparsers require positional before optionals

I'm trying to create a python script that will execute another script, depending on the first positional parameter. Think along the lines of how git add behaves.
Problem is that ArgumentParser appears to want the positional sub-command to be listed... at the end. Which is pretty counter-intuitive. (When you want to list all files, you do ls -a [FILE positional], not -a ls [FILE positional], so why would it require scriptname [optionals] subcommand instead of scriptname subcommand [optionals] since 'subcommand' is the 'real' command?)
Toy example:
def get_arg_parser():
parser = argparse.ArgumentParser()
# set up subprocessors
subparser = parser.add_subparsers(required=True)
parser.add_argument('--verbose', action='store_const', const=True, default=False, help="Enable verbose output.")
subcommand1_subparser = subparser.add_parser('subcommand1')
subcommand1_subparser.add_argument('--foo1', type=float)
subcommand2_subparser = subparser.add_parser('subcommand2')
subcommand2_subparser.add_argument('--foo2', type=float)
return parser
if __name__ == "__main__":
if len(sys.argv) > 1:
get_arg_parser().parse_args()
# more
else:
get_arg_parser().print_help()
Problem is that if I try to run python toyexample.py subcommand1 --verbose, it complains about error: unrecognized arguments: --verbose. Meanwhile, python toyexample.py --verbose subcommand1 works, but it's requiring the optionals before the name of the command you're actually intending to run.
How do I override this?

Thanks to #hpaulj, I found a solution: simply add the shared arguments to both subparsers.
I put the parser.add_argument('--verbose', action='store_const', const=True, default=False, help="Enable verbose output.") line in a add_shared_args_to_parser to function, which I then call twice, passing the subparsers.
Net result is that the subparsers have some unfortunate (but not terrible) duplication and the main parser has nothing but subparsers.

Configure argparse to accept quoted arguments

I am writing a program which, among other things, allows the user to specify through an argument a module to load (and then use to perform actions). I am trying to set up a way to easily pass arguments through to this inner module, and I was attempting to use ArgParse's action='append' to have it build a list of arguments that I would then pass through.
Here is a basic layout of the arguments that I am using
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
required=True)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
However - if I then try to run this program with python my_program -M module:class -A "-f filename" (where I would like to pass through the -f filename to my module) it seems to be parsing the -f as its own argument (and I get the error my_program: error: argument -A/--module_args: expected one argument
Any ideas?

With:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
import sys
print(sys.argv)
print(parser.parse_args())
I get:
1028:~/mypy$ python stack45146728.py -M module:class -A "-f filename"
['stack45146728.py', '-M', 'module:class', '-A', '-f filename']
Namespace(module='module:class', module_args=['-f filename'])
This is using a linux shell. The quoted string remains one string, as seen in the sys.argv, and is properly interpreted as an argument to -A.
Without the quotes the -f is separate and interpreted as a flag.
1028:~/mypy$ python stack45146728.py -M module:class -A -f filename
['stack45146728.py', '-M', 'module:class', '-A', '-f', 'filename']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
Are you using windows or some other OS/shell that doesn't handle quotes the same way?
In Argparse `append` not working as expected
you asked about a slightly different command line:
1032:~/mypy$ python stack45146728.py -A "-k filepath" -A "-t"
['stack45146728.py', '-A', '-k filepath', '-A', '-t']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
As I already noted -k filepath is passed through as one string. Because of the space, argparse does not interpret that as a flag. But it does interpret the bare '-t' as a flag.
There was a bug/issue about the possibility of interpreting undefined '-xxx' strings as arguments instead of flags. I'd have to look that up to see whether anything made it into to production.
Details of how strings are categorized as flag or argument can be found in argparse.ArgumentParser._parse_optional method. It contains a comment:
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
http://bugs.python.org/issue9334 argparse does not accept options taking arguments beginning with dash (regression from optparse) is an old and long bug/issue on the topic.

The solution is to accept arbitrary arguments - there's an example in argparse's doc here:
argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

Intrepret parameter starting with a hyphen as a string [duplicate]

I would like to parse a required, positional argument containing a comma-separated list of integers. If the first integer contains a leading minus ('-') sign, argparse complains:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-t', '--test', action='store_true')
opts = parser.parse_args()
print opts
$ python example.py --test 1,2,3,4
Namespace(positional='1,2,3,4', test=True)
$ python example.py --test -1,2,3,4
usage: example.py [-h] [-t] positional
example.py: error: too few arguments
$ python example.py --test "-1,2,3,4"
usage: example.py [-h] [-t] positional
example.py: error: too few arguments
I've seen people suggest using some other character besides - as the flag character, but I'd rather not do that. Is there another way to configure argparse to allow both --test and -1,2,3,4 as valid arguments?

You need to insert a -- into your command-line arguments:
$ python example.py --test -- -1,2,3,4
Namespace(positional='-1,2,3,4', test=True)
The double-dash stops argparse looking for any more optional switches; it's the defacto standard way of handling exactly this use case for command-line tools.

From the documentation:
The parse_args() method attempts to give errors whenever the user has
clearly made a mistake, but some situations are inherently ambiguous.
For example, the command-line argument -1 could either be an attempt
to specify an option or an attempt to provide a positional argument.
The parse_args() method is cautious here: positional arguments may
only begin with - if they look like negative numbers and there are no
options in the parser that look like negative numbers:
Since -1,2,3,4 does not look like a negative number you must "escape" it with the -- as in most *nix systems.
An other solution would be to use nargs for the positional and pass the numbers as space separated:
#test.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional', nargs='*') #'+' for one or more numbers
print parser.parse_args()
Output:
$ python test.py -1 2 3 -4 5 6
Namespace(positional=['-1', '2', '3', '-4', '5', '6'])
A third way to obtain what you want is to use parse_known_args instead of parse_args.
You do not add the positional argument to the parser and parse it manually instead:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--test', action='store_true')
parsed, args = parser.parse_known_args()
print parsed
print args
Result:
$ python test.py --test -1,2,3,4
Namespace(test=True)
['-1,2,3,4']
This has the disadvantage that the help text will be less informative.

How can I get argparse to accept "--" as an argument to an option?

I have a command-line option that requires an argument. I would like to be able to supply "--" as the argument, but I can't figure out how to do it.
Sample code: (test-argparse.py)
#!/usr/bin/env python
from __future__ import print_function
import argparse
import sys
def main(argv):
ap = argparse.ArgumentParser()
ap.add_argument("-x", "--foo", metavar="VALUE", default="",
help="Test option.")
args = ap.parse_args(argv[1:])
print(args.foo)
if __name__ == "__main__":
sys.exit(main(sys.argv))
All my attempts to try to pass "--" as an argument fail:
$ test-argparse.py --foo --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: argument -x/--foo: expected one argument
$ test-argparse.py --foo -- --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: argument -x/--foo: expected one argument
$ test-argparse.py --foo=--
[]
$ test-argparse.py --foo=-- --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: unrecognized arguments: --
$ test-argparse.py --foo="--"
[]
$ test-argparse.py '--foo --'
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: unrecognized arguments: --foo --
$ test-argparse.py -x--
[]
$ test-argparse.py '-x --'
--
The last case is the closest, but it includes the space (and I can't just strip whitespace, because what if I want to allow " " as a value?). Is there any way that I can accomplish this?
That argparse forces argument permutation on clients (leading to unnecessary ambiguity) is very frustrating.
(I am using Python 2.7.12.)

Ideally --foo=-- should work, but the current parser deletes all '--', leaving an empty string in its place, hence the foo=[] result. I proposed a patch a couple of years ago that should have fixed that, but it's caught in the argparse backlog. http://bugs.python.org/issue13922, http://bugs.python.org/issue14364, http://bugs.python.org/issue9571
Python argparse with -- as the value suggests preprocessing sys.argv replacing one or more of the -- with something else.
If you are game for patching your argparse.py file (or subclass the ArgumentParser class), I could revisit my earlier work and suggest a fix. The trick is to accept that =-- but still use the first free -- as the 'rest-are-positionals' flag (and retain any following --). Unfortunately one method that needs to be patched is nested in a much larger one.

There is a specific reason that this doesn't work: -- means "Skip this token and consider the rest of the arguments to be positional, even if they start with a dash."
Many, many programs won't accept -- as an argument, but they will accept -. The single dash is even a standard way of specifying "Use standard input or output" in place of a filename.
So the best thing you can do for the users of your program is probably to not design it to require --, because that's not something that's usually done, and not something that most modern command-line parsing libraries are likely able to parse.
You could use -- as a positional option, so you could probably support this:
--foo -- --
If you make --foo have action='store_true' (i.e. it is an option taking no argument), plus one non-mandatory positional argument. That will probably work, because the first -- means "stop processing dashes as options" and the second is a positional argument.

Can argparse in python 2.7 be told to require a minimum of TWO arguments?

My application is a specialized file comparison utility and obviously it does not make sense to compare only one file, so nargs='+' is not quite appropriate.
nargs=N only excepts a maximum of N arguments, but I need to accept an infinite number of arguments as long as there are at least two of them.

Short answer is you can't do that because nargs doesn't support something like '2+'.
Long answer is you can workaround that using something like this:
parser = argparse.ArgumentParser(usage='%(prog)s [-h] file file [file ...]')
parser.add_argument('file1', nargs=1, metavar='file')
parser.add_argument('file2', nargs='+', metavar='file', help=argparse.SUPPRESS)
namespace = parser.parse_args()
namespace.file = namespace.file1 + namespace.file2
The tricks that you need are:
Use usage to provide you own usage string to the parser
Use metavar to display an argument with a different name in the help string
Use SUPPRESS to avoid displaying help for one of the variables
Merge two different variables just adding a new attribute to the Namespace object that the parser returns
The example above produces the following help string:
usage: test.py [-h] file file [file ...]
positional arguments:
file
optional arguments:
-h, --help show this help message and exit
and will still fail when less than two arguments are passed:
$ python test.py arg
usage: test.py [-h] file file [file ...]
test.py: error: too few arguments

Couldn't you do something like this:
import argparse
parser = argparse.ArgumentParser(description = "Compare files")
parser.add_argument('first', help="the first file")
parser.add_argument('other', nargs='+', help="the other files")
args = parser.parse_args()
print args
When I run this with -h I get:
usage: script.py [-h] first other [other ...]
Compare files
positional arguments:
first the first file
other the other files
optional arguments:
-h, --help show this help message and exit
When I run it with only one argument, it won't work:
usage: script.py [-h] first other [other ...]
script.py: error: too few arguments
But two or more arguments is fine. With three arguments it prints:
Namespace(first='one', other=['two', 'three'])