Develop Reference

Python - argparse - (Singleton) Argument w/ optional parameter [duplicate]

I'm trying to implement the following argument dependency using the argparse module:
./prog [-h | [-v schema] file]
meaning the user must pass either -h or a file, if a file is passed the user can optionally pass -v schema.
That's what I have now but that doesn't seem to be working:
import argparse
parser = argparse.ArgumentParser()
mtx = parser.add_mutually_exclusive_group()
mtx.add_argument('-h', ...)
grp = mtx.add_argument_group()
grp.add_argument('-v', ...)
grp.add_argument('file', ...)
args = parser.parse_args()
It looks like you can't add an arg group to a mutex group or am I missing something?

If -h means the default help, then this is all you need (this help is already exclusive)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('file')
parser.add_argument('-s','--schema')
parser.parse_args('-h'.split()) # parser.print_help()
producing
usage: stack23951543.py [-h] [-s SCHEMA] file
...
If by -h you mean some other action, lets rename it -x. This would come close to what you describe
parser = argparse.ArgumentParser()
parser.add_argument('-s','--schema', default='meaningful default value')
mxg = parser.add_mutually_exclusive_group(required=True)
mxg.add_argument('-x','--xxx', action='store_true')
mxg.add_argument('file', nargs='?')
parser.parse_args('-h'.split())
usage is:
usage: stack23951543.py [-h] [-s SCHEMA] (-x | file)
Now -x or file is required (but not both). -s is optional in either case, but with a meaningful default, it doesn't matter if it is omitted. And if -x is given, you can just ignore the -s value.
If necessary you could test args after parsing, to confirm that if args.file is not None, then args.schema can't be either.
Earlier I wrote (maybe over thinking the question):
An argument_group cannot be added to a mutually_exclusive_group. The two kinds of groups have different purposes and functions. There are previous SO discussions of this (see 'related'), as well as a couple of relevant Python bug issues. If you want tests that go beyond a simple mutually exclusive group, you probably should do your own testing after parse_args. That may also require your own usage line.
An argument_group is just a means of grouping and labeling arguments in the help section.
A mutually_exclusive_group affects the usage formatting (if it can), and also runs tests during parse_args. The use of 'group' for both implies that they are more connected than they really are.
http://bugs.python.org/issue11588 asks for nested groups, and the ability to test for 'inclusivity' as well. I tried to make the case that 'groups' aren't general enough to express all the kinds of testing that users want. But it's one thing to generalize the testing mechanism, and quite another to come up with an intuitive API. Questions like this suggest that argparse does need some sort of 'nested group' syntax.

How do you get argparse to choose a default subparser?

I have the following code in script.py:
import argparse
parser = argparse.ArgumentParser()
sp = parser.add_subparsers(dest='command')
sp.default = 'a'
a_parser = sp.add_parser('a')
b_parser = sp.add_parser('b')
a_parser.add_argument('--thing', default='thing')
b_parser.add_argument('--nothing', default='nothing')
args = parser.parse_args()
print(args)
I can call this three different ways:
$ python3 script.py
Namespace(command='a')
$ python3 script.py a
Namespace(command='a', thing='thing')
$ python3 script.py b
Namespace(command='b', nothing='nothing')
There's only one problem with this: what I want is that if I provide zero arguments on the command line is for a_parser to be the one that ends out parsing and doing stuff. Clearly it's not, the sp.default is just setting command='a', not what I expect, which is to say, "Oh yeah, the user didn't provide any arguments on the command line, but I know that this should be processed by a_parser. Here's Namespace(command='a', thing='thing')!"
Is there a way that I can do this with argparse? I've looked for a few different options, but none of them really seem to provide what I'm after. I guess I could do some jiggery with making 3 distinct ArgumentParsers and then passing on the arguments to each of them, though that sounds a bit gross.
Any better options?

First a historical note - subparsers were not optional, and they still aren't in Python2. The fact that they are optional in Py3 is something of a bug that was introduced several years ago. There was a change in the test for required arguments, and subparsers (a kind of positional) fell through the cracks. If done right you should have had to explicitly set subparsers as not-required.
Subparsers don't behave like other non-required arguments, ones with nargs='?' or flagged without the required parameter.
In any case, your sp.default defines the value that will be put in the command dest, but it does not trigger the use of a_parser. That command='a' is never 'evaluated'.
The use of parse_known_args might allow you to devaluate the remaining strings with a_parser.
Without any arguments, we can do:
In [159]: args, extras = parser.parse_known_args([])
In [160]: args
Out[160]: Namespace(command='a')
In [161]: extras
Out[161]: []
Then conditionally run a_parser (if command is 'a' but no 'thing')
In [163]: a_parser.parse_args(extras,namespace=args)
Out[163]: Namespace(command='a', thing='thing')
But if I try to include a --thing value:
In [164]: args, extras = parser.parse_known_args('--thing ouch'.split())
usage: ipython3 [-h] {a,b} ...
ipython3: error: argument command: invalid choice: 'ouch' (choose from 'a', 'b')
It tries to parse the 'ouch' as subparser name. The main parser doesn't known anything about the --thing argument.
As I explained in the other argparse question today, the toplevel parser parses the inputs until it finds something that fits the 'subparsers' command (or in this example raises an error). Then it passes the parsing to the subparser. It does not resume parsing after.
Add top level argparse arguments after subparser args
How to Set a Default Subparser using Argparse Module with Python 2.7
My answer to this Py2 request might work for you. I first run a parse_known_args with a parser that doesn't have subparsers, and conditionally run a second parser that handles the subparsers.
In [165]: firstp = argparse.ArgumentParser()
In [166]: args, extras = firstp.parse_known_args('--thing ouch'.split())
In [167]: args
Out[167]: Namespace()
If extras doesn't have 'a' or 'b' call a_parser (alternatively just look at sys.argv[1:] directly):
In [168]: extras
Out[168]: ['--thing', 'ouch']
In [169]: a_parser.parse_args(extras)
Out[169]: Namespace(thing='ouch')
Or modify extras to include the missing subparser command:
In [170]: extras = ['a']+extras
In [171]: parser.parse_args(extras)
Out[171]: Namespace(command='a', thing='ouch')
In any case, optional subparsers is not well developed in argparse. It's the side effect of a change made a while back, rather than a well thought out feature.

Using argparse with argument values that begin with a dash ("-") [duplicate]

Is there a way to make argparse recognize anything between two quotes as a single argument? It seems to keep seeing the dashes and assuming that it's the start of a new option
I have something like:
mainparser = argparse.ArgumentParser()
subparsers = mainparser.add_subparsers(dest='subcommand')
parser = subparsers.add_parser('queue')
parser.add_argument('-env', '--extraEnvVars', type=str,
help='String of extra arguments to be passed to model.')
...other arguments added to parser...
But when I run:
python Application.py queue -env "-s WHATEVER -e COOL STUFF"
It gives me:
Application.py queue: error: argument -env/--extraEnvVars: expected one argument
If I leave off the first dash, it works totally fine, but it's kind of crucial that I be able to pass in a string with dashes in it. I've tried escaping it with \ , which causes it to succeed but adds the \ to the argument string Does anyone know how to get around this? This happens whether or not -s is an argument in parser.
EDIT: I'm using Python 2.7.
EDIT2:
python Application.py -env " -env"
works perfectly fine, but
python Application.py -env "-env"
does not.
EDIT3: Looks like this is actually a bug that's being debated already: http://www.gossamer-threads.com/lists/python/bugs/89529, http://python.6.x6.nabble.com/issue9334-argparse-does-not-accept-options-taking-arguments-beginning-with-dash-regression-from-optp-td578790.html. It's only in 2.7 and not in optparse.
EDIT4: The current open bug report is: http://bugs.python.org/issue9334

Updated answer:
You can put an equals sign when you call it:
python Application.py -env="-env"
Original answer:
I too have had troubles doing what you are trying to do, but there is a workaround build into argparse, which is the parse_known_args method. This will let all arguments that you haven't defined pass through the parser with the assumption that you would use them for a subprocess. The drawbacks are that you won't get error reporting with bad arguments, and you will have to make sure that there is no collision between your options and your subprocess's options.
Another option could be to force the user's to use a plus instead of a minus:
python Application.py -e "+s WHATEVER +e COOL STUFF"
and then you change the '+' to '-' in post processing before passing to your subprocess.

This issue is discussed in depth in http://bugs.python.org/issue9334. Most of the activity was in 2011. I added a patch last year, but there's quite a backlog of argparse patches.
At issue is the potential ambiguity in a string like '--env', or "-s WHATEVER -e COOL STUFF" when it follows an option that takes an argument.
optparse does a simple left to right parse. The first --env is an option flag that takes one argument, so it consumes the next, regardless of what it looks like. argparse, on the other hand, loops through the strings twice. First it categorizes them as 'O' or 'A' (option flag or argument). On the second loop it consumes them, using a re like pattern matching to handle variable nargs values. In this case it looks like we have OO, two flags and no arguments.
The solution when using argparse is to make sure an argument string will not be confused for an option flag. Possibilities that have been shown here (and in the bug issue) include:
--env="--env" # clearly defines the argument.
--env " --env" # other non - character
--env "--env " # space after
--env "--env one two" # but not '--env "-env one two"'
By itself '--env' looks like a flag (even when quoted, see sys.argv), but when followed by other strings it does not. But "-env one two" has problems because it can be parsed as ['-e','nv one two'], a `'-e' flag followed by a string (or even more options).
-- and nargs=argparse.PARSER can also be used to force argparse to view all following strings as arguments. But they only work at the end of argument lists.
There is a proposed patch in issue9334 to add a args_default_to_positional=True mode. In this mode, the parser only classifies strings as option flags if it can clearly match them with defined arguments. Thus '--one' in '--env --one' would be classed as as an argument. But the second '--env' in '--env --env' would still be classed as an option flag.
Expanding on the related case in
Using argparse with argument values that begin with a dash ("-")
parser = argparse.ArgumentParser(prog="PROG")
parser.add_argument("-f", "--force", default=False, action="store_true")
parser.add_argument("-e", "--extra")
args = parser.parse_args()
print(args)
produces
1513:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra='--foo one', force=False)
1513:~/mypy$ python3 stack16174992.py --extra "-foo one"
usage: PROG [-h] [-f] [-e EXTRA]
PROG: error: argument -e/--extra: expected one argument
1513:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra='-bar one', force=False)
1514:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra='one', force=True)
The "-foo one" case fails because the -foo is interpreted as the -f flag plus unspecified extras. This is the same action that allows -fe to be interpreted as ['-f','-e'].
If I change the nargs to REMAINDER (not PARSER), everything after -e is interpreted as arguments for that flag:
parser.add_argument("-e", "--extra", nargs=argparse.REMAINDER)
All cases work. Note the value is a list. And quotes are not needed:
1518:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra=['--foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-foo one"
Namespace(extra=['-foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra=['-bar one'], force=False)
1519:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra=['one'], force=True)
1520:~/mypy$ python3 stack16174992.py --extra --foo one
Namespace(extra=['--foo', 'one'], force=False)
1521:~/mypy$ python3 stack16174992.py --extra -foo one
Namespace(extra=['-foo', 'one'], force=False)
argparse.REMAINDER is like '*', except it takes everything that follows, whether it looks like a flag or not. argparse.PARSER is more like '+', in that it expects a positional like argument first. It's the nargs that subparsers uses.
This uses of REMAINDER is documented, https://docs.python.org/3/library/argparse.html#nargs

You can start the argument with a space python tst.py -e ' -e blah' as a very simple workaround. Simply lstrip() the option to put it back to normal, if you like.
Or, if the first "sub-argument" is not also a valid argument to the original function then you shouldn't need to do anything at all. That is, the only reason that python tst.py -e '-s hi -e blah' doesn't work is because -s is a valid option to tst.py.
Also, the optparse module, now deprecated, works without any issue.

I have ported a script from optparse to argparse, where certain arguments took values that could start with a negative number. I ran into this problem because the script is used in many places without using the '=' sign to join negative values to the flag. After reading the discussion here and in http://bugs.python.org/issue9334, I know the arguments only take one value and there was no risk in accepting a succeeding argument (ie, a missing value) as the value. FWIW, my solution was to preprocess the arguments and join the problematic ones with '=' before passing to parse_args():
def preprocess_negative_args(argv, flags=None):
if flags is None:
flags = ['--time', '--mtime']
result = []
i = 0
while i < len(argv):
arg = argv[i]
if arg in flags and i+1 < len(argv) and argv[i+1].startswith('-'):
arg = arg + "=" + argv[i+1]
i += 1
result.append(arg)
i += 1
return result
This approach at least does not require any user changes, and it only modifies the arguments which explicitly need to allow negative values.
>>> import argparse
>>> parser = argparse.ArgumentParser("prog")
>>> parser.add_argument("--time")
>>> parser.parse_args(preprocess_negative_args("--time -1d,2".split()))
Namespace(time='-1d,2')
It would be more convenient to tell argparse which arguments should explicitly allow values with a leading dash, but this approach seems like a reasonable compromise.

Similar problem. And I solve this by replace space by "\ ". For example:
replace
python Application.py "cmd -option"
by
python Application.py "cmd\ -option".
Not sure for your problem.

paser.add_argument("--argument_name", default=None, nargs=argparse.REMAINDER)
python_file.py --argument_name "--abc=10 -a=1 -b=2 cdef"
Note: the argument values have to be passed only within double quotes and this doesn't work with single quotes

To bypass having to deal with argparse even looking at a '-' for something that isn't a flag you want, you can edit sys.argv before argparse reads it. Just save the argument that you don't want seen, put a filler argument in it's place, and then replace the filler with the original after argparse process sys.argv. I just had to do this for my own code. It's not pretty, but it works and it's easy. You could also use a for loop to iterate through sys.argv if your flags aren't always in the same order.
parser.add_argument('-n', '--input', nargs='*')
spot_saver = ''
if sys.argv[1] == '-n': #'-n' can be any flag you use
if sys.argv[2][0] == '-': #This checks the first character of the element
spot_saver = sys.argv[2]
sys.argv[2] = "fillerText"
args = parser.parse_args()
if args.input[0] == 'fillerText':
args.input[0] = spot_saver

How to require one command line action argument among several possible but exclusive?

Using argparse, is there a simple way to specify arguments which are mutually exclusive so that the application asks for one of these arguments have to be provided but only one of them?
Example of fictive use-case:
> myapp.py foo --bar
"Foo(bar) - DONE"
> myapp.py read truc.txt
"Read: truc.txt - DONE"
>myapp.py foo read
Error: use "myapp.py foo [options]" or "myapp.py read [options]" (or something similar).
> myapp.py foo truc.txt
Error: "foo" action don't need additional info.
> myapp.py read --bar
Error: "read" action don't have a "--bar" option.
My goal is to have a "driver" application(1) that would internally apply one action depending on the first command line argument and have arguments depending on the action.
So far I see no obvious ways to do this with argparse without manually processing the arguments myself, but maybe I missed something Pythonic? (I'm not a Python3 expert...yet)
I call it "driver" because it might be implemented by calling another application, like gcc does with different compilers.

What you're trying to do is actually supported quite well in Python.
See Mutual Exclusion
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('foo', dest='foo', nargs=1)
group.add_argument('read', dest='read', nargs=1)
args = parser.parse_args()
return args

Mutual exclusion between argument groups

I'm trying to implement the following argument dependency using the argparse module:
./prog [-h | [-v schema] file]
meaning the user must pass either -h or a file, if a file is passed the user can optionally pass -v schema.
That's what I have now but that doesn't seem to be working:
import argparse
parser = argparse.ArgumentParser()
mtx = parser.add_mutually_exclusive_group()
mtx.add_argument('-h', ...)
grp = mtx.add_argument_group()
grp.add_argument('-v', ...)
grp.add_argument('file', ...)
args = parser.parse_args()
It looks like you can't add an arg group to a mutex group or am I missing something?

If -h means the default help, then this is all you need (this help is already exclusive)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('file')
parser.add_argument('-s','--schema')
parser.parse_args('-h'.split()) # parser.print_help()
producing
usage: stack23951543.py [-h] [-s SCHEMA] file
...
If by -h you mean some other action, lets rename it -x. This would come close to what you describe
parser = argparse.ArgumentParser()
parser.add_argument('-s','--schema', default='meaningful default value')
mxg = parser.add_mutually_exclusive_group(required=True)
mxg.add_argument('-x','--xxx', action='store_true')
mxg.add_argument('file', nargs='?')
parser.parse_args('-h'.split())
usage is:
usage: stack23951543.py [-h] [-s SCHEMA] (-x | file)
Now -x or file is required (but not both). -s is optional in either case, but with a meaningful default, it doesn't matter if it is omitted. And if -x is given, you can just ignore the -s value.
If necessary you could test args after parsing, to confirm that if args.file is not None, then args.schema can't be either.
Earlier I wrote (maybe over thinking the question):
An argument_group cannot be added to a mutually_exclusive_group. The two kinds of groups have different purposes and functions. There are previous SO discussions of this (see 'related'), as well as a couple of relevant Python bug issues. If you want tests that go beyond a simple mutually exclusive group, you probably should do your own testing after parse_args. That may also require your own usage line.
An argument_group is just a means of grouping and labeling arguments in the help section.
A mutually_exclusive_group affects the usage formatting (if it can), and also runs tests during parse_args. The use of 'group' for both implies that they are more connected than they really are.
http://bugs.python.org/issue11588 asks for nested groups, and the ability to test for 'inclusivity' as well. I tried to make the case that 'groups' aren't general enough to express all the kinds of testing that users want. But it's one thing to generalize the testing mechanism, and quite another to come up with an intuitive API. Questions like this suggest that argparse does need some sort of 'nested group' syntax.