Develop Reference

Argparse and mutually exclusive command line arguments

I have created a pastebin terminal client in python. It can take some command line arguments, like -o to open a file, -n to set a paste name etc. It also has option -l which lists the pastes and allows you to delete or view pastes. The problem is that I don't know how to do it in a nice way (using argparse) - it should not allow to use -l with any other options.
I added a simple logic:
if args.name:
if args.list:
print('The -l should be used alone. Check "pb -h" for help.')
sys.exit()
Can it be done using just argparse?
I know about mutually exclusive groups, I even have one (to set paste privacy) but I haven't figured this one yet.
Full code is available here: https://github.com/lkorba/pbstc/blob/master/pb

I don't think you can use argparse to achieve your goal in "a nice way" as you say.
I see 2 options here:
1) The simpler solution as I get it would be to just check your arguments after parsing them. Nothing fancy just:
args = parser.parse_args()
if args.list is not None:
if not (args.name is None and args.open is None and
args.public is None and args.format is None and args.exp is None):
parser.error('Cannot use list with name, open, public, format or exp argument')
2) On the other hand you could revise a bit your program and use
subparsers like:
subparsers = parser.add_subparsers(title="commands", dest="command")
parser_a = subparsers.add_parser('list', help='list help')
parser_b = subparsers.add_parser('action', help='Any action here')
parser_b.add_argument('-f', action="store", help='Choose paste format/syntax: text=None, '
'mysql=MYSQL, perl=Perl, python=Python etc...')
parser_b.add_argument('-n', '--name', action="store")
parser_b.add_argument('-o', '--open', action="store", help='Open file')
...
args = parser.parse_args()
if args.command == 'list':
...
elif args.command == 'action':
...
So, for example if you pass list -n='Name' as arguments in the latter case you will get an error:
usage: subparser_example.py [-h] {list,action} ...
subparser_example.py: error: unrecognized arguments: -n='Name'
Of course you also get (as overhead) one extra parameter action here...

Argparse: options for subparsers override main if both share parent

I'm using argparse with several subparsers. I want my program to take options for verbosity anywhere in the args, including the subparser.
from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('--verbose', '-v', action='count')
sub = p.add_subparsers()
a = sub.add_parser('a')
print(p.parse_args())
By default, options for the main parser will throw an error if used for subparsers:
$ python tmp.py -v a
Namespace(verbose=1)
$ python tmp.py a -v
usage: tmp.py [-h] [--verbose] {a} ...
tmp.py: error: unrecognized arguments: -v
I looked into parent parsers, from this answer.
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main = ArgumentParser(parents=[parent])
sub = main.add_subparsers()
a = sub.add_parser('a', parents=[parent])
print(main.parse_args())
For some reason though, none of the shared flags work on the main parser.
$ python tmp2.py a -vvv
Namespace(verbose=3)
$ python tmp2.py -vvv a
Namespace(verbose=None)
Note that the main parser definitely has the appropriate arguments, because when I change it to main = ArgumentParser() I get error: unrecognized arguments: -v. What am I missing here?

First, a couple of general comments.
The main parser handles the input upto the subparser invocation, then the subparser is called and given the remaining argv. When it is done, it's namespace is merged back into the the main namespace.
The parents mechanism copies Actions from the parent by reference. So your main and subparsers share the same verbose Action object. That's been a problem when the subparser tries to set a different default or help. It may not be an issue here, but just keep it in mind.
Even without the parents mechanism, sharing a dest or options flag between main and subparser can be tricky. Should the default of the subparser Action be used? What if both are used? Does the subparser overwrite the main parser's actions?
Originally the main namespace was passed to the subparser, which it modified and returned. This was changed a while back (I can find the bug/issue if needed). Now the subparser starts with a default empty namespace, fills it. And these values are then merged into the main.
So in your linked SO question, be wary of older answers. argparse may have changed since then.
I think what's happening in your case is that the main and subparser verbose are counting separately. And when you get None it's the subparser's default that you see.
The __call__ for _Count_Action is:
def __call__(self, parser, namespace, values, option_string=None):
new_count = _ensure_value(namespace, self.dest, 0) + 1
setattr(namespace, self.dest, new_count)
I suspect that in older argparse when the namespace was shared, the count would have been cumulative, but I can't test it without recreating an older style subparser action class.
https://bugs.python.org/issue15327 - here the original developer suggests giving the two arguments different dest. That records the inputs from both main and sub. Your own code can then merge the results if needed.
https://bugs.python.org/issue27859 argparse - subparsers does not retain namespace. Here I suggest a way of recreating the older style.
https://bugs.python.org/issue9351 argparse set_defaults on subcommands should override top level set_defaults - this is the issue in 2014 that changed the namespace use.

My workaround for this behavior, which is very well described in #hpaulj's answer is to create a second parser that does not have subparsers but only the positional arguments that were first found.
The first parse_args, used with the first parser, will validate the positional arguments and flags, show an error message if needed or show the proper help.
The second parse_args, for the second parser, will correctly fill in the namespace.
Building on your example:
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main1 = ArgumentParser(parents=[parent])
sub = main1.add_subparsers()
# eg: tmp.py -vv a -v
a = sub.add_parser('a', parents=[parent])
a.set_defaults(which='a')
# eg: tmp.py -vv v -v --output toto
b = sub.add_parser('b', parents=[parent])
b.add_argument('--output', type=str)
b.set_defaults(which='b')
args = main1.parse_args()
print(args)
# parse a second time with another parser
main2 = ArgumentParser(parents=[parent])
if args.which == 'a':
main2.add_argument('a')
elif args.which == 'b':
main2.add_argument('b')
main2.add_argument('--output', type=str)
print(main2.parse_args())
Which gives:
$ ./tmp.py -vv a -v
Namespace(verbose=1, which='a')
Namespace(a='a', verbose=3)
$ ./tmp.py -vv b -v --output toto
Namespace(output='toto', verbose=1, which='b')
Namespace(b='b', output='toto', verbose=3)
$ ./tmp.py -vv a --output
usage: tmp.py [-h] [--verbose] {a,b} ...
tmp.py: error: unrecognized arguments: --output
I use this technique with multiple nested subparsers.

Using argparse.REMAINDER at beginning of parser / sub parser

I want to implement an arg parser that allows me to run unittests as one of the sub commands, blindly passing the arguments on to unittest.main(). e.g.,
$ foo.py unittest [args to pass to unittest.main()]
along with other sub commands:
$ foo.py foo ...
$ foo.py bar ...
Following argparse's example, this works:
#!/usr/bin/python
import argparse
p = argparse.ArgumentParser(prog='PROG')
p.add_argument('-v', '--verbose', action='store_true')
sub = p.add_subparsers(dest='cmd')
foo = sub.add_parser('foo')
bar = sub.add_parser('bar')
unittest = sub.add_parser('unittest')
unittest.add_argument('command') # Need to add this to make it work.
unittest.add_argument('args', nargs=argparse.REMAINDER)
print(p.parse_args('unittest command -blah blah'.split()))
Output:
Namespace(args=['-blah', 'blah'], cmd='unittest', command='command', verbose=False)
But this doesn't. It seems to require a "normal" argument first:
#!/usr/bin/python
import argparse
p = argparse.ArgumentParser(prog='PROG')
p.add_argument('-v', '--verbose', action='store_true')
sub = p.add_subparsers(dest='cmd')
foo = sub.add_parser('foo')
bar = sub.add_parser('bar')
unittest = sub.add_parser('unittest')
unittest.add_argument('args', nargs=argparse.REMAINDER)
print(p.parse_args('unittest -blah blah'.split()))
Output:
$ /tmp/foo.py
usage: PROG [-h] [-v] {foo,bar,unittest} ...
PROG: error: unrecognized arguments: -blah
I can do print(p.parse_args('unittest -- -f -g'.split())), but requiring -- kind of defeats the purpose of argparse.REMAINDER.
Is there a way to get argparse to do what I want? Or do I just need to hand parse this case?
Python 2.7.5

Looks like the same issue discussed in http://bugs.python.org/issue17050, argparse.REMAINDER doesn't work as first argument
My deduction from 4 years ago still holds - the -blah is being classed as an optional's flag even before REMAINDER has a chance to act. '--' is parsed earlier, but ... is, in a sense just a generalization of '*'. And not a widely used one. For what it's worth the 'subparsers' Action has a nargs='+...' value (argparse.PARSER) - it's like REMAINDER except it requires at least one string, the 'cmd'.
The possible fix in http://bugs.python.org/issue9334 has not been acted on. So you either need to handle the '-blah' by itself, or use '--'. parse_known_args might also work in your case.

As noted, the existing behavior is bad. One workaround is to implement a simple
ArgumentParser subclass and use that for your subparser:
class SubcommandParser(argparse.ArgumentParser):
"""This subparser puts all remaining arguments in args attribute of namespace"""
def parse_known_args(self, args=None, namespace=None):
if namespace is None:
namespace = argparse.Namespace()
setattr(namespace, 'args', args)
return namespace, []
...
p.add_subparsers(dest='cmd', parser_class=SubcommandParser)

Argparse optional stdin argument

I am trying to specify an optional argument that takes stdin. This will be mainly used for piping data in my program, so someprog that outputs | python my_prog.
I followed the argparse documentation and I read a lot of questions/answers on this on Stackoverflow but none of them seem to work for me.
Here's what I originally have:
parser = argparse.ArgumentParser(description='Upgrade Instance.')
parser.add_argument('--app', '-a', dest='app', action='store', required=True)
parser.add_argument('--version', '-v', dest='version', action='store', default='', required=False)
parser.add_argument('--config', '-c', dest='config', action='store', default = '', required=False)
args = parser.parse_args()
Now what I want to do is allow the user to pass in version using a pipe, instead of passing it in.
I added parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin) to the top but that makes it a positional argument. How is that possible? I thought nargs=? makes it optional.
I need it to be an optional argument. So I changed it to:
parser.add_argument('--infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin)
This makes it an optional argument, but the program hangs waiting for stdin as thats default, if no pipe is passed. Removing the default=sys.stdin and piping something into my program I get:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
when running it. When I print args, I get: Namespace(app='app', config='', g=False, hosts='03.app', infile=None, version='').
It seems what I am doing is very simple, common and many people asked about it. But it doesn't seem to be working with me.
Any suggestions on how I can get it working?

This does it... without specifying arguments. If you pass pipe input to the program it goes, it you don't, it still goes. raw_input() will work as well.
import sys
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
print stdin
sys.stdin = open('/dev/tty')
else:
print "No stdin"
test_raw = raw_input()
print test_raw
Demo -
rypeck$ echo "stdin input" | python test_argparse.py -a test
['stdin input\n']
raw_input working!
raw_input working!
rypeck$ python test_argparse.py -a test
No stdin
raw_input working!
raw_input working!

I was poking at this issue myself and found a small improvement on the options here--hopefully it'll help future travelers. It's not 100% clear if you were hoping to only read from stdin when you provide a flag, but it seems that may have been your goal; my answer is predicated on this assumption. I'm working in 3.4, in case that becomes an issue...
I can declare a single optional argument like:
parser.add_argument("-v", "--version", nargs='?', const=sys.stdin, action=StreamType)
I'm not specifying a default because it is used only if the option is completely absent, while the const is used only if the flag is present but has no arguments. StreamType is a tiny argparse.Action subclass I wrote which just tries to read a line from the stdin stream and just saves the raw value if that doesn't work:
class StreamType(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
try:
setattr(namespace, self.dest, values.readline().strip())
except AttributeError:
setattr(namespace, self.dest, values)
This should produce something like:
$ blah --version v1.1.1
Namespace(version='v1.1.1')
$ echo "v1.0.3" | blah --version
Namespace(version='v1.0.3')

What do you do with args.infile? since you get a Namespace, argparse is not the part that is hanging or giving the error.
p = argparse.ArgumentParser()
p.add_argument('--infile', type=argparse.FileType('r'),default='-')
# p.add_argument('infile', nargs='?', type=argparse.FileType('r'),default='-') # positional version
args = p.parse_args()
print(args)
print args.infile.read()
-----------
$ cat myprog.py | python myprog.py --infile -
$ cat myprog.py | python myprog.py
$ python myprog.py myprog.py # with the positional version
$ python myprog.py - < myprog.py # with the positional version
echos the code nicely. The second call works with the 'optional positional' as well.
There is an unfortunate overlap in terminology, optional/positional and optional/required.
If a positional argument (yes, another use of 'positional') has a prefix character like - or -- it is called optional. By default its required parameter is False, but you may set it to True. But if the argument is 'infile' (no prefix), it is positional, even though with ? is is optional (not required).
By the way, default action is 'store', so you don't need to specify that. Also you don't need to specify required, unless it is True.
With a FileType, a handy way of specifying stdin is -.
Don't use '?' with --infile unless you really want a None

I'm not sure if I got your question correctly, but even if not, as I came here trying to solve my problem, this answer may help.
import argparse
import sys
def get_args():
parser = argparse.ArgumentParser(prog="my program")
parser.add_argument("--version", type=str, default = None)
args,unknown = parser.parse_known_args()
print(f"normal unknown: {unknown}")
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
stdin = ' '.join( (x.strip() for x in stdin ))
args, unknown2 = parser.parse_known_args(stdin.split(), namespace = args)
unknown.extend(unknown2)
print(f"stdin unknown: {unknown2}")
print(f"args: {args}")
get_args()
Now, I get
echo A --version=1.2.3 B | parse.py --C=some_value
normal unknown: ['--C=some_value']
stdin unknown: ['A', 'B']
args: Namespace(version='1.2.3')

gem/git-style command line arguments in Python

Is there a Python module for doing gem/git-style command line arguments? What I mean by gem/git style is:
$ ./MyApp.py
The most commonly used MyApp commands are:
add Add file contents to the index
bisect Find by binary search the change that introduced a bug
branch List, create, or delete branches
checkout Checkout a branch or paths to the working tree
...
$ ./MyApp.py branch
* current-branch
master
With no arguments, the output tells you how you can proceed. And there is a special "help" command:
$ ./MyApp.py help branch
Which gets you deeper tips about the "branch" command.
Edit:
And by doing I mean it does the usage printing for you, exits with invalid input, runs your functions according to your CLI specification. Sort of a "URL mapper" for the command line.

Yes, argparse with add_subparsers().
It's all well explained in the Sub-commands section.
Copying one of the examples from there:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> checkout = subparsers.add_parser('checkout', aliases=['co'])
>>> checkout.add_argument('foo')
>>> parser.parse_args(['checkout', 'bar'])
Namespace(foo='bar')
Edit: Unfortunately there's no self generated special help command, but you can get the verbose help message (that you seem to want) with -h or --help like one normally would after the command:
$ ./MyApp.py branch --help
By verbose I don't mean that is like a man page, it's like every other --help kind of help: listing all the arguments, etc...
Example:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(description='Sub description')
>>> checkout = subparsers.add_parser('checkout', description='Checkout description')
>>> checkout.add_argument('foo', help='This is the foo help')
>>> parser.parse_args(['checkout', '--help'])
usage: checkout [-h] foo
Checkout description
positional arguments:
foo This is the foo help
optional arguments:
-h, --help show this help message and exit
If you need to, it should be easy to implement an help command that redirects to --help.

A reasonable hack to get the gem/git style "help" behavior (I just wrote this for what I'm working on anyway):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='sub_commands')
parser_branch = subparsers.add_parser('branch', description='list of branches')
parser_help = subparsers.add_parser('help')
parser_help.add_argument('command', nargs="?", default=None)
# I can't find a legitimate way to set a default subparser in the docs
# If you know of one, please let me know!
if len(sys.argv) < 2:
sys.argv.append('--help')
parsed = parser.parse_args()
if parsed.sub_commands == "help":
if not parsed.command:
parser.parse_args(['--help'])
else:
parser.parse_args([parsed.command, '--help'])
argparse is definitely a step up from optparse and other python solutions I've come across. But IMO the gem/git style of handling args is just a more logical and safer way to do things so it's annoying that it's not supported.

I wanted to do something similar to git commands, where I would load a second script based off of one of the command line options, and have that script populate more command line options, and also have the help work.
I was able to do this by disabling the help option, parse known args, add more arguments, re-enable the help option, and then parse the rest of the arguments.
This is what I came up with.
import argparse
#Note add_help=False
arg_parser = argparse.ArgumentParser(description='Add more arguments after parsing.',add_help=False)
arg_parser.add_argument('MODE', default='default',type=str, help='What commands to use')
args = arg_parser.parse_known_args()[0]
if args.MODE == 'branch':
arg_parser.add_argument('-d', '--delete', default='Delete a branch')
arg_parser.add_argument('-m', '--move', default='move a branch')
elif args.MODE == 'clone' :
arg_parser.add_argument('--local', '-l')
arg_parser.add_argument('--shared')
#Finally re-enable the help option, and reparse the arguments
arg_parser.add_argument(
'-h', '--help',
action='help', default=argparse.SUPPRESS,
help=argparse._('show this help message and exit'))
args = arg_parser.parse_args()