Develop Reference

Python argparse proper formatting of choice arguments with flag

I'm trying to create well formatted help messages for 'choice' type command line arguments with Python's argparse. For the command I allow the name '--operation' and the alias '-o'. Currently, argparse is printing the list of options next to both in the help message.
Please note that this question is different to the question of formatting the help messages of the options (That problem has a good answer here by Anthon: Python argparse: How to insert newline in the help text?)
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('-o', '--operation', help="operation to perform", type=str, choices=["create", "update", "delete"])
_StoreAction(option_strings=['-o', '--operation'], dest='operation', nargs=None, const=None, default=None, type=<class 'str'>, choices=['create', 'update', 'delete'], help='operation to perform', metavar=None)
>>> parser.print_help()
usage: [-h] [-o {create,update,delete}]
optional arguments:
-h, --help show this help message and exit
-o {create,update,delete}, --operation {create,update,delete}
operation to perform
>>>
My problem is this line:
-o {create,update,delete}, --operation {create,update,delete}
It's very clunky how the list of options is repeated twice. Especially since I will have lists that are even longer. It would be better I think to have this:
-o, --operation {create,update,delete}
This is assuming of course that there isn't some POSIX rule about how this has to work. I don't think there is.
How can I achieve the desired output? Or is there a good reason that I shouldn't be trying to?

This is quite a hack, but there doesn't appear to be a good place to hook into this.
Define your own formatter, which overrides (by basically copying) the _format_action_invocation method. The only change you'll make is to add the choices only to the last option string.
class MyHelpFormatter(HelpFormatter):
def _format_action_invocation(self, action):
if not action.option_strings:
default = self._get_default_metavar_for_positional(action)
metavar, = self._metavar_formatter(action, default)(1)
return metavar
else:
parts = []
# if the Optional doesn't take a value, format is:
# -s, --long
if action.nargs == 0:
parts.extend(action.option_strings)
# if the Optional takes a value, format is:
# -s ARGS, --long ARGS
else:
default = self._get_default_metavar_for_optional(action)
args_string = self._format_args(action, default)
for option_string in action.option_strings[:-1]:
parts.append('%s' % (option_string,))
parts.append('%s %s' % (action.option_strings[-1], args_string)
return ', '.join(parts)

Understanding argument parsing with argparse in Python

I am now starting exploring Python, and was testing how arguments can be passed to a script with "argparse".
The way I wrote a sample script was as following, where arguments passed through flags -i and -o are compulsory and flag -u is optional:
#!/usr/bin/python
import sys
import argparse
## set usage options and define arguments
usage = "usage: %prog [options]"
parser = argparse.ArgumentParser(usage)
parser.add_argument("-i", action="store", dest="input", help="input file")
parser.add_argument("-o", action="store", dest="output", help="output file")
parser.add_argument("-u", action="store_true", dest="isunfolded", default=False, help="optional flag")
args = parser.parse_args()
print len(sys.argv)
if len(sys.argv) < 2:
# parser.print_help()
print 'Incorrect number of params'
exit()
else:
print "Correct number of params: ", len(sys.argv)
Running this script:
> ./test-args.py -i a -o b
prints:
5
Correct number of params: 5
I understand the printing statement in the if conditional (5 is higher than 2), however, after reading the argparse documentation (https://docs.python.org/3/library/argparse.html) I still don't quite understand why -i and -o flags are counted as arguments. This behaviour seems to be quite different from e.g. perl Getopt::Std, which I'm more used to.
So, the question is what is the best way of parsing arguments in Python and to evaluate the presence of mandatory arguments (without using required=True)

It gives you 5 because sys.argv contains the raw input passed to python as arguments (the script name and 4 arguments).
You can see argparse as an abstraction for this, so once you use it, you can forget about sys.argv. In most cases it is better not to mix these two methods.
argparse its a nice way to handle arguments, I don't quite get why you don't want to use the required option when that's exactly the way to go. Another alternative is to parse the sys.argv yourself (regex maybe?) and drop argparse altogether.

There's a Python getopt which probably is similar to the Perl one (assuming both are modelled after the C/Unix version).
https://docs.python.org/2/library/getopt.html
In your code, sys.argv is a list of strings from the command line (as interpreted by the shell and interpreter). It is the raw input for any of the parsers ('getopt', 'optparse', 'argparse'). And it is available for your parsing as well. When learning it is a good idea to include a
print sys.argv
line. parser.parse_args() uses this list. sys.argv[0] is used as prog attribute (in the default usage), while sys.argv[1:] (the rest) is parsed according to the rules you define with add_argument. For testing I often like to use parse_args with a custom list of strings, e.g.
print parser.parse_args(['-i', 'input', '-o', 'output', '-u'])
With your definition I'd expect to see something like:
Namespace(input='input', output='output', isunfolded=True)
The parser returns an object (type argparse.Namespace), which has attributes defined by your arguments. Values are usually accessed with expressions like args.input, args.isunfolded. The docs also show how you easily express this as a dictionary.
By long standing UNIX conventions, arguments flagged by strings like '-i' are options, that is they are optional. argparse generalizes this concept by letting you specify a required=True parameter.
Other arguments are positionals. They are interpreted according to their order. And as such they are normally required. What argparse adds is the ability to define those positionals, such as type, nargs, etc. With nargs='?' they are optional. Many of the nargs values are similar to regular expression characters (e.g. +?*). In fact argparse uses a form a regular expression parsing to allocate strings among arguments.
I'd refine your arguments thus (taking advantage of various defaults)
a1 = parser.add_argument("-i", "--input", help="input file") # 'store' is the default
a2 = parser.add_argument("-o", "--output",help="output file") # use the --output as dest
a3 = parser.add_argument("-u", "--isunfolded", action="store_true", help="optional flag")
If input and output were required, I could change them to:
parser.add_argument("input", help="input file") # 'store' is the default
parser.add_argument("output",help="output file") # use the --output as dest
parser.add_argument("-u", "--isunfolded", action="store_true", help="optional flag")
Now input and output are positional arguments, as in test.py -u inputfile outputfile
By using a1 = parser... I can look at the object produced by this statement.
print a1
produces
_StoreAction(option_strings=['-i', '--input'], dest='input', nargs=None, const=None,
default=None, type=None, choices=None, help='input file', metavar=None)
This tells me that a1 is a _StoreAction object (a subclass of argparse.Action). It also displays a number (not all) of its attributes, ones that define its action. A positional, on the other hand, has values like these:
a2 = p.add_argument("output", help="output file")
_StoreAction(option_strings=[], dest='output', nargs=None, const=None,
default=None, type=None, choices=None, help='output file', metavar=None)
It may also be instructive to look at a1.required and a2.required, which are respectively False and True. required is an Action attribute that is not routinely displayed, but is, never the less accessible.
I've pulled all these test values from a parser defined in an interactive shell (Ipython). It's a great way to explore Python and modules like argparse.

After reading other related posts it seems that the best way to do this is as was suggested by #Rufflewind and inspect the args itself:
if not args.input or not args.output:
print 'Incorrect number of params'
exit()
else:
print "Correct number of params"

Argparse optional stdin argument

I am trying to specify an optional argument that takes stdin. This will be mainly used for piping data in my program, so someprog that outputs | python my_prog.
I followed the argparse documentation and I read a lot of questions/answers on this on Stackoverflow but none of them seem to work for me.
Here's what I originally have:
parser = argparse.ArgumentParser(description='Upgrade Instance.')
parser.add_argument('--app', '-a', dest='app', action='store', required=True)
parser.add_argument('--version', '-v', dest='version', action='store', default='', required=False)
parser.add_argument('--config', '-c', dest='config', action='store', default = '', required=False)
args = parser.parse_args()
Now what I want to do is allow the user to pass in version using a pipe, instead of passing it in.
I added parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin) to the top but that makes it a positional argument. How is that possible? I thought nargs=? makes it optional.
I need it to be an optional argument. So I changed it to:
parser.add_argument('--infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin)
This makes it an optional argument, but the program hangs waiting for stdin as thats default, if no pipe is passed. Removing the default=sys.stdin and piping something into my program I get:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
when running it. When I print args, I get: Namespace(app='app', config='', g=False, hosts='03.app', infile=None, version='').
It seems what I am doing is very simple, common and many people asked about it. But it doesn't seem to be working with me.
Any suggestions on how I can get it working?

This does it... without specifying arguments. If you pass pipe input to the program it goes, it you don't, it still goes. raw_input() will work as well.
import sys
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
print stdin
sys.stdin = open('/dev/tty')
else:
print "No stdin"
test_raw = raw_input()
print test_raw
Demo -
rypeck$ echo "stdin input" | python test_argparse.py -a test
['stdin input\n']
raw_input working!
raw_input working!
rypeck$ python test_argparse.py -a test
No stdin
raw_input working!
raw_input working!

I was poking at this issue myself and found a small improvement on the options here--hopefully it'll help future travelers. It's not 100% clear if you were hoping to only read from stdin when you provide a flag, but it seems that may have been your goal; my answer is predicated on this assumption. I'm working in 3.4, in case that becomes an issue...
I can declare a single optional argument like:
parser.add_argument("-v", "--version", nargs='?', const=sys.stdin, action=StreamType)
I'm not specifying a default because it is used only if the option is completely absent, while the const is used only if the flag is present but has no arguments. StreamType is a tiny argparse.Action subclass I wrote which just tries to read a line from the stdin stream and just saves the raw value if that doesn't work:
class StreamType(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
try:
setattr(namespace, self.dest, values.readline().strip())
except AttributeError:
setattr(namespace, self.dest, values)
This should produce something like:
$ blah --version v1.1.1
Namespace(version='v1.1.1')
$ echo "v1.0.3" | blah --version
Namespace(version='v1.0.3')

What do you do with args.infile? since you get a Namespace, argparse is not the part that is hanging or giving the error.
p = argparse.ArgumentParser()
p.add_argument('--infile', type=argparse.FileType('r'),default='-')
# p.add_argument('infile', nargs='?', type=argparse.FileType('r'),default='-') # positional version
args = p.parse_args()
print(args)
print args.infile.read()
-----------
$ cat myprog.py | python myprog.py --infile -
$ cat myprog.py | python myprog.py
$ python myprog.py myprog.py # with the positional version
$ python myprog.py - < myprog.py # with the positional version
echos the code nicely. The second call works with the 'optional positional' as well.
There is an unfortunate overlap in terminology, optional/positional and optional/required.
If a positional argument (yes, another use of 'positional') has a prefix character like - or -- it is called optional. By default its required parameter is False, but you may set it to True. But if the argument is 'infile' (no prefix), it is positional, even though with ? is is optional (not required).
By the way, default action is 'store', so you don't need to specify that. Also you don't need to specify required, unless it is True.
With a FileType, a handy way of specifying stdin is -.
Don't use '?' with --infile unless you really want a None

I'm not sure if I got your question correctly, but even if not, as I came here trying to solve my problem, this answer may help.
import argparse
import sys
def get_args():
parser = argparse.ArgumentParser(prog="my program")
parser.add_argument("--version", type=str, default = None)
args,unknown = parser.parse_known_args()
print(f"normal unknown: {unknown}")
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
stdin = ' '.join( (x.strip() for x in stdin ))
args, unknown2 = parser.parse_known_args(stdin.split(), namespace = args)
unknown.extend(unknown2)
print(f"stdin unknown: {unknown2}")
print(f"args: {args}")
get_args()
Now, I get
echo A --version=1.2.3 B | parse.py --C=some_value
normal unknown: ['--C=some_value']
stdin unknown: ['A', 'B']
args: Namespace(version='1.2.3')

Python argparse: command-line argument that can be either named or positional

I am trying to make a Python program that uses the argparse module to parse command-line options.
I want to make an optional argument that can either be named or positional. For example, I want myScript --username=batman to do the same thing as myScript batman. I also want myScript without a username to be valid. Is this possible? If so, how can it be done?
I tried various things similar to the code below without any success.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin")
group.add_argument("user-name", default="admin")
args = parser.parse_args()
EDIT: The above code throws an exception saying ValueError: mutually exclusive arguments must be optional.
I am using Python 2.7.2 on OS X 10.8.4.
EDIT: I tried Gabriel Jacobsohn's suggestion but I couldn't get it working correctly in all cases.
I tried this:
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin", nargs="?")
group.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)
and running myScript batman would print Namespace(user_name='batman'), but myScript -u batman and myScript --user-name=batman would print Namespace(user_name='admin').
I tried changing the name user-name to user_name in the 1st add_argument line and that sometimes resulted in both batman and admin in the namespace or an error, depending on how I ran the program.
I tried changing the name user_name to user-name in the 2nd add_argument line but that would print either Namespace(user-name='batman', user_name='admin') or Namespace(user-name='admin', user_name='batman'), depending on how I ran the program.

The way the ArgumentParser works, it always checks for any trailing positional arguments after it has parsed the optional arguments. So if you have a positional argument with the same name as an optional argument, and it doesn't appear anywhere on the command line, it's guaranteed to override the optional argument (either with its default value or None).
Frankly this seems like a bug to me, at least when used in a mutually exclusive group, since if you had specified the parameter explicitly it would have been an error.
That said, my suggested solution, is to give the positional argument a different name.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument('-u','--username')
group.add_argument('static_username',nargs='?',default='admin')
Then when parsing, you use the optional username if present, otherwise fallback to the positional static_username.
results = parser.parse_args()
username = results.username or results.static_username
I realise this isn't a particularly neat solution, but I don't think any of the answers will be.

Here is a solution that I think does everything you want:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
# Gather all extra args into a list named "args.extra"
parser.add_argument("extra", nargs='*')
args = parser.parse_args()
# Set args.user_name to first extra arg if it is not given and len(args.extra) > 0
if args.user_name == parser.get_default("user_name") and args.extra:
args.user_name = args.extra.pop(0)
print args
If you run myScript -u batman or myScript --user-name=batman, args.user_name is set to 'batman'. If you do myScript batman, args.user_name is again set to 'batman'. And finally, if you just do myScript, args.user_name is set to 'admin'.
Also, as an added bonus, you now have all of the extra arguments that were passed to the script stored in args.extra. Hope this helps.

Try to use the "nargs" parameter of the add_argument method.
This way it works for me.
Now you can add the username twice,
so you have to check it and raise an error, if you want.
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
parser.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)

gem/git-style command line arguments in Python

Is there a Python module for doing gem/git-style command line arguments? What I mean by gem/git style is:
$ ./MyApp.py
The most commonly used MyApp commands are:
add Add file contents to the index
bisect Find by binary search the change that introduced a bug
branch List, create, or delete branches
checkout Checkout a branch or paths to the working tree
...
$ ./MyApp.py branch
* current-branch
master
With no arguments, the output tells you how you can proceed. And there is a special "help" command:
$ ./MyApp.py help branch
Which gets you deeper tips about the "branch" command.
Edit:
And by doing I mean it does the usage printing for you, exits with invalid input, runs your functions according to your CLI specification. Sort of a "URL mapper" for the command line.

Yes, argparse with add_subparsers().
It's all well explained in the Sub-commands section.
Copying one of the examples from there:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> checkout = subparsers.add_parser('checkout', aliases=['co'])
>>> checkout.add_argument('foo')
>>> parser.parse_args(['checkout', 'bar'])
Namespace(foo='bar')
Edit: Unfortunately there's no self generated special help command, but you can get the verbose help message (that you seem to want) with -h or --help like one normally would after the command:
$ ./MyApp.py branch --help
By verbose I don't mean that is like a man page, it's like every other --help kind of help: listing all the arguments, etc...
Example:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(description='Sub description')
>>> checkout = subparsers.add_parser('checkout', description='Checkout description')
>>> checkout.add_argument('foo', help='This is the foo help')
>>> parser.parse_args(['checkout', '--help'])
usage: checkout [-h] foo
Checkout description
positional arguments:
foo This is the foo help
optional arguments:
-h, --help show this help message and exit
If you need to, it should be easy to implement an help command that redirects to --help.

A reasonable hack to get the gem/git style "help" behavior (I just wrote this for what I'm working on anyway):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='sub_commands')
parser_branch = subparsers.add_parser('branch', description='list of branches')
parser_help = subparsers.add_parser('help')
parser_help.add_argument('command', nargs="?", default=None)
# I can't find a legitimate way to set a default subparser in the docs
# If you know of one, please let me know!
if len(sys.argv) < 2:
sys.argv.append('--help')
parsed = parser.parse_args()
if parsed.sub_commands == "help":
if not parsed.command:
parser.parse_args(['--help'])
else:
parser.parse_args([parsed.command, '--help'])
argparse is definitely a step up from optparse and other python solutions I've come across. But IMO the gem/git style of handling args is just a more logical and safer way to do things so it's annoying that it's not supported.

I wanted to do something similar to git commands, where I would load a second script based off of one of the command line options, and have that script populate more command line options, and also have the help work.
I was able to do this by disabling the help option, parse known args, add more arguments, re-enable the help option, and then parse the rest of the arguments.
This is what I came up with.
import argparse
#Note add_help=False
arg_parser = argparse.ArgumentParser(description='Add more arguments after parsing.',add_help=False)
arg_parser.add_argument('MODE', default='default',type=str, help='What commands to use')
args = arg_parser.parse_known_args()[0]
if args.MODE == 'branch':
arg_parser.add_argument('-d', '--delete', default='Delete a branch')
arg_parser.add_argument('-m', '--move', default='move a branch')
elif args.MODE == 'clone' :
arg_parser.add_argument('--local', '-l')
arg_parser.add_argument('--shared')
#Finally re-enable the help option, and reparse the arguments
arg_parser.add_argument(
'-h', '--help',
action='help', default=argparse.SUPPRESS,
help=argparse._('show this help message and exit'))
args = arg_parser.parse_args()