Develop Reference

Argparse and mutually exclusive command line arguments

I have created a pastebin terminal client in python. It can take some command line arguments, like -o to open a file, -n to set a paste name etc. It also has option -l which lists the pastes and allows you to delete or view pastes. The problem is that I don't know how to do it in a nice way (using argparse) - it should not allow to use -l with any other options.
I added a simple logic:
if args.name:
if args.list:
print('The -l should be used alone. Check "pb -h" for help.')
sys.exit()
Can it be done using just argparse?
I know about mutually exclusive groups, I even have one (to set paste privacy) but I haven't figured this one yet.
Full code is available here: https://github.com/lkorba/pbstc/blob/master/pb

I don't think you can use argparse to achieve your goal in "a nice way" as you say.
I see 2 options here:
1) The simpler solution as I get it would be to just check your arguments after parsing them. Nothing fancy just:
args = parser.parse_args()
if args.list is not None:
if not (args.name is None and args.open is None and
args.public is None and args.format is None and args.exp is None):
parser.error('Cannot use list with name, open, public, format or exp argument')
2) On the other hand you could revise a bit your program and use
subparsers like:
subparsers = parser.add_subparsers(title="commands", dest="command")
parser_a = subparsers.add_parser('list', help='list help')
parser_b = subparsers.add_parser('action', help='Any action here')
parser_b.add_argument('-f', action="store", help='Choose paste format/syntax: text=None, '
'mysql=MYSQL, perl=Perl, python=Python etc...')
parser_b.add_argument('-n', '--name', action="store")
parser_b.add_argument('-o', '--open', action="store", help='Open file')
...
args = parser.parse_args()
if args.command == 'list':
...
elif args.command == 'action':
...
So, for example if you pass list -n='Name' as arguments in the latter case you will get an error:
usage: subparser_example.py [-h] {list,action} ...
subparser_example.py: error: unrecognized arguments: -n='Name'
Of course you also get (as overhead) one extra parameter action here...

Configure argparse to accept quoted arguments

I am writing a program which, among other things, allows the user to specify through an argument a module to load (and then use to perform actions). I am trying to set up a way to easily pass arguments through to this inner module, and I was attempting to use ArgParse's action='append' to have it build a list of arguments that I would then pass through.
Here is a basic layout of the arguments that I am using
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
required=True)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
However - if I then try to run this program with python my_program -M module:class -A "-f filename" (where I would like to pass through the -f filename to my module) it seems to be parsing the -f as its own argument (and I get the error my_program: error: argument -A/--module_args: expected one argument
Any ideas?

With:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
import sys
print(sys.argv)
print(parser.parse_args())
I get:
1028:~/mypy$ python stack45146728.py -M module:class -A "-f filename"
['stack45146728.py', '-M', 'module:class', '-A', '-f filename']
Namespace(module='module:class', module_args=['-f filename'])
This is using a linux shell. The quoted string remains one string, as seen in the sys.argv, and is properly interpreted as an argument to -A.
Without the quotes the -f is separate and interpreted as a flag.
1028:~/mypy$ python stack45146728.py -M module:class -A -f filename
['stack45146728.py', '-M', 'module:class', '-A', '-f', 'filename']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
Are you using windows or some other OS/shell that doesn't handle quotes the same way?
In Argparse `append` not working as expected
you asked about a slightly different command line:
1032:~/mypy$ python stack45146728.py -A "-k filepath" -A "-t"
['stack45146728.py', '-A', '-k filepath', '-A', '-t']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
As I already noted -k filepath is passed through as one string. Because of the space, argparse does not interpret that as a flag. But it does interpret the bare '-t' as a flag.
There was a bug/issue about the possibility of interpreting undefined '-xxx' strings as arguments instead of flags. I'd have to look that up to see whether anything made it into to production.
Details of how strings are categorized as flag or argument can be found in argparse.ArgumentParser._parse_optional method. It contains a comment:
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
http://bugs.python.org/issue9334 argparse does not accept options taking arguments beginning with dash (regression from optparse) is an old and long bug/issue on the topic.

The solution is to accept arbitrary arguments - there's an example in argparse's doc here:
argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

argparse option of options

I am trying to add option of options in argparse.
Currently I have:
group = parser.add_mutually_exclusive_group()
group.add_argument("--md", help="Create xyz file for each ionic step for"
" visualization", action='store_true')
group.add_argument("--force", help="See which atom has maximum force",
action='store_true')
group.add_argument("--opt", help="grep string from file",
nargs=2, metavar=("str", "file"))
parser.add_argument("--xsf", help="Create xsf file for md(default is xyz)"
" visualization", action='store_true')
parser.add_argument("-N", help="Showing first N line",
metavar='integer', type=int)
parser.add_argument("-n", help="Showing last n line",
metavar='integer', type=int)
args = parser.parse_args()
which gives:
./foo.py --h
usage: foo.py [-h]
[--md | --force | --opt str file]
[--xsf] [-N integer] [-n integer]
But I want --xsf as a suboption for --md, -N,-n for --opt; e.g.
./foo.py --h
usage: foo.py [-h]
[--md [--xsf]| --force | --opt str file [-N integer] [-n integer]]
But I dont know how to achieve that. May be I am missing something, but there is no option like that in argparse doc
Is there any other way of getting that?

The mutually_exclusive_group mechanism is quite simple, and does not work with any kind of nesting, or subgrouping.
There is a Python bug/issue requesting a more comprehensive grouping mechanism, but the proposed patch is rather complicated. The problem isn't just with testing, it's with defining the groups in a user friendly way, and with generating the usage line. It's nice that you included a desired usage, but that format is well beyond the capabilities of the current help formatter.
You might look into recasting your problem as a subparser one. subparsers are mutually exclusive (you can only give one command name), and you could specify --xsf as an argument for md, and -N as argument for --opt. But subparsers has its own help issues.
Another route is to write your own usage, and do your own testing of arguments after parsing. With a suitable choice of defaults you can usually tell whether an argument has been provided or not (the user can't specify None) or you can ignore unnecessary ones.

Python Can metavar of argparse option be accessed?

I was wondering if there was a way to use the metavar from argparse be grabbed from elsewhere. Example, there is a -f FILE option, and a -d DIR option. Can I make the help of -d grab the file.metavar or some such?
Maybe:
parser.add_argument("-f", "--file",
metavar = 'FILE',
action="store", dest="file_name", default="foo.bar",
help="Name of the {} to be loaded".format(metavar))
parser.add_argument("-d", "--dir",
metavar = 'DIR',
action = 'store', dest='file_dir', default=".",
help="Directory of {} if it is not current directory".format(option.file_name.metavar)
I know this code is wrong (string doesn't have metavar and options doesn't get set until the parser.parse_args() is run), but I have a few other times I want to just grab the metavar without having a bunch of:
meta_file = 'FILE'
meta_dir = 'DIR'
meta_inquisition = 'SURPRISE'
floating around.
Thanks.
EDIT: s/add_option/add_argument/

In argparse it is add_argument(). This method returns an Action object (actually a subclass of that depending the action parameter). You can access various parameters of that object, either to use, or even change. For example:
action1 = parser.add_argument(
"-f",
"--file",
metavar = "FILE",
dest="file_name",
default="foo.bar",
help="Name of the %(metavar)s to be loaded"
)
action2 = parser.add_argument(
"-d",
"--dir",
metavar="DIR",
dest="file_dir",
default=".",
help="Directory of %s if it is not current directory" % action1.metavar
)
print(action1.metavar) # read the metavar attribute
action2.metvar = "DIRECTORY" # change the metavar attribute
The help reads:
usage: ipython [-h] [-f FILE] [-d DIR]
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE Name of the FILE to be loaded
-d DIR, --dir DIR Directory of FILE if it is not current directory
I removed action="store" since that is the default value (no big deal though).
I changed the help values to use %(metavar)s. This is used to incorporate various action attributes. Most commonly it is used for the default.
From the argparse docs:
The help strings can include various format specifiers to avoid repetition of things like the program name or the argument default. The available specifiers include the program name, %(prog)s and most keyword arguments to add_argument(), e.g. %(default)s, %(type)s, etc.:
I am using action1.metavar to place FILE in the help line for action2. It's not a common usage, but I don't see anything wrong with it.
Note that action1.metavar is used once when setting up the parser (to create the action2 help line), and then later when the help is formatted.
In [17]: action2.help
Out[17]: 'Directory of FILE if it is not current directory'
In [18]: action1.help
Out[18]: 'Name of the %(metavar)s to be loaded'
py3 style formatting can be use for action2:
help2 = "Directory of {} if it is not current directory".format(action2.metavar)
action2.help = help2
but py2 style has to be used for action1. Unless you did:
action1.help = "Name of the {} to be loaded".format(action1.metavar)
After creating both actions you even could use:
action1.help = "Name of the {} to be loaded from {}".format(action1.metavar, action2.metavar)
But that's just ordinary Python coding.

According to the docs for the help argument, the help text can contain format specifiers like %(default)s, %(type)s, etc. You can write %(metavar)s and it will be expanded to the value of the metavar (or None if it's not specified).
I don't think there's a way to grab another's argument metavar.

With Python's optparse module, how do you create an option that takes a variable number of arguments?

With Perl's Getopt::Long you can easily define command-line options that take a variable number of arguments:
foo.pl --files a.txt --verbose
foo.pl --files a.txt b.txt c.txt --verbose
Is there a way to do this directly with Python's optparse module? As far as I can tell, the nargs option attribute can be used to specify a fixed number of option arguments, and I have not seen other alternatives in the documentation.

This took me a little while to figure out, but you can use the callback action to your options to get this done. Checkout how I grab an arbitrary number of args to the "--file" flag in this example.
from optparse import OptionParser,
def cb(option, opt_str, value, parser):
args=[]
for arg in parser.rargs:
if arg[0] != "-":
args.append(arg)
else:
del parser.rargs[:len(args)]
break
if getattr(parser.values, option.dest):
args.extend(getattr(parser.values, option.dest))
setattr(parser.values, option.dest, args)
parser=OptionParser()
parser.add_option("-q", "--quiet",
action="store_false", dest="verbose",
help="be vewwy quiet (I'm hunting wabbits)")
parser.add_option("-f", "--filename",
action="callback", callback=cb, dest="file")
(options, args) = parser.parse_args()
print options.file
print args
Only side effect is that you get your args in a list instead of tuple. But that could be easily fixed, for my particular use case a list is desirable.

My mistake: just found this Callback Example 6.

I believe optparse does not support what you require (not directly -- as you noticed, you can do it if you're willing to do all the extra work of a callback!-). You could also do it most simply with the third-party extension argparse, which does support variable numbers of arguments (and also adds several other handy bits of functionality).
This URL documents argparse's add_argument -- passing nargs='*' lets the option take zero or more arguments, '+' lets it take one or more arguments, etc.

Wouldn't you be better off with this?
foo.pl --files a.txt,b.txt,c.txt --verbose

I recently has this issue myself: I was on Python 2.6 and needed an option to take a variable number of arguments. I tried to use Dave's solution but found that it wouldn't work without also explicitly setting nargs to 0.
def arg_list(option, opt_str, value, parser):
args = set()
for arg in parser.rargs:
if arg[0] == '-':
break
args.add(arg)
parser.rargs.pop(0)
setattr(parser.values, option.dest, args)
parser=OptionParser()
parser.disable_interspersed_args()
parser.add_option("-f", "--filename", action="callback", callback=arg_list,
dest="file", nargs=0)
(options, args) = parser.parse_args()
The problem was that, by default, a new option added by add_options is assumed to have nargs = 1 and when nargs > 0 OptionParser will pop items off rargs and assign them to value before any callbacks are called. Thus, for options that do not specify nargs, rargs will always be off by one by the time your callback is called.
This callback is can be used for any number of options, just have callback_args be the function to be called instead of setattr.

Here's one way: Take the fileLst generating string in as a string and then use http://docs.python.org/2/library/glob.html to do the expansion ugh this might not work without escaping the *
Actually, got a better way:
python myprog.py -V -l 1000 /home/dominic/radar/*.json <- If this is your command line
parser, opts = parse_args()
inFileLst = parser.largs