I want to implement an arg parser that allows me to run unittests as one of the sub commands, blindly passing the arguments on to unittest.main(). e.g.,
$ foo.py unittest [args to pass to unittest.main()]
along with other sub commands:
$ foo.py foo ...
$ foo.py bar ...
Following argparse's example, this works:
#!/usr/bin/python
import argparse
p = argparse.ArgumentParser(prog='PROG')
p.add_argument('-v', '--verbose', action='store_true')
sub = p.add_subparsers(dest='cmd')
foo = sub.add_parser('foo')
bar = sub.add_parser('bar')
unittest = sub.add_parser('unittest')
unittest.add_argument('command') # Need to add this to make it work.
unittest.add_argument('args', nargs=argparse.REMAINDER)
print(p.parse_args('unittest command -blah blah'.split()))
Output:
Namespace(args=['-blah', 'blah'], cmd='unittest', command='command', verbose=False)
But this doesn't. It seems to require a "normal" argument first:
#!/usr/bin/python
import argparse
p = argparse.ArgumentParser(prog='PROG')
p.add_argument('-v', '--verbose', action='store_true')
sub = p.add_subparsers(dest='cmd')
foo = sub.add_parser('foo')
bar = sub.add_parser('bar')
unittest = sub.add_parser('unittest')
unittest.add_argument('args', nargs=argparse.REMAINDER)
print(p.parse_args('unittest -blah blah'.split()))
Output:
$ /tmp/foo.py
usage: PROG [-h] [-v] {foo,bar,unittest} ...
PROG: error: unrecognized arguments: -blah
I can do print(p.parse_args('unittest -- -f -g'.split())), but requiring -- kind of defeats the purpose of argparse.REMAINDER.
Is there a way to get argparse to do what I want? Or do I just need to hand parse this case?
Python 2.7.5
Looks like the same issue discussed in http://bugs.python.org/issue17050, argparse.REMAINDER doesn't work as first argument
My deduction from 4 years ago still holds - the -blah is being classed as an optional's flag even before REMAINDER has a chance to act. '--' is parsed earlier, but ... is, in a sense just a generalization of '*'. And not a widely used one. For what it's worth the 'subparsers' Action has a nargs='+...' value (argparse.PARSER) - it's like REMAINDER except it requires at least one string, the 'cmd'.
The possible fix in http://bugs.python.org/issue9334 has not been acted on. So you either need to handle the '-blah' by itself, or use '--'. parse_known_args might also work in your case.
As noted, the existing behavior is bad. One workaround is to implement a simple
ArgumentParser subclass and use that for your subparser:
class SubcommandParser(argparse.ArgumentParser):
"""This subparser puts all remaining arguments in args attribute of namespace"""
def parse_known_args(self, args=None, namespace=None):
if namespace is None:
namespace = argparse.Namespace()
setattr(namespace, 'args', args)
return namespace, []
...
p.add_subparsers(dest='cmd', parser_class=SubcommandParser)
Related
Optparse, the old version just ignores all unrecognised arguments and carries on. In most situations, this isn't ideal and was changed in argparse. But there are a few situations where you want to ignore any unrecognised arguments and parse the ones you've specified.
For example:
parser = argparse.ArgumentParser()
parser.add_argument('--foo', dest="foo")
parser.parse_args()
$python myscript.py --foo 1 --bar 2
error: unrecognized arguments: --bar
Is there anyway to overwrite this?
Replace
args = parser.parse_args()
with
args, unknown = parser.parse_known_args()
For example,
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
args, unknown = parser.parse_known_args(['--foo', 'BAR', 'spam'])
print(args)
# Namespace(foo='BAR')
print(unknown)
# ['spam']
You can puts the remaining parts into a new argument with parser.add_argument('args', nargs=argparse.REMAINDER) if you want to use them.
Actually argparse does still "ignore" _unrecognized_args. As long as these "unrecognized" arguments don't use the default prefix you will hear no complaints from the parser.
Using #anutbu's configuration but with the standard parse.parse_args(), if we were to execute our program with the following arguments.
$ program --foo BAR a b +cd e
We will have this Namespaced data collection to work with.
Namespace(_unrecognized_args=['a', 'b', '+cd', 'e'], foo='BAR')
If we wanted the default prefix - ignored we could change the ArgumentParser and decide we are going to use a + for our "recognized" arguments instead.
parser = argparse.ArgumentParser(prefix_chars='+')
parser.add_argument('+cd')
The same command will produce
Namespace(_unrecognized_args=['--foo', 'BAR', 'a', 'b'], cd='e')
Put that in your pipe and smoke it =)
nJoy!
I have a script that consumes command line arguments and I would like to implement two argument-passing schemes, namely:
Typing the arguments out at the command line.
Storing the argument list in a file, and passing the name of this file to the program via the command line.
To that end I am passing the argument fromfile_prefix_chars to the ArgumentParser constructor.
script.py
from argparse import ArgumentParser
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
args.txt
--foo
2
--bar
2
Sample use cases
$ python script.py --foo 3
Namespace(bar=1, filename=None, foo='3')
$ python script.py #args.txt --foo 3
Namespace(bar='2', filename=None, foo='3')
I was expecting that args.filename would retain the name of the file, but surprinsingly enough it has the value None instead. I am aware that I could get the file name from sys.argv through a bit of processing. Is there a cleaner way (ideally an argparse-based approach) to elicit the name of the arguments file?
Your script.py, plus the file. I have added the file name to the file itself.
args.txt
args.txt
--foo
2
--bar
2
testing:
1803:~/mypy$ python3 stack56811067.py --foo 3
Namespace(bar=1, filename=None, foo='3')
1553:~/mypy$ python3 stack56811067.py #args.txt --foo 3
Namespace(bar='2', filename='args.txt', foo='3')
From my testing, using fromfile_prefix_chars means that argparse will not actually pass the argument to your program. Instead, argparse sees the #args.txt, intercepts it, reads from it, and passes the arguments without #args.txt to your program. This is presumably because most people don't really need the filename, just need the arguments within, so argparse saves you the trouble of creating another argument to store something you don't need.
Unfortunately, all of the arguments are stored as local variables in argparse.py, so we cannot access them. I suppose that you could override some of argparse's functions. Keep in mind that this is a horrible, disgusting, hacky solution and I feel that parsing sys.argv is 100% better.
from argparse import ArgumentParser
# Most of the following is copied from argparse.py
def customReadArgs(self, arg_strings):
# expand arguments referencing files
new_arg_strings = []
for arg_string in arg_strings:
# for regular arguments, just add them back into the list
if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
new_arg_strings.append(arg_string)
# replace arguments referencing files with the file content
else:
try:
fn = arg_string[1:]
with open(fn) as args_file:
# What was changed: before was []
arg_strings = [fn]
for arg_line in args_file.read().splitlines():
for arg in self.convert_arg_line_to_args(arg_line):
arg_strings.append(arg)
arg_strings = self._read_args_from_files(arg_strings)
new_arg_strings.extend(arg_strings)
except OSError:
err = _sys.exc_info()[1]
self.error(str(err))
# return the modified argument list
return new_arg_strings
ArgumentParser._read_args_from_files = customReadArgs
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
Just for the record, here's a quick and dirty solution I came up with. It basically consists in creating a copy of parser and set its from_file_prefix_chars attribute to None:
import copy
parser_dupe = copy.copy(parser)
parser_dupe.fromfile_prefix_chars = None
args_raw = parser_dupe.parse_args()
if args_raw.filename:
args.filename = args_raw.filename[1:]
I'm using argparse with several subparsers. I want my program to take options for verbosity anywhere in the args, including the subparser.
from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('--verbose', '-v', action='count')
sub = p.add_subparsers()
a = sub.add_parser('a')
print(p.parse_args())
By default, options for the main parser will throw an error if used for subparsers:
$ python tmp.py -v a
Namespace(verbose=1)
$ python tmp.py a -v
usage: tmp.py [-h] [--verbose] {a} ...
tmp.py: error: unrecognized arguments: -v
I looked into parent parsers, from this answer.
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main = ArgumentParser(parents=[parent])
sub = main.add_subparsers()
a = sub.add_parser('a', parents=[parent])
print(main.parse_args())
For some reason though, none of the shared flags work on the main parser.
$ python tmp2.py a -vvv
Namespace(verbose=3)
$ python tmp2.py -vvv a
Namespace(verbose=None)
Note that the main parser definitely has the appropriate arguments, because when I change it to main = ArgumentParser() I get error: unrecognized arguments: -v. What am I missing here?
First, a couple of general comments.
The main parser handles the input upto the subparser invocation, then the subparser is called and given the remaining argv. When it is done, it's namespace is merged back into the the main namespace.
The parents mechanism copies Actions from the parent by reference. So your main and subparsers share the same verbose Action object. That's been a problem when the subparser tries to set a different default or help. It may not be an issue here, but just keep it in mind.
Even without the parents mechanism, sharing a dest or options flag between main and subparser can be tricky. Should the default of the subparser Action be used? What if both are used? Does the subparser overwrite the main parser's actions?
Originally the main namespace was passed to the subparser, which it modified and returned. This was changed a while back (I can find the bug/issue if needed). Now the subparser starts with a default empty namespace, fills it. And these values are then merged into the main.
So in your linked SO question, be wary of older answers. argparse may have changed since then.
I think what's happening in your case is that the main and subparser verbose are counting separately. And when you get None it's the subparser's default that you see.
The __call__ for _Count_Action is:
def __call__(self, parser, namespace, values, option_string=None):
new_count = _ensure_value(namespace, self.dest, 0) + 1
setattr(namespace, self.dest, new_count)
I suspect that in older argparse when the namespace was shared, the count would have been cumulative, but I can't test it without recreating an older style subparser action class.
https://bugs.python.org/issue15327 - here the original developer suggests giving the two arguments different dest. That records the inputs from both main and sub. Your own code can then merge the results if needed.
https://bugs.python.org/issue27859 argparse - subparsers does not retain namespace. Here I suggest a way of recreating the older style.
https://bugs.python.org/issue9351 argparse set_defaults on subcommands should override top level set_defaults - this is the issue in 2014 that changed the namespace use.
My workaround for this behavior, which is very well described in #hpaulj's answer is to create a second parser that does not have subparsers but only the positional arguments that were first found.
The first parse_args, used with the first parser, will validate the positional arguments and flags, show an error message if needed or show the proper help.
The second parse_args, for the second parser, will correctly fill in the namespace.
Building on your example:
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main1 = ArgumentParser(parents=[parent])
sub = main1.add_subparsers()
# eg: tmp.py -vv a -v
a = sub.add_parser('a', parents=[parent])
a.set_defaults(which='a')
# eg: tmp.py -vv v -v --output toto
b = sub.add_parser('b', parents=[parent])
b.add_argument('--output', type=str)
b.set_defaults(which='b')
args = main1.parse_args()
print(args)
# parse a second time with another parser
main2 = ArgumentParser(parents=[parent])
if args.which == 'a':
main2.add_argument('a')
elif args.which == 'b':
main2.add_argument('b')
main2.add_argument('--output', type=str)
print(main2.parse_args())
Which gives:
$ ./tmp.py -vv a -v
Namespace(verbose=1, which='a')
Namespace(a='a', verbose=3)
$ ./tmp.py -vv b -v --output toto
Namespace(output='toto', verbose=1, which='b')
Namespace(b='b', output='toto', verbose=3)
$ ./tmp.py -vv a --output
usage: tmp.py [-h] [--verbose] {a,b} ...
tmp.py: error: unrecognized arguments: --output
I use this technique with multiple nested subparsers.
I have 2 group which are exclusive, you can define either arguments from group1 or group2 but group2 have to be exclusive within it's arguments too.
parser = argparse.ArgumentParser()
group_exclusive = parser.add_mutually_exclusive_group()
sub_exclusive_1 = group_exclusive.add_argument_group()
sub_exclusive_1.add_argument("-a")
sub_exclusive_1.add_argument("-b")
sub_exclusive_1.add_argument("-c")
sub_exclusive_1.add_argument("-d")
sub_exclusive_2 = group_exclusive.add_mutually_exclusive_group()
sub_exclusive_2.add_argument("-AA")
sub_exclusive_2.add_argument("-BB")
args = parser.parse_args()
The code have to terminate if [-a and -AA or -BB] or [-AA and -BB] have been defined but still have to work with [-a and/or -b],
The problem is that it's not terminating...
I found this thread and edited my code to
subparsers = parser.add_subparsers()
parser_a = subparsers.add_parser('command_1')
parser_a.add_argument("-a")
parser_a.add_argument("-b")
parser_a.add_argument("-c")
parser_a.add_argument("-d")
parser_b = subparsers.add_parser('command_2')
parser_b.add_argument("-AA")
parser_b.add_argument("-BB")
still does not work, traceback: main.py: error: too few arguments
What do i do wrong?
current workaround:
parser = argparse.ArgumentParser()
parser.add_argument("-a")
...
parser.add_argument("-AA")
args = parser.parse_args()
if (args.a or args.b or args.c or args.d) and (args.AA or args.BB) or (args.AA and args.BB):
raise SystemExit()
At the risk of repeating my answer from the earlier question, let's focus on your case
parser = argparse.ArgumentParser()
group_exclusive = parser.add_mutually_exclusive_group()
sub_exclusive_1 = group_exclusive.add_argument_group()
...
sub_exclusive_2 = group_exclusive.add_mutually_exclusive_group()
sub_exclusive_2.add_argument("-AA")
sub_exclusive_2.add_argument("-BB")
Despite similar names (and class nesting), the functionality of argument_groups and mutually_exclusive_groups is quite different. And the former does not nest meaningfully within the second.
An argument group is a tool to organize arguments in the help. It does not enter arguments 'as a group' into another group, and has NO effect on parsing or error checking.
If it did act as you want, what would the usage line look like?
With the subparser formulation the parser responds with:
prog command1 -a -b -c # ok
prog command1 -a -AA # error - not recognize -AA
prog command2 -AA -BB # ok
prog command2 -a -AA # error - -a not recognized
prog -AA # error - too few arg
The subparser mechanism is similar to
parser.add_argument('cmd', choices=['command1','command2']
The 'command1' string tells it - parser the reset of the strings using the '-a -b ...' group of arguments. It has to know which group you expect it to use.
Short of using the bug/issue patch that I worked on a while back, you need to do your own 'mutually-exclusive' testing after parsing. As long as you use the default default None, it is is easy to test whether an argument has been used or now (args.AA is not None).
https://stackoverflow.com/a/30337890/901925 is a recent example of doing post-parsing testing.
I've a Python app like this:
$ app.py -a arg1 -a arg2 -b file1.b -b file2.b
Depending on the option, in app.py, I use two different functions to process the input files. As each option can have several file arguments, I would like something like this:
$ app.py -a arg1 arg2 -b *.b
After searching on the net, I find these three modules: getopt, argparse, optparse
And I wrote a simple one for my app:
optionDict= {'-a':aFunction, '-b':bFunction}
for arg in sys.argv[1:]:
if arg in optionDict: # arg is an option here.
funcName = optionDict[arg]
else: # arg is not an option, then it's a fileName. Deal with this file with the function found according to previous arg.
funcName(arg)
My quesion: Are there any other modules or better ways to do this?
Since 2012 Python has an easy, powerful and very cool module for argument parsing called docopt. It works with Python from 2.5 to 3.3 and needs no installation. Here is the code for your particular case:
'''app.py
Usage:
app.py -a ARG ... -b FILE ...
app.py (-h | -v)
'''
from docopt import docopt
args = docopt(__doc__, version='1.0')
# now do something based on the value of args dict
So this is it: one line of code plus your doc string which is essential. I told you it's cool -- didn't I ;-)
If you found argparse to be too hard to get started with because of the complexity, then you would probably like plac which is available on PyPi. It is a wrapper around argparse which hides the complexity, and it takes care of a lot of boilerplate argument checking for you.
It works for both Python 2 and 3, and has a decorator that allows Python 2 users to get access to a nice Python 3 feature for function parameters. And you can even nest commands in one another so that you could have -arguments (beginning with dash) and .arguments (beginning with dot) on one command line.
Here are the docs http://plac.googlecode.com/hg/doc/plac.html
I think that argparse is the way to go. It does a great job, it is easy to use and in the standard library since 2.7. Take a look a the nargs argument. From the docs
Fixed elements to consume
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs=2)
parser.add_argument('bar', nargs=1)
parser.parse_args('c --foo a b'.split())
Namespace(bar=['c'], foo=['a', 'b'])
One arg will be consumed from the command line if possible, and produced as a single item. If no command-line arg is present, the value from default will be produced
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs='?', const='c', default='d')
parser.add_argument('bar', nargs='?', default='d')
parser.parse_args('XX --foo YY'.split())
Namespace(bar='XX', foo='YY')
parser.parse_args('XX --foo'.split())
Namespace(bar='XX', foo='c')
parser.parse_args(''.split())
Namespace(bar='d', foo='d')
All command-line args present are gathered into a list.
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs='*')
parser.add_argument('--bar', nargs='*')
parser.add_argument('baz', nargs='*')
parser.parse_args('a b --foo x y --bar 1 2'.split())
Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y'])