For parsing boolean command-line options using Python's built-in argparse package, I am aware of this question and its several answers: Parsing boolean values with argparse.
Several of the answers (correctly, IMO) point out that the most common and straightforward idiom for boolean options (from the caller's point of view) is to accept both --foo and --no-foo options, which sets some value in the program to True or False, respectively.
However, all the answers I can find don't actually accomplish the task correctly, it seems to me. They seem to generally fall short on one of the following:
A suitable default can be set (True, False, or None).
Help text given for program.py --help is correct and helpful, including showing what the default is.
Either of (I don't really care which, but both are sometimes desirable):
An argument --foo can be overridden by a later argument --no-foo and vice versa;
--foo and --no-foo are incompatible and mutually exclusive.
What I'm wondering is whether this is even possible at all using argparse.
Here's the closest I've come, based on answers by #mgilson and #fnkr:
def add_bool_arg(parser, name, help_true, help_false, default=None, exclusive=True):
if exclusive:
group = parser.add_mutually_exclusive_group(required=False)
else:
group = parser
group.add_argument('--' + name, dest=name, action='store_true', help=help_true)
group.add_argument('--no-' + name, dest=name, action='store_false', help=help_false)
parser.set_defaults(**{name: default})
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
add_bool_arg(parser, 'foo', "Do foo", "Don't foo", exclusive=True)
add_bool_arg(parser, 'bar', "Do bar", "Don't bar", default=True, exclusive=False)
That does most things well, but the help-text is confusing:
usage: argtest.py [-h] [--foo | --no-foo] [--bar] [--no-bar]
optional arguments:
-h, --help show this help message and exit
--foo Do foo (default: None)
--no-foo Don't foo (default: None)
--bar Do bar (default: True)
--no-bar Don't bar (default: True)
A better help text would be something like this:
usage: argtest.py [-h] [--foo | --no-foo] [--bar] [--no-bar]
optional arguments:
-h, --help show this help message and exit
--foo --no-foo Whether to foo (default: None)
--bar --no-bar Whether to bar (default: True)
But I don't see a way to accomplish that, since "--*" and "--no-*" must always be declared as separate arguments (right?).
In addition to the suggestions at the SO question mentioned above, I've also tried creating a custom action using techniques shown in this other SO question: Python argparse custom actions with additional arguments passed . These fail immediately saying either "error: argument --foo: expected one argument", or (if I set nargs=0) "ValueError: nargs for store actions must be > 0". From poking into the argparse source, it looks like this is because actions other than the pre-defined 'store_const', 'store_true', 'append', etc. must use the _StoreAction class, which requires an argument.
Is there some other way to accomplish this? If someone has a combination of ideas I haven't thought of yet, please let me know!
(BTW- I'm creating this new question, rather than trying to add to the first question above, because the original question above was actually asking for a method to handle --foo TRUE and --foo FALSE arguments, which is different and IMO less commonly seen.)
One of the answers in your linked question, specifically the one by Robert T. McGibbon, includes a code snippet from an enhancement request that was never accepted into the standard argparse. It works fairly well, though, if you discount one annoyance. Here is my reproduction, with a few small modifications, as a stand-alone module with a little bit of pydoc string added, and an example of its usage:
import argparse
import re
class FlagAction(argparse.Action):
"""
GNU style --foo/--no-foo flag action for argparse
(via http://bugs.python.org/issue8538 and
https://stackoverflow.com/a/26618391/1256452).
This provides a GNU style flag action for argparse. Use
as, e.g., parser.add_argument('--foo', action=FlagAction).
The destination will default to 'foo' and the default value
if neither --foo or --no-foo are specified will be None
(so that you can tell if one or the other was given).
"""
def __init__(self, option_strings, dest, default=None,
required=False, help=None, metavar=None,
positive_prefixes=['--'], negative_prefixes=['--no-']):
self.positive_strings = set()
# self.negative_strings = set()
# Order of strings is important: the first one is the only
# one that will be shown in the short usage message! (This
# is an annoying little flaw.)
strings = []
for string in option_strings:
assert re.match(r'--[a-z]+', string, re.IGNORECASE)
suffix = string[2:]
for positive_prefix in positive_prefixes:
s = positive_prefix + suffix
self.positive_strings.add(s)
strings.append(s)
for negative_prefix in negative_prefixes:
s = negative_prefix + suffix
# self.negative_strings.add(s)
strings.append(s)
super(FlagAction, self).__init__(option_strings=strings, dest=dest,
nargs=0, default=default,
required=required, help=help,
metavar=metavar)
def __call__(self, parser, namespace, values, option_string=None):
if option_string in self.positive_strings:
setattr(namespace, self.dest, True)
else:
setattr(namespace, self.dest, False)
if __name__ == '__main__':
p = argparse.ArgumentParser()
p.add_argument('-a', '--arg', help='example')
p.add_argument('--foo', action=FlagAction, help='the boolean thing')
args = p.parse_args()
print(args)
(this code works in Python 2 and 3 both).
Here is the thing in action:
$ python flag_action.py -h
usage: flag_action.py [-h] [-a ARG] [--foo]
optional arguments:
-h, --help show this help message and exit
-a ARG, --arg ARG example
--foo, --no-foo the boolean thing
Note that the initial usage message does not mention the --no-foo option. There is no easy way to correct this other than to use the group method that you dislike.
$ python flag_action.py -a something --foo
Namespace(arg='something', foo=True)
$ python flag_action.py --no-foo
Namespace(arg=None, foo=False)
Related
I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?
I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.
You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.
Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'
For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.
Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)
Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.
This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.
Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)
We're trying to build a wrapper script over a command line tool we're using. We would like to set some tool arguments based on options in our wrapper scripts. We would also like to have the possibility to pass native arguments to the command line tool directly as they are written on the command line.
Here is what we came up with:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-f', '--foo', action='store_true')
parser.add_argument('-b', '--bar', action='store_true')
parser.add_argument('native_arg', nargs='*')
args = parser.parse_args()
print (args)
positional is mandatory. Based on the options -f and -b we would add some extra options to our tool call. Anything that is left afterwards (if anything) should be treated as a native tool argument and given to the tool directly. Calling our script with -h produces the following usage:
usage: test.py [-h] [-f] [-b] positional [native_arg [native_arg ...]]
The trick is that these native arguments are themselves options for the tool and contain leading dashes, for example -native0 and -native1. We already know about the trick with the double dash to stop argparse from looking for more options. The following call:
./test.py pos -- -native0 -native1
produces the expected parsed arguments:
Namespace(bar=False, foo=False, native_arg=['-native0', '-native1'], positional='pos')
Trying to add an option after the first positional argument doesn't work, though. More specifically, the following call:
./test.py pos --foo -- -native0 -native1
produces the following output:
usage: [...shortened...]
test.py: error: unrecognized arguments: -- -native0 -native1
Putting the optional arguments before the positionals:
./test.py --foo pos -- -native0 -native1
seems to work, as the following is printed:
Namespace(bar=False, foo=True, native_arg=['-native0', '-native1'], positional='pos')
Even stranger, changing the value of nargs for native_arg to '+' works in all the above situations (with the caveat, of course, that at least one native_arg is expected).
Are we doing something wrong in our Python code or is this some kind of argparse bug?
argparse does have a hard time when you mix non-required positional arguments with optional arguments (see https://stackoverflow.com/a/47208725/1399279 for details into the bug report). Rather than suggesting a way to solve this issue, I am going to present an alternative approach.
You should check out the parse_known_args method, which was created for the situation you describe (i.e. passing options to a wrapped tool).
In [1]: import argparse
In [2]: parser = argparse.ArgumentParser()
In [3]: parser.add_argument('positional')
In [4]: parser.add_argument('-f', '--foo', action='store_true')
In [5]: parser.add_argument('-b', '--bar', action='store_true')
In [6]: parser.parse_known_args(['pos', '--foo', '-native0', '-native1'])
Out[6]: (Namespace(bar=False, foo=True, positional='pos'), ['-native0', '-native1'])
Unlike parse_args, the output of parse_known_args is a two-element tuple. The first element is the Namespace instance you would expect to get from parse_args, and it contains all the attributes defined by calls to add_argument. The second element is a list of all the arguments not known to the parser.
I personally prefer this method because the user does not need to remember any tricks about how to call your program, or which option order does not result in errors.
This is a known issue (https://bugs.python.org/issue15112, argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional)
The parsing alternates handling positionals and optionals. When dealing with positionals it tries to handle as many as the input strings require. But an ? or * positional is satisfied with [], an empty list of strings. + on the other hand requires at least one string
./test.py pos --foo -- -native0 -native1
The parser gives 'pos' to positional, and [] to native-arg. Then it gives '--foo' to its optional. There aren't anymore positionals left to hand the remaining strings, so it raises the error.
The allocation of input strings is done with a stylized form of regex string matching. Imagine matching a pattern that looks like AA?.
To correct this, parser would have to look ahead, and delay handling native-arg. We've suggested patches but they aren't in production.
#SethMMorton's suggestion of using parse_known_args is a good one.
Earlier parsers (e.g. Optparse) handle all the flagged arguments, but return the rest, the positionals, as a undifferentiated list. It's up to the user to split that list. argparse has added the ability to name and parse positionals, but the algorithm works best with fixed nargs, and gets flaky with too many variable nargs.
I have the following command line tool:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('positional')
parser.add_argument('--optional1')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
The output of python args.py is:
usage: args.py [-h] [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
positional
however I would like to have:
usage: args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
How could I have that reordering?
You would either have to provide your own help formatter, or specify an explicit usage string:
parser = argparse.ArgumentParser(
description="A cool application.",
usage="args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]")
The order in the help message, though, does not affect the order in which you can specify the arguments. argparse processes any defined options left-to-right, then assigns any remaining arguments to the positional parameters from left to right. Options and positional arguments can, for the most part, be mixed.
With respect to each other the order of positionals is fixed - that's why they are called that. But optionals (the flagged arguments) can occur in any order, and usually can be interspersed with the postionals (there are some practical constrains when allowing variable length nargs.)
For the usage line, argparse moves the positionals to the end of the list, but that just a display convention.
There have been SO questions about changing that display order, but I think that is usually not needed. If you must change the display order, using a custom usage parameter is the simplest option. The programming way requires subclassing the help formatter and modifying a key method.
I want to have positional arguments with an optional argument. Smth like my_command foo --version=0.1 baz bar --version=0.2. That should parse into a list of args [foo, bar, baz] with version set for 2 of them.
Without optional argument it's trivial to set nargs=* or nargs=+, but I'm struggling with providing an optional argument for positional ones. Is it even possible with argparse?
Multiple invocation of the same subcommand in a single command line
This tries to parse something like
$ python test.py executeBuild --name foobar1 executeBuild --name foobar2 ....
Both proposed solutions call a parser multiple times. Each call handles a cmd --name value pair. One splits sys.argv before hand, the other collects unparsed strings with a argparse.REMAINDER argument.
Normally optionals can occur in any order. They are identified solely by that - flag value. Positionals have to occur in a particular order, but optionals may occur BETWEEN different positionals. Note also that in the usage display, optionals are listed first, followed by positionals.
usage: PROG [-h] [--version Version] [--other OTHER] FOO BAR BAZ
subparsers are the only way to link a positional argument with one or more optionals. But normally a parser is allowed to have only one subparsers argument.
Without subparsers, append is the only way to collect data from repeated uses of an optional:
parser.add_argument('--version',action='append')
parser.add_argument('foo')
parser.add_argument('bar')
parser.add_argument('baz')
would handle your input string, producing a namespace like:
namespace(version=['0.1','0.2'],foo='foo',bar='bar',baz='baz')
But there's no way of identifying baz as the one that is 'missing' a version value.
Regarding groups - argument groups just affect the help display. They have nothing to do with parsing.
How would you explain to your users (or yourself 6 mths from now) how to use this interface? What would the usage and help look like? It might be simpler to change the design to something that is easier to implement and to explain.
Here's a script which handles your sample input.
import argparse
usage = 'PROG [cmd [--version VERSION]]*'
parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('cmd')
parser.add_argument('-v','--version')
parser.add_argument('rest', nargs=argparse.PARSER)
parser.print_usage()
myargv = 'foo --version=0.1 baz bar --version=0.2'.split()
# myargv = sys.argv[1:] # in production
myargv += ['quit'] # end loop flag
args = argparse.Namespace(rest=myargv)
collect = argparse.Namespace(cmd=[])
while True:
args = parser.parse_args(args.rest)
collect.cmd += [(args.cmd, args.version)]
print(args)
if args.rest[0]=='quit':
break
print collect
It repeatedly parses a positional and optional, collecting the rest in a argparse.PARSER argument. This is like + in that it requires at least one string, but it collects ones that look like optionals as well. I needed to add a quit string so it wouldn't throw an error when there wasn't anything to fill this PARSER argument.
producing:
usage: PROG [cmd [--version VERSION]]*
Namespace(cmd='foo', rest=['baz', 'bar', '--version=0.2', 'quit'], version='0.1')
Namespace(cmd='baz', rest=['bar', '--version=0.2', 'quit'], version=None)
Namespace(cmd='bar', rest=['quit'], version='0.2')
Namespace(cmd=[('foo', '0.1'), ('baz', None), ('bar', '0.2')])
The positional argument that handles subparsers also uses this nargs value. That's how it recognizes and collects a cmd string plus everything else.
So it is possible to parse an argument string such as you want. But I'm not sure the code complexity is worth it. The code is probably fragile as well, tailored to this particular set of arguments, and just a few variants.
Is there a way to make Python's optparse print the default value of an option or flag when showing the help with --help?
Try using the %default string placeholder:
# This example taken from http://docs.python.org/library/optparse.html#generating-help
parser.add_option("-m", "--mode",
default="intermediate",
help="interaction mode: novice, intermediate, "
"or expert [default: %default]")
And if you want to add default values automatically to all options that you have specified, you can do the following:
for option in parser.option_list:
if option.default != ("NO", "DEFAULT"):
option.help += (" " if option.help else "") + "[default: %default]"
And if you need programmatic access to the default values, you can get to them via the defaults attribute of the parser (it's a dict)
The comments to your question already indicate there's another way to parse arguments called argparse. It's been introduced in Python 3.2. It actually deprecates optparse but is used similarly.
argpass comes with different formatting classes and for instance argparse.ArgumentDefaultsHelpFormatter will also print the default values without you having to manipulate the help string manually.
ArgumentParser objects allow the help formatting to be customized by
specifying an alternate formatting class. Currently, there are four
such classes:
class argparse.RawDescriptionHelpFormatter
class argparse.RawTextHelpFormatter
class argparse.ArgumentDefaultsHelpFormatter
class argparse.MetavarTypeHelpFormatter
An example from the python docs:
>>> parser = argparse.ArgumentParser(
... prog='PROG',
... formatter_class=argparse.ArgumentDefaultsHelpFormatter)
>>> parser.add_argument('--foo', type=int, default=42, help='FOO!')
>>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!')
>>> parser.print_help()
usage: PROG [-h] [--foo FOO] [bar [bar ...]]
positional arguments:
bar BAR! (default: [1, 2, 3])
optional arguments:
-h, --help show this help message and exit
--foo FOO FOO! (default: 42)
see argparse formatting classes
Add argparse.ArgumentDefaultsHelpFormatter to your parser
import argparse
parser = argparse.ArgumentParser(
description='Your application description',
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
from documentation:
ArgumentDefaultsHelpFormatter automatically adds information about
default values to each of the argument help messages:
Blockquote