Develop Reference

argparse with required and optional arguments [duplicate]

I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?

I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.

You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.

Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'

For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.

Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)

Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.

This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.

Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)

Python argparse with subparsers and optional positional arguments

I would like to have a program with subparsers that handles specific arguments while also keep some positional and optional arguments to the previous parsers (In fact what I really want is only one option, I mean, a valid subparser OR a valid local argument).
Example of something I wish to have: Program [{sectionName [{a,b}]}] [{c,d}]. Being c/d incompatible if sectionName was provided and viceversa.
However, the best I could achieve is this test.py [-h] {sectionName} ... [{c,d}]. This means, argparse don't allow me to use the positional arguments c or d without specifying a valid sectionName.
Here is the code:
import argparse
mainparser = argparse.ArgumentParser()
# Subparser
subparser = mainparser.add_subparsers(title="section", required=False)
subparser_parser = subparser.add_parser("sectionName")
subparser_parser.add_argument("attribute", choices=['a', 'b'], nargs='?')
# Main parser positional and optional attributes
mainparser.add_argument("attribute", choices=['c', 'd'], nargs='?')
mainparser.parse_args()
I'm getting crazy with this. Any help would be much appreciated!
Edit: I'm using Python 3.8

The subparser object is actually a positional Action, one that takes choices - in this case {'sectionName'}. positinal arguments are filled in the order that they are defined, using the nargs pattern to allocate strings.
Once the main parser gets the 'sectionName' it passes the parsing to subparser_parser. Than handles the rest of the input, such as the {'a','b'} positional. Anything it can't handle is put on the 'unrecognized' list, and control returns main for final processing. main does not do any further argument processing. Thus your attribute argument is ignored.
You could put define a attribute positional before the add_subparsers, but I wouldn't try to make it nargs='?'.
So it's best to define all main arguments before the subparsers, and to use optionals. This will give the cleanest and most reliable parsing.

'argparse' with optional positional arguments that start with dash

We're trying to build a wrapper script over a command line tool we're using. We would like to set some tool arguments based on options in our wrapper scripts. We would also like to have the possibility to pass native arguments to the command line tool directly as they are written on the command line.
Here is what we came up with:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-f', '--foo', action='store_true')
parser.add_argument('-b', '--bar', action='store_true')
parser.add_argument('native_arg', nargs='*')
args = parser.parse_args()
print (args)
positional is mandatory. Based on the options -f and -b we would add some extra options to our tool call. Anything that is left afterwards (if anything) should be treated as a native tool argument and given to the tool directly. Calling our script with -h produces the following usage:
usage: test.py [-h] [-f] [-b] positional [native_arg [native_arg ...]]
The trick is that these native arguments are themselves options for the tool and contain leading dashes, for example -native0 and -native1. We already know about the trick with the double dash to stop argparse from looking for more options. The following call:
./test.py pos -- -native0 -native1
produces the expected parsed arguments:
Namespace(bar=False, foo=False, native_arg=['-native0', '-native1'], positional='pos')
Trying to add an option after the first positional argument doesn't work, though. More specifically, the following call:
./test.py pos --foo -- -native0 -native1
produces the following output:
usage: [...shortened...]
test.py: error: unrecognized arguments: -- -native0 -native1
Putting the optional arguments before the positionals:
./test.py --foo pos -- -native0 -native1
seems to work, as the following is printed:
Namespace(bar=False, foo=True, native_arg=['-native0', '-native1'], positional='pos')
Even stranger, changing the value of nargs for native_arg to '+' works in all the above situations (with the caveat, of course, that at least one native_arg is expected).
Are we doing something wrong in our Python code or is this some kind of argparse bug?

argparse does have a hard time when you mix non-required positional arguments with optional arguments (see https://stackoverflow.com/a/47208725/1399279 for details into the bug report). Rather than suggesting a way to solve this issue, I am going to present an alternative approach.
You should check out the parse_known_args method, which was created for the situation you describe (i.e. passing options to a wrapped tool).
In [1]: import argparse
In [2]: parser = argparse.ArgumentParser()
In [3]: parser.add_argument('positional')
In [4]: parser.add_argument('-f', '--foo', action='store_true')
In [5]: parser.add_argument('-b', '--bar', action='store_true')
In [6]: parser.parse_known_args(['pos', '--foo', '-native0', '-native1'])
Out[6]: (Namespace(bar=False, foo=True, positional='pos'), ['-native0', '-native1'])
Unlike parse_args, the output of parse_known_args is a two-element tuple. The first element is the Namespace instance you would expect to get from parse_args, and it contains all the attributes defined by calls to add_argument. The second element is a list of all the arguments not known to the parser.
I personally prefer this method because the user does not need to remember any tricks about how to call your program, or which option order does not result in errors.

This is a known issue (https://bugs.python.org/issue15112, argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional)
The parsing alternates handling positionals and optionals. When dealing with positionals it tries to handle as many as the input strings require. But an ? or * positional is satisfied with [], an empty list of strings. + on the other hand requires at least one string
./test.py pos --foo -- -native0 -native1
The parser gives 'pos' to positional, and [] to native-arg. Then it gives '--foo' to its optional. There aren't anymore positionals left to hand the remaining strings, so it raises the error.
The allocation of input strings is done with a stylized form of regex string matching. Imagine matching a pattern that looks like AA?.
To correct this, parser would have to look ahead, and delay handling native-arg. We've suggested patches but they aren't in production.
#SethMMorton's suggestion of using parse_known_args is a good one.
Earlier parsers (e.g. Optparse) handle all the flagged arguments, but return the rest, the positionals, as a undifferentiated list. It's up to the user to split that list. argparse has added the ability to name and parse positionals, but the algorithm works best with fixed nargs, and gets flaky with too many variable nargs.

argparse subparser implied by other parameters

The usually way to define a subparser is to do
master_parser = argparse.ArgumentParser()
subparsers = master_parser.add_subparsers()
parser = subparsers.add_parser('sub')
parser.add_argument('--subopt')
and the subparser would be called with
command sub --subopt
I am implementing a package that calls a number of converters. If I use the usual subparser approach, I would have to do
convert ext1_to_ext2 file.ext1 file.ext2 --args
which is both repetitive and error prone because users might call
convert ext1_to_ext3 file.ext1 file.ext2 --args
I would much prefer that the subparser is automatically determined from the master parser so users can use command
convert file.ext1 file.ext2 EXTRA
and argparse would determine subparser ext1_to_ext2 from file.ext1 and file.ext2 and call the subparser ext1_to_ext2 to parse EXTRA. Of course EXTRA here is subparser specific.
I tried to use groups of parameters for each converter (add_argument_group) but the parameters in argument groups cannot overlap and I got a messy list of combined arguments from all parsers, so using subparser seems to be the way to go.
I tried to use parse_known_args with two positional arguments, determine and use the appropriate subparser to parse the remaining args, but it is difficult to provide users a list of converters and their arguments from help message.
Is there a way to do this?

Inferring the subparser to use is tricky, since it requires reimplementing a lot of the logic used by argparse itself while you are examining each of the following arguments.
A simpler approach is to take the subparser command, which subsquently allows you to "typecheck" the following arguments to ensure they use the correct argument. For example
# This allows a file name ending with any of the given extensions,
# or allows "file.1" in place of "file.1.jpg"
def jpeg_file(s):
for ext in ("jpg", "jpeg"):
if s.endswith(ext) or os.path.exists("%s.%s" % (s, ext)):
return s
raise argparse.ArgumentTypeError()
def tiff_file(s):
# similar to jpeg_file
master_parser = argparse.ArgumentParser()
subparsers = master_parser.add_subparsers()
jpg_to_tiff_parser = subparsers.add_parser('sub')
jpg_to_tiff_parser = parse.add_argument('jpg', type=jpg_file)
jpg_to_tiff_parser = parse.add_argument('tiff', type=tiff_file)

This gives you a little more control in my opinion. It's along the way towards what your asking for. Just add file extension checking for your needs.
#prog.py
topParser=argparse.ArgumentParser()
subParsers = topParser.add_subparsers(
title='SubCommands',
description='Use "prog.py <subCommand> (-h | --help)" to learn more about each subcommand',
help='I can do stuff')
subParser1 = subParsers.add_parser('use1', help="Use1 does one thing")
subParser2 = subParsers.add_parser('use2', help='Use2 does another thing')
subParser1.add_argument(
'-f','--first-arg-for-use1',
help="A text file",
required=True
)
subParser1.add_argument(
'-s','--second-arg-for-use1',
help="An encoding",
required=True
)
subParser2.add_argument(
'-f','--first-arg-for-use2',
help="An image format",
required=True
)
args = topParser.parse_args()
print(args)
If nothing else, it shows how to handle the help text for the different layers.

argparse: make one argument default when no arguments are called

What's the best way to set a group argument as the default when no arguments are called.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("--a", action="store_true") #call when no arguments are provided
group.add_argument("--b", action="store_true")
group.add_argument("--c", action="store_true")
Let's call my program argparse_ex.py. I want argparse.py (with no arguments) and argparse.py --a to return the same output.

I would just add a simple test after parsing
if not any([args.a, args.b, args.c]):
args.a=True
This is simpler than any attempt to make parse_args to do this. The parser will parse all arguments independently - and in any order. So you really can't tell until the parsing is all done whether any of the options has been selected or not.