Develop Reference

argparse with required and optional arguments [duplicate]

I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?

I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.

You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.

Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'

For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.

Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)

Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.

This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.

Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)

parsing mutually exclusive optional and positional arguments followed by pass-thru arguments

I am trying to emulate the Python interpreter command-line behavior, as would be represented by the help text:
command [options] [-m mod | file] [arg] ...
That is:
any number of arbitrary options (which are of the form -[a-zA-Z] that serve as a flag or with a single argument)
one of:
-m mod
file
zero or more arguments which should be available as-is
I have tried using the built-in argparse module, but unsuccessfully.
import argparse
parser = argparse.ArgumentParser()
selector = parser.add_mutually_exclusive_group(required=True)
selector.add_argument('file', nargs='?', help='path to script')
selector.add_argument('-m', help='module name')
parser.add_argument('args', nargs=argparse.REMAINDER)
parser.parse_args(['-m', 'hello', '--', 'arg1'])
Running this yields
usage: test.py [-h] [-m M] [file] ...
test.py: error: argument file: not allowed with argument -m
which makes sense, given that argparse seems to generally disregard the ordering of options - any positional arguments remaining after parsing options fill-in the positional arguments from first to last as specified.
I have tried defining custom argparse.Actions to do the job but it ends up looking pretty hacky, since the Action class corresponding to one of the arguments in the group needs to save an accumulated value for later inclusion in args.
I have also tried pre-processing the input to parser.parse_args, but do not like that approach since information about which options have values (to distinguish an option argument from the file argument) and which options are part of the group of terminal arguments (which should be considered the start of the pass-thru arguments [arg] ...) would be duplicated between the argparse.add_argument... calls and the pre-processing code.
What would be a good approach (other than requiring path to be provided with e.g. -f)?
Additional constraints:
I prefer to use argparse or something with a nice interface that correlates the arguments to help text and doesn't take long to load (argparse imports in 6ms for me)
I only need be compatible with Python 3.6 and above.
It is not ideal, but I am OK requiring users to include -- as the first arg if subsequent arguments (which would be passed through to the module or file) start with a - or may otherwise be mistaken for something in [options].

Even without the mutually exclusive grouping, file and args don't play nicely together:
In [2]: parser = argparse.ArgumentParser()
In [3]: parser.add_argument('-m');
In [4]: parser.add_argument('file', nargs='?');
In [6]: parser.add_argument('args', nargs=argparse.REMAINDER);
OK:
In [7]: parser.parse_args('-m foo a b c '.split())
Out[7]: Namespace(args=['b', 'c'], file='a', m='foo')
'--' just lets us use '-b' as a plain string:
In [8]: parser.parse_args('-m foo a -- -b c '.split())
Out[8]: Namespace(args=['-b', 'c'], file='a', m='foo')
'a' goes to 'file', and rest to 'args' - that's because all 'contiguous' positionals are evaluated together. With remainder, the -m flag is ignored, and treated like a plain string.
In [9]: parser.parse_args('a -m foo -- -b c '.split())
Out[9]: Namespace(args=['-m', 'foo', '--', '-b', 'c'], file='a', m=None)
In [10]: parser.parse_args('a -- -b c '.split())
Out[10]: Namespace(args=['-b', 'c'], file='a', m=None)
Argument allocation occurs even before the Action is called, so custom Action classes don't alter this behavior.
Flagged arguments give you the best control - over order and mutual-exclusivity.

Setting options from environment variables to positional arguments when using argparse

I have a small program that uses argparse and a positional argument. I'm trying to allow that argument to be set by using an environment variable, but are not getting it to work.
I have seen this post: Setting options from environment variables when using argparse which mentions the same problem, but not for positional arguments.
This is the code so far:
import argparse
import os
class EnvDefault(argparse.Action):
def __init__(self, envvar, required=True, default=None, **kwargs):
if not default and envvar:
if envvar in os.environ:
default = os.environ[envvar]
if required and default:
required = False
super(EnvDefault, self).__init__(default=default, required=required, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('testvar', help="Test variable", action=EnvDefault, envvar='TEST_VAR')
parser.add_argument('--othervar', help="Other variable", action='store_true')
args = parser.parse_args()
if not args.testvar: exit(parser.print_usage())
print args.testvar
Which returns this:
$ TEST_VAR="bla" ./test.py
usage: test.py [-h] [--othervar] testvar
test.py: error: too few arguments

You need to make positional argument optional, try nargs='?':
...
parser.add_argument('testvar', help="Test variable", action=EnvDefault,
envvar='TEST_VAR', nargs='?')
...
Note that output changes slightly:
$ python test.py
usage: test.py [-h] [--othervar] [testvar]
Note: There's one side effect - it doesn't return error, even if env variable is not set.

The too few error message indicates that you have slightly older version of Python/argparse. Here's the code that generates the message. It occurs at the end of parsing:
# if we didn't use all the Positional objects, there were too few
# arg strings supplied.
if positionals:
self.error(_('too few arguments'))
# make sure all required actions were present
for action in self._actions:
if action.required:
if action not in seen_actions:
name = _get_action_name(action)
self.error(_('argument %s is required') % name)
positionals starts a list of all the positional arguments, which are removed as they get matched up with argument strings. So it is testing if any were not matched.
Note that the test of the required attribute occurs after this positional test, so changing it, as your Action does, does not help.
The only way to make a positional optional is with nargs - ? or *. An empty list of strings matches those, so such a positional is always consumed.
Check the docs for these. The const parameter might be useful.
The latest version drops that if positionals test, using only the required test to generate a list of arguments that were not used. Your special Action might work in that code.

How do I get back the option string using argparse?

parser = argparse.ArgumentParser()
parser.add_argument("-p", "--pattern", help="Pattern file")
args = parser.parse_args()
Now is it possible to get back the string "--pattern" from args?
I need the string so that I can construct a cmd list to pass to Popen like Popen(['some_other_program', args.pattern.option_string, args.pattern], ...) without repeating it (and having to maintain it in two places) (Popen(['some_other_prog', '--pattern', args.pattern], ...)).
I need to create a wrapper for another program. Some of the args need to be passed to the wrapped program (via Popen) and some are required by the wrapper.
Is there a better method than the following example?
pass_1 = '--to-be-passed'
parser = argparse.ArgumentParser()
parser.add_argument("-p", pass_1, help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
...
process = Popen(['wrapped_program', pass_1, args.pass_1], ...)
...
This method of keeping the args in variables is not very good as:
Maintaining short options along with long options becomes difficult.
Popen if called in another function requires passing these variables(or a dict of them) to the function. This seems redundant as args passed to it should be sufficient.

Add a dest to your add_argument call.
parser.add_argmument("p", "--pattern", dest="pattern", help="your help text")
args = parser.parse_args()
args = vars(args)
The you can reference the pattern with args["pattern"] .

There doesn't seem to be an easy way to get the original option strings from the result of a parser.parse_args(), but you can get them from the parser object. You just need to peek into its __dict__, in order to retrieve the parser settings after it's created. In your case you want the _option_string_actions field. Unfortunately this doesn't seem officially supported, as I couldn't find a ArgumentParser method dedicated to this, so YMMV. On Python 3:
Demo:
parser = argparse.ArgumentParser()
parser.add_argument('--foo', '-f', type=int, default=1000, help='intensity of bar')
parser.add_argument('--bar', '-b', type=str, default='bla', help='whatever')
store_action_dict=vars(parser)['_option_string_actions']
print(store_action_dict.keys()) # dict_keys(['--help', '-b', '-f', '-h', '--foo', '--bar'])

The deleted answers and comments indicate there is some confusion as to what you want. So I'll add to that confusion.
Normally the parser does not record the option string. However it is provided to the Action __call__ method. So a custom Action class could save it. The FooAction custom class example in the argparse docs illustrates this.
If I define this action subclass:
In [324]: class PassThru(argparse._StoreAction):
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, [values, option_string])
In [324]: p.add_argument('-o','--other',action=PassThru)
The option string is recorded along with the value ('-o' or '--other'):
In [322]: p.parse_args('-p test -o teseting'.split())
Out[322]: Namespace(other=['teseting', '-o'], pass_me_on='test')
In [323]: p.parse_args('-p test --other teseting'.split())
Out[323]: Namespace(other=['teseting', '--other'], pass_me_on='test')
Obviously the option_string and value could be recorded in a different order, in a dictionary, as seperate attributes in the Namespace, etc.
There are other ways of passing options to another program, particularly if the wrapping parser does not need to handle them itself.
argparse gets the arguments from sys.argv[1:], and does not change it. So even if your parser uses some of the arguments, you could pass that list on to popen (all or in part).
The argparse docs has an example, under nargs=REMAINDER, of parsing some arguments for itself, and collecting the rest to pass to another program. This is their example:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')
So you could call popen with something like
plist = ['wrapped_program']
plist.extend(args.args)
popen(plist, ...)
Using parse.parse_known_args can also be used to collect unparsed words into an 'extras' list. That section of the docs talks about passing those strings on to another program (just as you are doing). In contrast with the REMAINDER case, the extra stuff does not have to be last.
These work, of course, only if this parser doesn't need --pattern for itself. If it parses it, then it won't appear appear in the REMAINDER or extras. In that case you will have to add it back to the list that you give popen
I would tweak your parser thus:
pass_1 = 'passed' # without the -- or internal -
dpass_` = '--'+pass_
parser = argparse.ArgumentParser()
parser.add_argument("-p", dpass_1, help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
process = Popen(['wrapped_program', dpass_1, getattr(args, pass_1)], ...)
another option:
parser = argparse.ArgumentParser()
pass_action = parser.add_argument("-p", '--pass-me-on', help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
If you print pass_action (in a shell) you'll get something like:
_StoreAction(option_strings=['-p', '--pass-me-on'], dest='pass_me_on', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
So you could pull the --name and dest from that object, thus:
process = Popen(['wrapped_program', pass_action.option_strings[-1], getattr(args, pass_action.dest), ...], ...)
You have to look in sys.argv to see which option_string was used (the long, short or other). The parser does not record that anywhere.
Note '--pass-me-on' produced dest='pass_me_on'. The conversion of - to _ can complicate deriving one string from the other.
If you have a dest string, you have to use getattr to pull it from the args namespace, or use vars(args)[dest] (dictionary access).
Another issue. If --patten has nargs='+', its value will be a list, as opposed to a string. You'd have to careful when merging that into thepopen` argument list.

Argparse: Optional arguments, distinct for different positional arguments

I want to have positional arguments with an optional argument. Smth like my_command foo --version=0.1 baz bar --version=0.2. That should parse into a list of args [foo, bar, baz] with version set for 2 of them.
Without optional argument it's trivial to set nargs=* or nargs=+, but I'm struggling with providing an optional argument for positional ones. Is it even possible with argparse?

Multiple invocation of the same subcommand in a single command line
This tries to parse something like
$ python test.py executeBuild --name foobar1 executeBuild --name foobar2 ....
Both proposed solutions call a parser multiple times. Each call handles a cmd --name value pair. One splits sys.argv before hand, the other collects unparsed strings with a argparse.REMAINDER argument.
Normally optionals can occur in any order. They are identified solely by that - flag value. Positionals have to occur in a particular order, but optionals may occur BETWEEN different positionals. Note also that in the usage display, optionals are listed first, followed by positionals.
usage: PROG [-h] [--version Version] [--other OTHER] FOO BAR BAZ
subparsers are the only way to link a positional argument with one or more optionals. But normally a parser is allowed to have only one subparsers argument.
Without subparsers, append is the only way to collect data from repeated uses of an optional:
parser.add_argument('--version',action='append')
parser.add_argument('foo')
parser.add_argument('bar')
parser.add_argument('baz')
would handle your input string, producing a namespace like:
namespace(version=['0.1','0.2'],foo='foo',bar='bar',baz='baz')
But there's no way of identifying baz as the one that is 'missing' a version value.
Regarding groups - argument groups just affect the help display. They have nothing to do with parsing.
How would you explain to your users (or yourself 6 mths from now) how to use this interface? What would the usage and help look like? It might be simpler to change the design to something that is easier to implement and to explain.
Here's a script which handles your sample input.
import argparse
usage = 'PROG [cmd [--version VERSION]]*'
parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('cmd')
parser.add_argument('-v','--version')
parser.add_argument('rest', nargs=argparse.PARSER)
parser.print_usage()
myargv = 'foo --version=0.1 baz bar --version=0.2'.split()
# myargv = sys.argv[1:] # in production
myargv += ['quit'] # end loop flag
args = argparse.Namespace(rest=myargv)
collect = argparse.Namespace(cmd=[])
while True:
args = parser.parse_args(args.rest)
collect.cmd += [(args.cmd, args.version)]
print(args)
if args.rest[0]=='quit':
break
print collect
It repeatedly parses a positional and optional, collecting the rest in a argparse.PARSER argument. This is like + in that it requires at least one string, but it collects ones that look like optionals as well. I needed to add a quit string so it wouldn't throw an error when there wasn't anything to fill this PARSER argument.
producing:
usage: PROG [cmd [--version VERSION]]*
Namespace(cmd='foo', rest=['baz', 'bar', '--version=0.2', 'quit'], version='0.1')
Namespace(cmd='baz', rest=['bar', '--version=0.2', 'quit'], version=None)
Namespace(cmd='bar', rest=['quit'], version='0.2')
Namespace(cmd=[('foo', '0.1'), ('baz', None), ('bar', '0.2')])
The positional argument that handles subparsers also uses this nargs value. That's how it recognizes and collects a cmd string plus everything else.
So it is possible to parse an argument string such as you want. But I'm not sure the code complexity is worth it. The code is probably fragile as well, tailored to this particular set of arguments, and just a few variants.