I have a small program that uses argparse and a positional argument. I'm trying to allow that argument to be set by using an environment variable, but are not getting it to work.
I have seen this post: Setting options from environment variables when using argparse which mentions the same problem, but not for positional arguments.
This is the code so far:
import argparse
import os
class EnvDefault(argparse.Action):
def __init__(self, envvar, required=True, default=None, **kwargs):
if not default and envvar:
if envvar in os.environ:
default = os.environ[envvar]
if required and default:
required = False
super(EnvDefault, self).__init__(default=default, required=required, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('testvar', help="Test variable", action=EnvDefault, envvar='TEST_VAR')
parser.add_argument('--othervar', help="Other variable", action='store_true')
args = parser.parse_args()
if not args.testvar: exit(parser.print_usage())
print args.testvar
Which returns this:
$ TEST_VAR="bla" ./test.py
usage: test.py [-h] [--othervar] testvar
test.py: error: too few arguments
You need to make positional argument optional, try nargs='?':
...
parser.add_argument('testvar', help="Test variable", action=EnvDefault,
envvar='TEST_VAR', nargs='?')
...
Note that output changes slightly:
$ python test.py
usage: test.py [-h] [--othervar] [testvar]
Note: There's one side effect - it doesn't return error, even if env variable is not set.
The too few error message indicates that you have slightly older version of Python/argparse. Here's the code that generates the message. It occurs at the end of parsing:
# if we didn't use all the Positional objects, there were too few
# arg strings supplied.
if positionals:
self.error(_('too few arguments'))
# make sure all required actions were present
for action in self._actions:
if action.required:
if action not in seen_actions:
name = _get_action_name(action)
self.error(_('argument %s is required') % name)
positionals starts a list of all the positional arguments, which are removed as they get matched up with argument strings. So it is testing if any were not matched.
Note that the test of the required attribute occurs after this positional test, so changing it, as your Action does, does not help.
The only way to make a positional optional is with nargs - ? or *. An empty list of strings matches those, so such a positional is always consumed.
Check the docs for these. The const parameter might be useful.
The latest version drops that if positionals test, using only the required test to generate a list of arguments that were not used. Your special Action might work in that code.
Related
I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?
I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.
You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.
Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'
For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.
Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)
Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.
This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.
Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)
I have this very simple ArgumentParser instance, with an optional positional argument and an option, which writes a constant to the same destination:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_const', dest='path', const='<all>')
parser.add_argument('path', nargs='?')
# Prints None instead of '<all>'
print(parser.parse_args(['-a']).path)
But no matter what, parsing the command line ['-a'] does not yield a Namespace instance with path set to that constant. Instead, the default from the positional argument is used.
What am I doing wrong?
My use case is that the user should be able to specify a path (actually a list of paths). This list of paths defaults to the current working directory. But instead of using that default, -a can be passed, which should result in some configured root directory to be used. The full code for this part of the argument parser is this:
all_sentinel = object()
parser = argparse.ArgumentParser()
paths_group = parser.add_mutually_exclusive_group()
paths_group.add_argument('-a', action='store_const', dest='paths', const=all_sentinel)
paths_group.add_argument('paths', nargs='*', default=['.'])
A positional with nargs='?' has some special handling of its default (here None).
Normally defaults are assigned to the Namespace at the start of parsing, and overwritten by the actions, such as the optional.
Because an empty list of values satisfies the nargs, that positional is always 'seen'. But rather than assign [] or some other 'blank' to it, the parser assigns the default. So the positional's default overwrites the value set by '-a'.
nargs='*' gets the same kind of special handling.
I suspect that if you had another positional argument before the '-a', that you wouldn't see this effect. The '?*' positional would be processed before the '-a', and not overwrite its value.
Optionals are only processed if the flag occurs. Positionals are always processed, regardless of the nargs. The 'optional' positionals are processed, but with some extra handling of the defaults. But when they are processed relative to flagged arguments can vary.
That's some tricky behavior that I'm aware of simply because I've studied the code in detail, and answered a lot questions here and on the Python bug/issues.
Sharing the dest often does work, but that's more by default than design. It's the result of other design choices. argparse makes no promises in that regard. So if it isn't reliable, don't use it.
I have the following command line tool:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('positional')
parser.add_argument('--optional1')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
The output of python args.py is:
usage: args.py [-h] [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
positional
however I would like to have:
usage: args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
How could I have that reordering?
You would either have to provide your own help formatter, or specify an explicit usage string:
parser = argparse.ArgumentParser(
description="A cool application.",
usage="args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]")
The order in the help message, though, does not affect the order in which you can specify the arguments. argparse processes any defined options left-to-right, then assigns any remaining arguments to the positional parameters from left to right. Options and positional arguments can, for the most part, be mixed.
With respect to each other the order of positionals is fixed - that's why they are called that. But optionals (the flagged arguments) can occur in any order, and usually can be interspersed with the postionals (there are some practical constrains when allowing variable length nargs.)
For the usage line, argparse moves the positionals to the end of the list, but that just a display convention.
There have been SO questions about changing that display order, but I think that is usually not needed. If you must change the display order, using a custom usage parameter is the simplest option. The programming way requires subclassing the help formatter and modifying a key method.
I want to call a Django management command from one of my tests. I'm using django.core.management.call_command for this. And it doesn't work.
I have a command with 4 required arguments. When I call it, it complains all arguments are missing even though I'm passing them:
call_command('my_command', url='12', project='abc', website='zbb', title='12345')
I get the base command error that --url, --project, --website and --title are missing.
I did not specify a different destination for these arguments.
I looked at the call_command source and pinpointed the problem to the following line in call_command:
if command.use_argparse:
# Use the `dest` option name from the parser option
opt_mapping = {sorted(s_opt.option_strings)[0].lstrip('-').replace('-', '_'): s_opt.dest
for s_opt in parser._actions if s_opt.option_strings}
arg_options = {opt_mapping.get(key, key): value for key, value in options.items()}
defaults = parser.parse_args(args=args) ****** THIS *****
defaults = dict(defaults._get_kwargs(), **arg_options)
# Move positional args out of options to mimic legacy optparse
args = defaults.pop('args', ())
args is the positional arguments passed to call_commands, which is empty. I'm only passing named arguments. parser.parse_args complains the required variables are missing.
This is in Django 1.8.3.
Here is my command's add_arguments function (I just removed the help strings for brevity):
def add_arguments(self, parser):
parser.add_argument('--url', action='store', required=True)
parser.add_argument('--project', action='store', required=True)
parser.add_argument('--continue-processing', action='store_true', default=False)
parser.add_argument('--website', action='store', required=True)
parser.add_argument('--title', action='store', required=True)
parser.add_argument('--duplicate', action='store_true',default=False)
Based on that piece of code which you posted, I've concluded in call_command argument is required
that the required named arguments have to be passed in through *args, not just the positionals.
**kwargs bypasses the parser. So it doesn't see anything you defined there. **kwargs may override the *args values, but *args still needs something for each required argument. If you don't want to do that, then turn off the required attribute.
I'm trying to write a Python program that could be extended by third parties. The program will be run from the command line with whatever arguments are supplied.
In order to allow third parties to create their own modules, I've created the following (simplified) base class:
class MyBaseClass(object):
def __init__(self):
self.description = ''
self.command = ''
def get_args(self):
# code that I can't figure out to specify argparse arguments here
# args = []
# arg.append(.....)
return args
Any arguments that they supply via get_args() will be added to a subparser for that particular module. I want them to be able to specify any type of argument.
I'm not sure of the best way to declare and then get the arguments from the subclassed modules into my main program. I successfully find all subclasses of MyBaseClass and loop through them to create the subparsers, but I cannot find a clean way to add the individual arguments to the subparser.
Here is the current code from the main program:
for module in find_modules():
m = module()
subparser_dict[module.__name__] = subparsers.add_parser(m.command, help=m.help)
for arg in m.get_args():
subparser_dict[module.__name__].add_argument(...)
How can I best specify the arguments in the external modules via get_args() or similar and then add them to the subparser? One of my failed attempts looked like the following, which doesn't work because it tries to pass every possible option to add_argument() whether it has a value or is None:
subparser_dict[module.__name__].add_argument(arg['long-arg'],
action=arg['action'],
nargs=arg['nargs'],
const=arg['const'],
default=arg['default'],
type=arg['type'],
choices=arg['choices'],
required=arg['required'],
help=arg['help'],
metavar=arg['metavar'],
dest=arg['dest'],
)
Without trying to fully understand your module structure, I think you want to be able to provide the arguments to a add_argument call as objects that you can import.
You could, for example, provide a list of positional arguments, and dictionary of keyword arguments:
args=['-f','--foo']
kwargs={'type':int, 'nargs':'*', 'help':'this is a help line'}
parser=argparse.ArgumentParser()
parser.add_argument(*args, **kwargs)
parser.print_help()
producing
usage: ipython [-h] [-f [FOO [FOO ...]]]
optional arguments:
-h, --help show this help message and exit
-f [FOO [FOO ...]], --foo [FOO [FOO ...]]
this is a help line
In argparse.py, the add_argument method (of a super class of ArgumentParser), has this general signature
def add_argument(self, *args, **kwargs):
The code of this method manipulates these arguments, adds the args to the kwargs, adds default values, and eventually passes kwargs to the appropriate Action class, returning the new action. (It also 'registers' the action with the parser or subparser). It's the __init__ of the Action subclasses that lists the arguments and their default values.
I would just return an ArgumentParser instance from your get_args method. Then you can create a new ArgumentParser to join all other argument parsers using the parents argument: https://docs.python.org/3/library/argparse.html#parents.