I would like use argparse to parse the arguments that it knows and then leave the rest untouched. For example I want to be able to run
performance -o output other_script.py -a opt1 -b opt2
Which uses the -o option and leaves the rest untouched.
The module profiler.py does a similar thing with optparse, but since I'm using argparse I'm doing:
def parse_arguments():
parser = new_argument_parser('show the performance of the given run script')
parser.add_argument('-o', '--output', default='profiled.prof')
return parser.parse_known_args()
def main():
progname = sys.argv[1]
ns, other_args = parse_arguments()
sys.argv[:] = other_args
Which also seems to work, but what happens if also other_script.py also has a -o flag?
Is there in general a better way to solve this problem?
You could also add a positional argument to your parser with nargs=argparse.REMAINDER, to capture the script and its options:
# In script 'performance'...
p = argparse.ArgumentParser()
p.add_argument("-o")
p.add_argument("command", nargs=argparse.REMAINDER)
args = p.parse_args()
print args
Running the above minimal script...
$ performance -o output other_script.py -a opt1 -b opt2
Namespace(command=['performance', '-a', 'opt1', '-b', 'opt2'], o='output')
argparse will stop to parse argument until EOF or --. If you want to have argument without beeing parsed by argparse, you can write::
python [PYTHONOPTS] yourfile.py [YOURFILEOPT] -- [ANYTHINGELSE]
Related
If I have the arguments '-a', '-b', '-c', '-d', with the add_mutually_exclusive_group() function my program will have to use just one of them. Is there a way to combine that, so that the program will accept only either '-a 999 -b 999' or '-c 999 -d 999'?
Edit: adding a simple program for more clarity:
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('-a')
>>> group.add_argument('-b')
>>> group.add_argument('-c')
>>> group.add_argument('-d')
Then only ./app.py -a | ./app.py -b | ./app.py -c | ./app.py -d can be called. Is it possible to have argparse group the exclusion groups, so that only ./app.py -a .. -b .. | ./app.py -c .. -d .. be called?
EDIT: Never mind. Because argparse makes the horrible choice of having to create an option when invoking group.add_argument. That wouldn't be my design choice. If you're desperate for this feature, you can try doing it with ConflictsOptionParser:
# exclusivegroups.py
import conflictsparse
parser = conflictsparse.ConflictsOptionParser()
a_opt = parser.add_option('-a')
b_opt = parser.add_option('-b')
c_opt = parser.add_option('-c')
d_opt = parser.add_option('-d')
import itertools
compatible_opts1 = (a_opt, b_opt)
compatible_opts2 = (c_opt, d_opt)
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
parser.register_conflict(exclusive_grp)
opts, args = parser.parse_args()
print "opts: ", opts
print "args: ", args
Thus when we invoke it, we can see we get the desired effect.
$ python exclusivegroups.py -a 1 -b 2
opts: {'a': '1', 'c': None, 'b': '2', 'd': None}
args: []
$ python exclusivegroups.py -c 3 -d 2
opts: {'a': None, 'c': '3', 'b': None, 'd': '2'}
args: []
$ python exclusivegroups.py -a 1 -b 2 -c 3
Usage: exclusivegroups.py [options]
exclusivegroups.py: error: -b, -c are incompatible options.
The warning message doesn't inform you that both '-a' and '-b' are incompatible with '-c', however a more appropriate error message could be crafted. Older, wrong answer below.
OLDER EDIT: [This edit is wrong, although wouldn't it be just a perfect world if argparse worked this way?] My previous answer actually was incorrect, you should be able to do this with argparse by specifying one group per mutually exclusive options. We can even use itertools to generalize the process. And make it so we don't have to type out all the combinations explicitly:
import itertools
compatible_opts1 = ('-a', '-b')
compatible_opts2 = ('-c', '-d')
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
group = parser.add_mutually_exclusive_group()
group.add_argument(exclusive_grp[0])
group.add_argument(exclusive_grp[1])
Just stumbled on this problem myself. From my reading of the argparse docs, there doesn't seem to be a simple way to achieve that within argparse. I considered using parse_known_args, but that soon amounts to writing a special-purpose version of argparse ;-)
Perhaps a bug report is in order. In the meanwhile, if you're willing to make your user do a tiny bit extra typing, you can fake it with subgroups (like how git and svn's arguments work), e.g.
subparsers = parser.add_subparsers()
p_ab = subparsers.add_parser('ab')
p_ab.add_argument(...)
p_cd = subparsers.add_parser('cd')
p_cd.add_argument(...)
Not ideal, but at least it gives you the good from argparse without too much ugly hackery. I ended up doing away with the switches and just using the subparser operations with required subarguments.
The argparse enhancement request referenced in #hpaulj's comment is still open after more than nine years, so I figured other people might benefit from the workaround I just discovered. Based on this comment in the enhancement request, I found I was able to add an option to two different mutually-exclusive groups using this syntax:
#!/usr/bin/env python
import argparse
import os
import sys
def parse_args():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("-d", "--device", help="Path to UART device", default="./ttyS0")
mutex_group1 = parser.add_mutually_exclusive_group()
mutex_group2 = parser.add_mutually_exclusive_group()
mutex_group1.add_argument(
"-o",
"--output-file",
help="Name of output CSV file",
default="sensor_data_sent.csv",
)
input_file_action = mutex_group1.add_argument(
"-i", "--input-file", type=argparse.FileType("r"), help="Name of input CSV file"
)
# See: https://bugs.python.org/issue10984#msg219660
mutex_group2._group_actions.append(input_file_action)
mutex_group2.add_argument(
"-t",
"--time",
type=int,
help="How long to run, in seconds (-1 = loop forever)",
default=-1,
)
# Add missing ']' to usage message
usage = parser.format_usage()
usage = usage.replace('usage: ', '')
usage = usage.replace(']\n', ']]\n')
parser.usage = usage
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
print("Args parsed successfully...")
sys.exit(0)
This works well enough for my purposes:
$ ./fake_sensor.py -i input.csv -o output.csv
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -o/--output-file: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv -t 30
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -t/--time: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv -t 30
Args parsed successfully...
Accessing private members of argparse is, of course, rather brittle, so I probably wouldn't use this approach in production code. Also, an astute reader may notice that the usage message is misleading, since it implies that -o and -i can be used together when they cannot(!) However, I'm using this script for testing only, so I'm not overly concerned. (Fixing the usage message 'for real' would, I think, require much more time than I can spare for this task, but please comment if you know a clever hack for this.)
Subparsers?
Similar to unhammer's answer, but with more user control. Note: I have not actually tested this method, but it should work in theory and with the capabilities of python.
You can create two parsers, one for each of the two groups, and use conditionals to do the mutually exclusive part. Essentially using argparse for only part of the argument parsing. Using this method, you can go beyond the limitations of unhammer's answer as well.
# Python 3
import argparse
try:
parser = argparse.ArgumentParser()
parser.add_argument('-a')
parser.add_argument('-b')
args = parser.parse_args
except argparse.ArgumentError:
parser = argparse.ArgumentParser()
parser.add_argument('-c')
parser.add_argument('-d')
args = parser.parse_args
How do I define an arbitrary string as an optional argument using argparse?
Example:
[user#host]$ ./script.py FOOBAR -a -b
Running "script.py"...
You set the option "-a"
You set the option "-b"
You passed the string "FOOBAR"
Ideally, I'd like the position of the arguments to not matter. i.e:
./script.py -a FOOBAR -b == ./script.py -a -b FOOBAR == ./script.py FOOBAR -a -b
In BASH, I can accomplish this while using getopts. After handling all desired switches in a case loop, I'd have a line that reads shift $((OPTIND-1)), and from there I can access all remaining arguments using the standard $1, $2, $3, etc...
Does something like that exisit for argparse?
To get exactly what you're looking for, the trick is to use parse_known_args() instead of parse_args():
#!/bin/env python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action="store_true")
parser.add_argument('-b', action="store_true")
opts = parser.parse_known_args()
# Print info about flags
if opts[0].a: print('You set the option "-a"')
if opts[0].b: print('You set the option "-b"')
# Collect remainder (opts[1] is a list (possibly empty) of all remaining args)
if opts[1]: print('You passed the strings %s' % opts[1])
EDIT:
The above code displays the following help information:
./clargs.py -h
usage: clargs_old.py [-h] [-a] [-b]
optional arguments:
-h, --help show this help message and exit
-a
-b
If you want to inform the user about the optional arbitrary argument, the only solution I can think of is to subclass ArgumentParser and write it in yourself.
For example:
#!/bin/env python
import os
import argparse
class MyParser(argparse.ArgumentParser):
def format_help(self):
help = super(MyParser, self).format_help()
helplines = help.splitlines()
helplines[0] += ' [FOO]'
helplines.append(' FOO some description of FOO')
helplines.append('') # Just a trick to force a linesep at the end
return os.linesep.join(helplines)
parser = MyParser()
parser.add_argument('-a', action="store_true")
parser.add_argument('-b', action="store_true")
opts = parser.parse_known_args()
# Print info about flags
if opts[0].a: print('You set the option "-a"')
if opts[0].b: print('You set the option "-b"')
# Collect remainder
if opts[1]: print('You passed the strings %s' % opts[1])
Which displays the following help information:
./clargs.py -h
usage: clargs.py [-h] [-a] [-b] [FOO]
optional arguments:
-h, --help show this help message and exit
-a
-b
FOO some description of FOO
Note the addition of the [FOO] in the "usage" line and the FOO in the help under "optional arguments".
I've written a file crawler and I'm trying to expand it. I want to use argparse to handle settings for the script including passing the starting directory in at the command line.
Example: /var/some/directory/
I have gotten several other arguments to work but I'm unable to pass this directory in correctly. I don't care if it's preceded by a flag or not, (e.g -d /path/to/start/) but I need to make sure that at the very least, this is argument is used as it will be the only mandatory option for the script to run.
Code Sample:
parser = argparse.ArgumentParser(description='py pub crawler...')
parser.add_argument('-v', '--verbose', help='verbose output from crawler', action="store_true")
parser.add_argument('-d', '--dump', help='dumps and replaces existing dictionaries', action="store_true")
parser.add_argument('-f', '--fake', help='crawl only, nothing stored to DB', action="store_true")
args = parser.parse_args()
if args.verbose:
verbose = True
if args.dump:
dump = True
if args.fake:
fake = True
Simply add:
parser.add_argument('directory',help='directory to use',action='store')
before your args = parser.parse_args() line. A simple test from the commandline shows that it does the right thing (printing args at the end of the script):
$ python test.py /foo/bar/baz
Namespace(directory='/foo/bar/baz', dump=False, fake=False, verbose=False)
$ python test.py
usage: test.py [-h] [-v] [-d] [-f] directory
test.py: error: too few arguments
I'm writing a wrapper around the ssh command line client. After the first positional argument that's part of command, all further options should also be treated as positional arguments.
Under optparse, I believe this would be done with disable_interspersed_args.
Presently I have something like this:
parser = argparse.ArgumentParser()
parser.add_argument('--parallel', default=False, action='store_true')
# maybe allow no command? this would ssh interactively into each machine...
parser.add_argument('command', nargs='+')
args = parser.parse_args()
But if options are passed as part of the command (such as my_wrapper ls -l), they're instead interpreted by ArgumentParser as unknown options. error: unrecognized arguments: -l
If I use parse_known_args(), the options may be taken out of order.
p = argparse.ArgumentParser()
p.add_argument('-a', action='store_true')
p.add_argument('command', nargs='+')
print(p.parse_known_args())
$ python3 bah.py -b ls -l -a
(Namespace(a=True, command=['ls']), ['-b', '-l'])
Here you can see that -b's position before ls has been lost, and -a has been parsed out from the command, which is not desired.
How can I:
Prevent arguments from being parsed after a certain point?
Disable parsing of interspersed arguments?
Allow arguments with a prefix to be consumed as positional arguments?
I had the same problem. I found the solution on the argparse bug tracker: http://code.google.com/p/argparse/issues/detail?id=52
The solution is simple: replace nargs='+' (or '*') with nargs=argparse.REMAINDER. This special value is not documented, but it does what you want.
I think your best bet to start solving these issues is to try out -- after all your optional args. -- is a pseudo-arg that tells ArgumentParser that everything after is a positional argument. Docs are here
As for prevent arguments from being parsed after a certain point, you can pass part of argv to parse_args. That combined with some introspection can be used to limit what is parsed.
What #dcolish suggested is the universal approach. Here is a sample implementation which also supports the standard -- separator, but its usage is not required for correct parsing.
Result:
# ./parse-pos.py -h
usage: parse-pos.py [-h] [-qa] [-qb] COMMAND [ARGS...]
# ./parse-pos.py -qa ls -q -h aa /bb
try_argv = ['-qa', 'ls']
cmd_rest_argv = ['-q', '-h', 'aa', '/bb']
parsed_args = Namespace(command='ls', qa=True, qb=False)
The code:
#!/usr/bin/python3
import argparse
import sys
from pprint import pprint
class CustomParserError(Exception):
pass
class CustomArgumentParser(argparse.ArgumentParser):
def error(self, message):
raise CustomParserError(message)
def original_error(self, message):
super().error(message)
def parse_argv():
parser = CustomArgumentParser(description='Example')
parser.add_argument('command', metavar='COMMAND [ARGS...]', help='the command to be executed')
parser.add_argument('-qa', action='store_true') # "ambiguous option" if you specify just "-q"
parser.add_argument('-qb', action='store_true') # "ambiguous option" if you specify just "-q"
def parse_until_positional(parser, _sys_argv = None):
if _sys_argv is None:
_sys_argv = sys.argv[1:] # skip the program name
for i in range(0, len(_sys_argv) + 1):
try_argv = _sys_argv[0:i]
try:
parsed_args = parser.parse_args(try_argv)
except CustomParserError as ex:
if len(try_argv) == len(_sys_argv):
# this is our last try and we still couldn't parse anything
parser.original_error(str(ex)) # sys.exit()
continue
# if we are here, we parsed our known optional & dash-prefixed parameters and the COMMAND
cmd_rest_argv = _sys_argv[i:]
break
return (parsed_args, cmd_rest_argv, try_argv)
(parsed_args, cmd_rest_argv, try_argv) = parse_until_positional(parser)
# debug
pprint(try_argv)
pprint(cmd_rest_argv)
pprint(parsed_args)
return (parsed_args, cmd_rest_argv)
def main():
parse_argv()
main()
Another option is to use parse_known_args, which stops parsing when an unknown argument is encountered.
If I have the arguments '-a', '-b', '-c', '-d', with the add_mutually_exclusive_group() function my program will have to use just one of them. Is there a way to combine that, so that the program will accept only either '-a 999 -b 999' or '-c 999 -d 999'?
Edit: adding a simple program for more clarity:
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('-a')
>>> group.add_argument('-b')
>>> group.add_argument('-c')
>>> group.add_argument('-d')
Then only ./app.py -a | ./app.py -b | ./app.py -c | ./app.py -d can be called. Is it possible to have argparse group the exclusion groups, so that only ./app.py -a .. -b .. | ./app.py -c .. -d .. be called?
EDIT: Never mind. Because argparse makes the horrible choice of having to create an option when invoking group.add_argument. That wouldn't be my design choice. If you're desperate for this feature, you can try doing it with ConflictsOptionParser:
# exclusivegroups.py
import conflictsparse
parser = conflictsparse.ConflictsOptionParser()
a_opt = parser.add_option('-a')
b_opt = parser.add_option('-b')
c_opt = parser.add_option('-c')
d_opt = parser.add_option('-d')
import itertools
compatible_opts1 = (a_opt, b_opt)
compatible_opts2 = (c_opt, d_opt)
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
parser.register_conflict(exclusive_grp)
opts, args = parser.parse_args()
print "opts: ", opts
print "args: ", args
Thus when we invoke it, we can see we get the desired effect.
$ python exclusivegroups.py -a 1 -b 2
opts: {'a': '1', 'c': None, 'b': '2', 'd': None}
args: []
$ python exclusivegroups.py -c 3 -d 2
opts: {'a': None, 'c': '3', 'b': None, 'd': '2'}
args: []
$ python exclusivegroups.py -a 1 -b 2 -c 3
Usage: exclusivegroups.py [options]
exclusivegroups.py: error: -b, -c are incompatible options.
The warning message doesn't inform you that both '-a' and '-b' are incompatible with '-c', however a more appropriate error message could be crafted. Older, wrong answer below.
OLDER EDIT: [This edit is wrong, although wouldn't it be just a perfect world if argparse worked this way?] My previous answer actually was incorrect, you should be able to do this with argparse by specifying one group per mutually exclusive options. We can even use itertools to generalize the process. And make it so we don't have to type out all the combinations explicitly:
import itertools
compatible_opts1 = ('-a', '-b')
compatible_opts2 = ('-c', '-d')
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
group = parser.add_mutually_exclusive_group()
group.add_argument(exclusive_grp[0])
group.add_argument(exclusive_grp[1])
Just stumbled on this problem myself. From my reading of the argparse docs, there doesn't seem to be a simple way to achieve that within argparse. I considered using parse_known_args, but that soon amounts to writing a special-purpose version of argparse ;-)
Perhaps a bug report is in order. In the meanwhile, if you're willing to make your user do a tiny bit extra typing, you can fake it with subgroups (like how git and svn's arguments work), e.g.
subparsers = parser.add_subparsers()
p_ab = subparsers.add_parser('ab')
p_ab.add_argument(...)
p_cd = subparsers.add_parser('cd')
p_cd.add_argument(...)
Not ideal, but at least it gives you the good from argparse without too much ugly hackery. I ended up doing away with the switches and just using the subparser operations with required subarguments.
The argparse enhancement request referenced in #hpaulj's comment is still open after more than nine years, so I figured other people might benefit from the workaround I just discovered. Based on this comment in the enhancement request, I found I was able to add an option to two different mutually-exclusive groups using this syntax:
#!/usr/bin/env python
import argparse
import os
import sys
def parse_args():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("-d", "--device", help="Path to UART device", default="./ttyS0")
mutex_group1 = parser.add_mutually_exclusive_group()
mutex_group2 = parser.add_mutually_exclusive_group()
mutex_group1.add_argument(
"-o",
"--output-file",
help="Name of output CSV file",
default="sensor_data_sent.csv",
)
input_file_action = mutex_group1.add_argument(
"-i", "--input-file", type=argparse.FileType("r"), help="Name of input CSV file"
)
# See: https://bugs.python.org/issue10984#msg219660
mutex_group2._group_actions.append(input_file_action)
mutex_group2.add_argument(
"-t",
"--time",
type=int,
help="How long to run, in seconds (-1 = loop forever)",
default=-1,
)
# Add missing ']' to usage message
usage = parser.format_usage()
usage = usage.replace('usage: ', '')
usage = usage.replace(']\n', ']]\n')
parser.usage = usage
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
print("Args parsed successfully...")
sys.exit(0)
This works well enough for my purposes:
$ ./fake_sensor.py -i input.csv -o output.csv
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -o/--output-file: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv -t 30
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -t/--time: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv -t 30
Args parsed successfully...
Accessing private members of argparse is, of course, rather brittle, so I probably wouldn't use this approach in production code. Also, an astute reader may notice that the usage message is misleading, since it implies that -o and -i can be used together when they cannot(!) However, I'm using this script for testing only, so I'm not overly concerned. (Fixing the usage message 'for real' would, I think, require much more time than I can spare for this task, but please comment if you know a clever hack for this.)
Subparsers?
Similar to unhammer's answer, but with more user control. Note: I have not actually tested this method, but it should work in theory and with the capabilities of python.
You can create two parsers, one for each of the two groups, and use conditionals to do the mutually exclusive part. Essentially using argparse for only part of the argument parsing. Using this method, you can go beyond the limitations of unhammer's answer as well.
# Python 3
import argparse
try:
parser = argparse.ArgumentParser()
parser.add_argument('-a')
parser.add_argument('-b')
args = parser.parse_args
except argparse.ArgumentError:
parser = argparse.ArgumentParser()
parser.add_argument('-c')
parser.add_argument('-d')
args = parser.parse_args