I have read quite a few questions and answers on how to define, parse and run multiple subparsers to run sth like
tool.py func_a -a 12 func_b -b 15 input.txt output.txt
^-- main parser args
^--------------- subparser b
^---------------------------- subparser a
They usually suggest something like:
def func_a(args):
pass
def func_b(args):
pass
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# Define subparsers
subp_a = subparsers.add_parser(func_a.__name__)
subp_a.set_defaults(func=func_a)
subp_a.add_argument('-a')
subp_b = subparsers.add_parser(func_b.__name__)
subp_b.set_defaults(func=func_b)
subp_b.add_argument('-b')
# Define global parameters
parser.add_argument('input', type=argparse.FileType('r'))
parser.add_argument('output', type=argparse.FileType('wb', 0))
# Parse and run through all arguments
rest = sys.argv[1:]
while rest:
args, rest = parser.parse_known_args(rest)
args.func(args)
However this implementation has one problem: The arguments input and output are defined for the main parser and should only be used once. However, each call of parser.parse_known_args(rest) expects the values to be set and consequently removes it from rest.
This means the first call to parse_known_args retrieves the values and each subsequent call fails due to missing arguments.
Is there a solution to overcome this without manually copying the values into the rest list?
Yes, the first parse_known_args consumes the filenames along with the 1st subparser level.
You could define 2 parsers, one with the 'global' positionals, one without. First parse with the 'with' parser, hanging onto its args (for the filenames). Then work your way down the subparser stack with the 'without' parser.
Here's a simpler example:
parser = argparse.ArgumentParser()
parser.add_argument('foo')
subp = parser.add_subparsers(dest='cmd')
cmd1 = subp.add_parser('cmd1')
cmd1.add_argument('-a')
cmd2 = subp.add_parser('cmd2')
# parser.add_argument('bar') # confusing location
parser1 = argparse.ArgumentParser()
subp = parser1.add_subparsers(dest='cmd')
cmd1 = subp.add_parser('cmd1', parents=[cmd1], add_help=False)
cmd2 = subp.add_parser('cmd2')
# Parse and run through all arguments
args, rest = parser.parse_known_args()
print args, rest
while rest:
args, rest = parser1.parse_known_args(rest)
print args, rest
I put the parser foo before subp because it creates less confusion there. A positional at the end (bar) may be confused with arguments for one of the subparsers, or one of the nested subparsers - especially when generating error messages.
To get an idea of how parse_args will allocate argument strings to the various positionals, try a script like:
parser = argparse.ArgumentParser()
parser.add_argument('foo')
parser.add_argument('cmd', nargs=argparse.PARSER, choices=['cmd1','cmd2'])
parser.add_argument('bar')
print parser.parse_args()
To parser, the subparsers argument looks just a like a positional that takes at least one string (kind of like '+'), and the 1st must match choices. It will receive all the 'remaining' strings, consistent with the demands of the other positionals.
Related
I have a script that consumes command line arguments and I would like to implement two argument-passing schemes, namely:
Typing the arguments out at the command line.
Storing the argument list in a file, and passing the name of this file to the program via the command line.
To that end I am passing the argument fromfile_prefix_chars to the ArgumentParser constructor.
script.py
from argparse import ArgumentParser
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
args.txt
--foo
2
--bar
2
Sample use cases
$ python script.py --foo 3
Namespace(bar=1, filename=None, foo='3')
$ python script.py #args.txt --foo 3
Namespace(bar='2', filename=None, foo='3')
I was expecting that args.filename would retain the name of the file, but surprinsingly enough it has the value None instead. I am aware that I could get the file name from sys.argv through a bit of processing. Is there a cleaner way (ideally an argparse-based approach) to elicit the name of the arguments file?
Your script.py, plus the file. I have added the file name to the file itself.
args.txt
args.txt
--foo
2
--bar
2
testing:
1803:~/mypy$ python3 stack56811067.py --foo 3
Namespace(bar=1, filename=None, foo='3')
1553:~/mypy$ python3 stack56811067.py #args.txt --foo 3
Namespace(bar='2', filename='args.txt', foo='3')
From my testing, using fromfile_prefix_chars means that argparse will not actually pass the argument to your program. Instead, argparse sees the #args.txt, intercepts it, reads from it, and passes the arguments without #args.txt to your program. This is presumably because most people don't really need the filename, just need the arguments within, so argparse saves you the trouble of creating another argument to store something you don't need.
Unfortunately, all of the arguments are stored as local variables in argparse.py, so we cannot access them. I suppose that you could override some of argparse's functions. Keep in mind that this is a horrible, disgusting, hacky solution and I feel that parsing sys.argv is 100% better.
from argparse import ArgumentParser
# Most of the following is copied from argparse.py
def customReadArgs(self, arg_strings):
# expand arguments referencing files
new_arg_strings = []
for arg_string in arg_strings:
# for regular arguments, just add them back into the list
if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
new_arg_strings.append(arg_string)
# replace arguments referencing files with the file content
else:
try:
fn = arg_string[1:]
with open(fn) as args_file:
# What was changed: before was []
arg_strings = [fn]
for arg_line in args_file.read().splitlines():
for arg in self.convert_arg_line_to_args(arg_line):
arg_strings.append(arg)
arg_strings = self._read_args_from_files(arg_strings)
new_arg_strings.extend(arg_strings)
except OSError:
err = _sys.exc_info()[1]
self.error(str(err))
# return the modified argument list
return new_arg_strings
ArgumentParser._read_args_from_files = customReadArgs
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
Just for the record, here's a quick and dirty solution I came up with. It basically consists in creating a copy of parser and set its from_file_prefix_chars attribute to None:
import copy
parser_dupe = copy.copy(parser)
parser_dupe.fromfile_prefix_chars = None
args_raw = parser_dupe.parse_args()
if args_raw.filename:
args.filename = args_raw.filename[1:]
I need to implement a command line interface in which the program accepts subcommands.
For example, if the program is called “foo”, the CLI would look like
foo cmd1 <cmd1-options>
foo cmd2
foo cmd3 <cmd3-options>
cmd1 and cmd3 must be used with at least one of their options and the three cmd* arguments are always exclusive.
I am trying to use subparsers in argparse, but with no success for the moment. The problem is with cmd2, that has no arguments:
if I try to add the subparser entry with no arguments, the namespace returned by parse_args will not contain any information telling me that this option was selected (see the example below).
if I try to add cmd2 as an argument to the parser (not the subparser), then argparse will expect that the cmd2 argument will be followed by any of the subparsers arguments.
Is there a simple way to achieve this with argparse? The use case should be quite common…
Here follows what I have attempted so far that is closer to what I need:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='Functions')
parser_1 = subparsers.add_parser('cmd1', help='...')
parser_1.add_argument('cmd1_option1', type=str, help='...')
parser_2 = subparsers.add_parser(cmd2, help='...')
parser_3 = subparsers.add_parser('cmd3', help='...')
parser_3.add_argument('cmd3_options', type=int, help='...')
args = parser.parse_args()
First of all subparsers are never inserted in the namespace. In the example you posted if you try to run the script as:
$python3 test_args.py cmd1 1
Namespace(cmd1_option1='1')
where test_args.py contain the code you provided (with the import argparse at the beginning and print(args) at the end).
Note that there is no mention to cmd1 only to its argument. This is by design.
As pointed out in the comments you can add that information passing the dest argument to the add_subparsers call.
The usual way to handle these circumstances is to use the set_defaults method of the subparsers:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='Functions')
parser_1 = subparsers.add_parser('cmd1', help='...')
parser_1.add_argument('cmd1_option1', type=str, help='...')
parser_1.set_defaults(parser1=True)
parser_2 = subparsers.add_parser('cmd2', help='...')
parser_2.set_defaults(parser2=True)
parser_3 = subparsers.add_parser('cmd3', help='...')
parser_3.add_argument('cmd3_options', type=int, help='...')
parser_3.set_defaults(parser_3=True)
args = parser.parse_args()
print(args)
Which results in:
$python3 test_args.py cmd1 1
Namespace(cmd1_option1='1', parser1=True)
$python3 test_args.py cmd2
Namespace(parser2=True)
In general different subparser will, most of the time, handle the arguments in completely different ways. The usual pattern is to have different functions to run the different commands and use set_defaults to set a func attribute. When you parse the arguments you simply call that callable:
subparsers = parser.add_subparsers()
parser_1 = subparsers.add_parser(...)
parser_1.set_defaults(func=do_command_one)
parser_k = subparsers.add_parser(...)
parser_k.set_defaults(func=do_command_k)
args = parser.parse_args()
if args.func:
args.func(args)
The subparser identity can be added to the main Namespace if the add_subparsers command is given a dest.
From the documentation:
However, if it is necessary to check the name of the subparser that was invoked, the dest keyword argument to the add_subparsers() call will work:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(dest='subparser_name')
>>> subparser1 = subparsers.add_parser('1')
>>> subparser1.add_argument('-x')
>>> subparser2 = subparsers.add_parser('2')
>>> subparser2.add_argument('y')
>>> parser.parse_args(['2', 'frobble'])
Namespace(subparser_name='2', y='frobble')
By default the dest is argparse.SUPPRESS, which keeps subparsers from adding the name to the namespace.
I need to implement a command line interface in which the program accepts subcommands.
For example, if the program is called “foo”, the CLI would look like
foo cmd1 <cmd1-options>
foo cmd2
foo cmd3 <cmd3-options>
cmd1 and cmd3 must be used with at least one of their options and the three cmd* arguments are always exclusive.
I am trying to use subparsers in argparse, but with no success for the moment. The problem is with cmd2, that has no arguments:
if I try to add the subparser entry with no arguments, the namespace returned by parse_args will not contain any information telling me that this option was selected (see the example below).
if I try to add cmd2 as an argument to the parser (not the subparser), then argparse will expect that the cmd2 argument will be followed by any of the subparsers arguments.
Is there a simple way to achieve this with argparse? The use case should be quite common…
Here follows what I have attempted so far that is closer to what I need:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='Functions')
parser_1 = subparsers.add_parser('cmd1', help='...')
parser_1.add_argument('cmd1_option1', type=str, help='...')
parser_2 = subparsers.add_parser(cmd2, help='...')
parser_3 = subparsers.add_parser('cmd3', help='...')
parser_3.add_argument('cmd3_options', type=int, help='...')
args = parser.parse_args()
First of all subparsers are never inserted in the namespace. In the example you posted if you try to run the script as:
$python3 test_args.py cmd1 1
Namespace(cmd1_option1='1')
where test_args.py contain the code you provided (with the import argparse at the beginning and print(args) at the end).
Note that there is no mention to cmd1 only to its argument. This is by design.
As pointed out in the comments you can add that information passing the dest argument to the add_subparsers call.
The usual way to handle these circumstances is to use the set_defaults method of the subparsers:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='Functions')
parser_1 = subparsers.add_parser('cmd1', help='...')
parser_1.add_argument('cmd1_option1', type=str, help='...')
parser_1.set_defaults(parser1=True)
parser_2 = subparsers.add_parser('cmd2', help='...')
parser_2.set_defaults(parser2=True)
parser_3 = subparsers.add_parser('cmd3', help='...')
parser_3.add_argument('cmd3_options', type=int, help='...')
parser_3.set_defaults(parser_3=True)
args = parser.parse_args()
print(args)
Which results in:
$python3 test_args.py cmd1 1
Namespace(cmd1_option1='1', parser1=True)
$python3 test_args.py cmd2
Namespace(parser2=True)
In general different subparser will, most of the time, handle the arguments in completely different ways. The usual pattern is to have different functions to run the different commands and use set_defaults to set a func attribute. When you parse the arguments you simply call that callable:
subparsers = parser.add_subparsers()
parser_1 = subparsers.add_parser(...)
parser_1.set_defaults(func=do_command_one)
parser_k = subparsers.add_parser(...)
parser_k.set_defaults(func=do_command_k)
args = parser.parse_args()
if args.func:
args.func(args)
The subparser identity can be added to the main Namespace if the add_subparsers command is given a dest.
From the documentation:
However, if it is necessary to check the name of the subparser that was invoked, the dest keyword argument to the add_subparsers() call will work:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(dest='subparser_name')
>>> subparser1 = subparsers.add_parser('1')
>>> subparser1.add_argument('-x')
>>> subparser2 = subparsers.add_parser('2')
>>> subparser2.add_argument('y')
>>> parser.parse_args(['2', 'frobble'])
Namespace(subparser_name='2', y='frobble')
By default the dest is argparse.SUPPRESS, which keeps subparsers from adding the name to the namespace.
I am trying to make a Python program that uses the argparse module to parse command-line options.
I want to make an optional argument that can either be named or positional. For example, I want myScript --username=batman to do the same thing as myScript batman. I also want myScript without a username to be valid. Is this possible? If so, how can it be done?
I tried various things similar to the code below without any success.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin")
group.add_argument("user-name", default="admin")
args = parser.parse_args()
EDIT: The above code throws an exception saying ValueError: mutually exclusive arguments must be optional.
I am using Python 2.7.2 on OS X 10.8.4.
EDIT: I tried Gabriel Jacobsohn's suggestion but I couldn't get it working correctly in all cases.
I tried this:
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin", nargs="?")
group.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)
and running myScript batman would print Namespace(user_name='batman'), but myScript -u batman and myScript --user-name=batman would print Namespace(user_name='admin').
I tried changing the name user-name to user_name in the 1st add_argument line and that sometimes resulted in both batman and admin in the namespace or an error, depending on how I ran the program.
I tried changing the name user_name to user-name in the 2nd add_argument line but that would print either Namespace(user-name='batman', user_name='admin') or Namespace(user-name='admin', user_name='batman'), depending on how I ran the program.
The way the ArgumentParser works, it always checks for any trailing positional arguments after it has parsed the optional arguments. So if you have a positional argument with the same name as an optional argument, and it doesn't appear anywhere on the command line, it's guaranteed to override the optional argument (either with its default value or None).
Frankly this seems like a bug to me, at least when used in a mutually exclusive group, since if you had specified the parameter explicitly it would have been an error.
That said, my suggested solution, is to give the positional argument a different name.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument('-u','--username')
group.add_argument('static_username',nargs='?',default='admin')
Then when parsing, you use the optional username if present, otherwise fallback to the positional static_username.
results = parser.parse_args()
username = results.username or results.static_username
I realise this isn't a particularly neat solution, but I don't think any of the answers will be.
Here is a solution that I think does everything you want:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
# Gather all extra args into a list named "args.extra"
parser.add_argument("extra", nargs='*')
args = parser.parse_args()
# Set args.user_name to first extra arg if it is not given and len(args.extra) > 0
if args.user_name == parser.get_default("user_name") and args.extra:
args.user_name = args.extra.pop(0)
print args
If you run myScript -u batman or myScript --user-name=batman, args.user_name is set to 'batman'. If you do myScript batman, args.user_name is again set to 'batman'. And finally, if you just do myScript, args.user_name is set to 'admin'.
Also, as an added bonus, you now have all of the extra arguments that were passed to the script stored in args.extra. Hope this helps.
Try to use the "nargs" parameter of the add_argument method.
This way it works for me.
Now you can add the username twice,
so you have to check it and raise an error, if you want.
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
parser.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)
I have a python program that runs depending on some parameters. Let's say, one of the parameters is C with a default value of 3. So when I run it without any arguments, it does
$ python parsing.py
C=3
When I load a file for my initial data, it can get some parameter values from that file. For example if my file MakeC5 says that the program should run with C=5, I will get
$ python parsing.py --file=MakeC5
C=5
On the other hand, if I specify a different value for C as optional argument, that value is taken, and has priority over the value in the file.
$ python parsing.py --C=4
C=4
$ python parsing.py --file=MakeC5 --C=4
C=4
Up to here, I can just check if the value specified on the command line is different from the default value and otherwise take the one from the file, as with
if parameters.C == parser.get_default('C'):
parameters.C = load_file(parameters.file)["C"]
But this approach does not work if I give the default value for C on the command line as in
$ python parsing.py --file=MakeC5 --C=3
C=3
How do I get that case right? Is there a way that does not need to parse the command line twice as in
parameters = parser.parse_args()
parameters_from_file = load_file(parameters.file)
parser.set_defaults(**parameters_from_file)
parameters = parser.parse_args()
which does not look like “the obvious pythonic way to do this” to me? This is quasi reading a configuration file that can be specified as argument, so I expect there to be a good way.
Assuming you use the argparse module, simply don't set a default argument when you add the argument. Then the attribute will be None if the argument was not present, and you can check for that instead of parsing arguments twice:
parser.add_argument('--C' , action='store', dest='C')
parser.add_argument('--file', action='store', dest='file')
# ...
args = parser.parse_args()
# ...
if args.C is None:
# Argument `--C` not given on command line
if args.file is not None:
# Load the file, set `args.C` if the file contains the `C` option
...
if args.C is None:
# `C` was neither specified on the command line argument given,
# nor in the configuration file, set default value
args.C = 3
print 'C =', args.C
Within argsparse there is fromfile-prefix-chars and it would seem to do some of what you wish if you can live with it's limitations.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', default="ABC")
args = parser.parse_args()
print '#1', args.foo
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', default="ABC")
args = parser.parse_args(['--foo', 'BAR' ])
print '#2', args.foo
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('--foo', default="ABC")
args = parser.parse_args(['--foo', 'BAR', '#myargs' ])
print '#3', args.foo
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('--foo', default="ABC")
args = parser.parse_args(['#myargs', '--foo', 'BAR' ])
print '#4', args.foo
myargs
--foo
FISH
Output
#1 ABC
#2 BAR
#3 FISH
#4 BAR
Notice how the position of #myargs on the command line affects the value of foo, when it's at the front it's valued if overriden by --foo and vice versa.
This would seem to do what you want where:
default-value < config-file < cli-args
Although it relies on you remembering to put the config file in the right order, unless of course you take the args and reorder them so that they first in the args.
Hopefully food for thought.