I have quite a big program which has a CLI interaction based on argparse, with several sub parsers. The list of supported choices for the subparsers arguments are determined based on DB queries, parsing different xml files, making different calculations etc, so it is quite IO intensive and time consuming.
The problem is that argparse seems to fetch choices for all sub parser when I run the script, which adds a considerable and annoying startup delay.
Is there a way to make argparse only fetch and validate choices for the currently used sub parser?
One solution could be to move all the validation logic deeper inside the code but that would mean quite a lot of work which I would like to avoid, if possible.
Thank you
To delay the fetching of choices, you could parse the command-line in two stages: In the first stage, you find only the subparser, and in the second stage, the subparser is used to parse the rest of the arguments:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('subparser', choices=['foo','bar'])
def foo_parser():
parser = argparse.ArgumentParser()
parser.add_argument('fooval', choices='123')
return parser
def bar_parser():
parser = argparse.ArgumentParser()
parser.add_argument('barval', choices='ABC')
return parser
dispatch = {'foo':foo_parser, 'bar':bar_parser}
args, unknown = parser.parse_known_args()
args = dispatch[args.subparser]().parse_args(unknown)
print(args)
It could be used like this:
% script.py foo 2
Namespace(fooval='2')
% script.py bar A
Namespace(barval='A')
Note that the top-level help message will be less friendly, since it can only tell you about the subparser choices:
% script.py -h
usage: script.py [-h] {foo,bar}
...
To find information about the choices in each subparser, the user would have to select the subparser and pass the -h to it:
% script.py bar -- -h
usage: script.py [-h] {A,B,C}
All arguments after the -- are considered non-options (to script.py) and are thus parsed by the bar_parser.
Here's a quick and dirty example of a 'lazy' choices. In this case choices are a range of integers. I think a case that requires expensive DB lookups could implemented in a similar fashion.
# argparse with lazy choices
class LazyChoice(object):
# large range
def __init__(self, argmax):
self.argmax=argmax
def __contains__(self, item):
# a 'lazy' test that does not enumerate all choices
return item<=self.argmax
def __iter__(self):
# iterable for display in error message
# use is in:
# tup = value, ', '.join(map(repr, action.choices))
# metavar bypasses this when formatting help/usage
return iter(['integers less than %s'%self.argmax])
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--regular','-r',choices=['one','two'])
larg = parser.add_argument('--lazy','-l', choices=LazyChoice(10))
larg.type = int
print parser.parse_args()
Implementing the testing part (__contains__) is easy. The help/usage can be customized with help and metavar attributes. Customizing the error message is harder. http://bugs.python.org/issue16468 discusses alternatives when choices are not iterable. (also on long list choices: http://bugs.python.org/issue16418)
I've also shown how the type can be changed after the initial setup. That doesn't solve the problem of setting type based on subparser choice. But it isn't hard to write a custom type, one that does some sort of Db lookup. All a type function needs to do is take a string, return the correct converted value, and raise ValueError if there's a problem.
I have solved the issue by creating a simple ArgumentParser subclass:
import argparse
class ArgumentParser(argparse.ArgumentParser):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.lazy_init = None
def parse_known_args(self, args=None, namespace=None):
if self.lazy_init is not None:
self.lazy_init()
self.lazy_init = None
return super().parse_known_args(args, namespace)
Then I can use it as following:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='command', title='commands', parser_class=ArgumentParser)
subparsers.required = True
subparser = subparsers.add_parser(
'do-something', help="do something",
description="Do something great.",
)
def lazy_init():
from my_database import data
subparser.add_argument(
'-o', '--option', choices=data.expensive_fetch(), action='save',
)
subparser.lazy_init = lazy_init
This will really initialize a sub-parser only when parent parser tries to parse arguments for the sub-parser. So if you do program -h it will not initialize the sub-parser, but if you do program do-something -h it will.
This is a script that tests the idea of delaying the creation of a subparser until it is actually needed. In theory it might save start up time, by only creating the subparser that's actually needed.
I use the nargs=argparse.PARSER to replicate the subparser behavior in the main parser. help behavior is similar.
# lazy subparsers test
# lazy behaves much like a regular subparser case, but only creates one subparser
# for N=5 time differences do not rise above the noise
import argparse
def regular(N):
parser = argparse.ArgumentParser()
sp = parser.add_subparsers(dest='cmd')
for i in range(N):
spp = sp.add_parser('cmd%s'%i)
spp.set_defaults(func='cmd%s'%(10*i))
spp.add_argument('-f','--foo')
spp.add_argument('pos', nargs='*')
return parser
def lazy(N):
parser = argparse.ArgumentParser()
sp = parser.add_argument('cmd', nargs=argparse.PARSER, choices=[])
for i in range(N):
sp.choices.append('cmd%s'%i)
return parser
def subpar(cmd):
cmd, argv = cmd[0], cmd[1:]
parser = argparse.ArgumentParser(prog=cmd)
parser.add_argument('-f','--foo')
parser.add_argument('pos', nargs='*')
parser.set_defaults(func=cmd)
args = parser.parse_args(argv)
return args
N = 5
mode = True #False
argv = 'cmd1 -f1 a b c'.split()
if mode:
args = regular(N).parse_args(argv)
print(args)
else:
args = lazy(N).parse_args(argv)
print(args)
if isinstance(args.cmd, list):
sargs = subpar(args.cmd)
print(sargs)
test runs with different values of mode (and N=5)
1004:~/mypy$ time python3 stack44315696.py
Namespace(cmd='cmd1', foo='1', func='cmd10', pos=['a', 'b', 'c'])
real 0m0.052s
user 0m0.044s
sys 0m0.008s
1011:~/mypy$ time python3 stack44315696.py
Namespace(cmd=['cmd1', '-f1', 'a', 'b', 'c'])
Namespace(foo='1', func='cmd1', pos=['a', 'b', 'c'])
real 0m0.051s
user 0m0.048s
sys 0m0.000s
N has to be much larger to start seeing a effect.
Related
For example with GNU ls you can control coloring by using the --color[=WHEN] option. Now in this case the equal sign is crucial since ls have to distinguish between an optional argument to --color and positional arguments (which is the files to list). That is ls --color lists file with colors, which is the same as ls --color=always, but ls --color always will list the file always (and with colors).
Now from what I've seen argparse seem to accept arguments to long options using the --longopt <argument> syntax as well which will lead to not being able to make the argument optional. That is if I try to implement myls with the same behavior as GNU ls (that's just an example) I would run into problems as now myls --color always means the same as myls --color=always (and not as required --color without argument and always as a positional argument).
I know that I can circumvent this by using myls --color -- always, but isn't there a way to make this work without that workaround? That is to tell argparse that the argument to --color has to be supplied with the --color[=WHEN] syntax.
Note that I don't want to rely on the fact that the --color option has finite number of valid arguments. Here's an example what I've tried that didn't work properly:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--foo",
action="store",
nargs="?")
parser.add_argument("frob",
action="store",
nargs=argparse.REMAINDER)
print(parser.parse_args(["alpha", "beta"]))
print(parser.parse_args(["--foo", "alpha", "beta"]))
print(parser.parse_args(["--foo=bar", "alpha", "beta"]))
With the output:
Namespace(foo=None, frob=['alpha', 'beta'])
Namespace(foo='alpha', frob=['beta'])
Namespace(foo='bar', frob=['alpha', 'beta'])
note the second where alpha was interpreted as argument to --foo. I wanted:
Namespace(foo=None, frob=['alpha', 'beta'])
Namespace(foo=None, frob=['alpha', 'beta'])
Namespace(foo='bar', frob=['alpha', 'beta'])
You've probably already tried the ? optional followed by required positional:
p=argparse.ArgumentParser()
p.add_argument('--foo', nargs='?',default='one', const='two')
p.add_argument('bar')
which fails with
In [7]: p.parse_args('--foo 1'.split())
usage: ipython3 [-h] [--foo [FOO]] bar
ipython3: error: the following arguments are required: bar
--foo consumes the 1, leaving nothing for bar.
http://bugs.python.org/issue9338 discusses this issue. The nargs='?' is greedy, consuming an argument, even though the following positional requires one. But the suggested patch is complicated, so I can't quickly apply it to a parser and test your case.
The idea of defining an Action that would work with --foo==value, but not consume value in --foo value, is interesting, but I have no idea of what it would take to implement. Certainly it doesn't work with the current parser. I'd have to review how it handles that explicit =.
============================
By changing a deeply nested function in parse_args,
def consume_optional(....):
....
# error if a double-dash option did not use the
# explicit argument
else:
msg = _('ignored explicit argument %r')
#raise ArgumentError(action, msg % explicit_arg)
# change for stack40989413
print('Warn ',msg)
stop = start_index + 1
args = [explicit_arg]
action.nargs=None
action_tuples.append((action, args, option_string))
break
and adding a custom Action class:
class MyAction(myparse._StoreConstAction):
# requies change in consume_optional
def __call__(self, parser, namespace, values, option_string=None):
if values:
setattr(namespace, self.dest, values)
else:
setattr(namespace, self.dest, self.const)
I can get the desired behavior from:
p = myparse.ArgumentParser()
p.add_argument('--foo', action=MyAction, const='C', default='D')
p.add_argument('bar')
Basically I'm modifying store_const to save the =explicit_arg if present.
I don't plan on proposing this as a formal patch, but I'd welcome feedback if it is useful. Use at your own risk. :)
Apparently this is not possible. This behaviour is supported by GNU getopt() (man getopt, man 3 getopt). man getopt says:
If the [long] option has an optional argument, it must be written directly after the long option name, separated by '=', if present
The Python getopt module, however, is clear that it doesn't support this:
Optional arguments [in long options] are not supported.
For argparse I don't find any specific reference in the manual, but I would be surprised if it supported it. In fact, I'm surprised GNU getopt supports it and that ls works the way you described. User interfaces should be simple, and this behaviour is far from simple.
Here's a workaround:
#!/usr/bin/python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("files", nargs="*", help="List of files", type=str)
parser.add_argument('--color', dest='color', action='store_true')
parser.add_argument('--color=ALWAYS', dest='color_always', action='store_true')
args = parser.parse_args()
print args
Results:
[~]$ ./test.py xyz --color
Namespace(color=True, color_always=False, files=['xyz'])
[~]$ ./test.py xyz --color=ALWAYS
Namespace(color=False, color_always=True, files=['xyz'])
Problemo solved!
It's a bit (a lot) hacky, but here you go.
The solution revolves around a class that inherits _StoreConstAction and tweaks it a little, but mainly tricks the help formatter when it tries to get its attributes.
I tested this in python3 under windows and linux.
import argparse
import inspect
class GnuStyleLongOption(argparse._StoreConstAction):
def __init__(self, **kw):
self._real_option_strings = kw['option_strings']
opts = []
for option_string in self._real_option_strings:
opts.append(option_string)
for choice in kw['choices']:
opts.append(f'{option_string}={choice}')
kw['option_strings'] = opts
self.choices = kw.pop('choices')
help_choices = [f"'{choice}'" for choice in self.choices]
kw['help'] += f"; {kw['metavar']} is {', or '.join([', '.join(help_choices[:-1]), help_choices[-1]])}"
super(GnuStyleLongOption, self).__init__(**kw)
def __getattribute__(self, attr):
caller_is_argparse_help = False
for frame in inspect.stack():
if frame.function == 'format_help' and frame.filename.endswith('argparse.py'):
caller_is_argparse_help = True
break
if caller_is_argparse_help:
if attr == 'option_strings':
return [f'{i}[=WHEN]' for i in self._real_option_strings]
if attr == 'nargs':
return 0
if attr == 'metavar':
return None
return super(GnuStyleLongOption, self).__getattribute__(attr)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, self.const if '=' not in option_string else option_string[option_string.find('=') + 1:])
p = argparse.ArgumentParser()
p.add_argument('--color', '--colour', action=GnuStyleLongOption, choices=['always', 'never', 'auto'], const='always', default='auto', help='use markers to highlight whatever we want', metavar='WHEN')
p.add_argument('filenames', metavar='filename', nargs='*', help='file to process')
args = p.parse_args()
print(f'color = {args.color}, filenames = {args.filenames}')
Results:
~ $ ./gnu_argparse.py --help
usage: gnu_argparse.py [-h] [--color[=WHEN]] [filename [filename ...]]
positional arguments:
filename file to process
optional arguments:
-h, --help show this help message and exit
--color[=WHEN], --colour[=WHEN]
use markers to highlight whatever we want; WHEN is
'always', 'never', or 'auto'
~ $ ./gnu_argparse.py
color = auto, filenames = []
~ $ ./gnu_argparse.py file
color = auto, filenames = ['file']
~ $ ./gnu_argparse.py --color file
color = always, filenames = ['file']
~ $ ./gnu_argparse.py --color never file
color = always, filenames = ['never', 'file']
~ $ ./gnu_argparse.py --color=never file
color = never, filenames = ['file']
~ $ ./gnu_argparse.py --colour=always file
color = always, filenames = ['file']
maybe nargs would be help.
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--color', nargs='?', const='c', default='d')
>>> parser.parse_args(['XX', '--color', 'always'])
Namespace(bar='XX', color='always')
>>> parser.parse_args(['XX', '--color'])
Namespace(bar='XX', color='c')
>>> parser.parse_args([])
Namespace(bar='d', color='d')
with nargs, you get different args and you will know what the input type is.
by the way, I think --color option could use action='store_true'.
parser.add_argument('--color', action='store_true')
Is there any intuitive alternative for argparse/optparse for subcommands? They both are bad - it is either insane config or insane output.
Real world example (stolen, not wanted):
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(title='subcommands',
... description='valid subcommands',
... help='additional help')
>>> subparsers.add_parser('foo')
>>> subparsers.add_parser('bar')
>>> parser.parse_args(['-h'])
usage: [-h] {foo,bar} ...
optional arguments:
-h, --help show this help message and exit
subcommands:
valid subcommands
{foo,bar} additional help
Wanted:
>>> parser = cmdline.Parser(
... tplheader='Usage: tool [command] [options]',
... tplcommandhead='Available commands:',
... tplfooter='Use \"tool help\" to get full list of supported commands.')
>>> parser.add('foo', help='foo.')
>>> parser.add('bar', help='bar.')
>>> parser.parse(['-h'])
Usage: tool [command] [options]
Available commands:
foo foo.
bar bar.
Use "tool help" to get full list of supported commands.
UPDATE: I would accept the answer that provides command validation and parsing example that gives the help message exactly as the last snippet.
You can get pretty close to your requested output just by changing your argparse code a little bit:
Set the usage text by specifying the usage parameter to ArgumentParser.
Omit the description and help arguments to add_subparsers.
Change the title parameter to Available subcommands.
Use the metavar parameter to override the unsightly {foo,bar} text.
Use the help arguments available in add_parser.
Here's the finished product:
import argparse
parser = argparse.ArgumentParser(usage='tool [command] [options]')
subparsers = parser.add_subparsers(title='Available commands', metavar='')
subparsers.add_parser('foo', help='foo.')
subparsers.add_parser('bar', help='bar.')
parser.parse_args(['-h'])
That code prints this:
usage: tool [command] [options]
optional arguments:
-h, --help show this help message and exit
Available commands:
foo foo.
bar bar.
Sounds like you are looking for argh.
Here's a snippet from the presentation on the home page.
A potentially modular application with multiple commands:
import argh
# declaring:
def echo(text):
"Returns given word as is."
return text
def greet(name, greeting='Hello'):
"Greets the user with given name. The greeting is customizable."
return greeting + ', ' + name
# assembling:
parser = argh.ArghParser()
parser.add_commands([echo, greet])
# dispatching:
if __name__ == '__main__':
parser.dispatch()
Of course it works:
$ ./app.py greet Andy
Hello, Andy
$ ./app.py greet Andy -g Arrrgh
Arrrgh, Andy
The help message on the site is slightly abridged. Here is what it actually outputs for me (argh 0.26.1).
$ ./app.py --help
usage: app.py [-h] {greet,echo} ...
positional arguments:
{greet,echo}
echo Returns given word as is.
greet Greets the user with given name. The greeting is customizable.
optional arguments:
-h, --help show this help message and exit
Does this win the prize? :)
Custom parameters
Rob Kennedy has a better customization.
In [158]: parser=argparse.ArgumentParser(usage='tool [command] [options]',
description= "Available commands:\n\n foo foo.\n bar bar.\n",
epilog= 'Use "tool help" to get full list of supported commands',
formatter_class=argparse.RawDescriptionHelpFormatter, add_help=False)
In [159]: parser.print_help()
usage: tool [command] [options]
Available commands:
foo foo.
bar bar.
Use "tool help" to get full list of supported commands
What I've done is customize the help with available parameters.
Alternative API and/or parser?
But your other lines, the parse.add() ones suggest you don't like the argparse method of defining 'commands'. You could add some methods to your parser that use this more compact syntax, but still end up calling the existing subparser mechanism.
But maybe you want to replace the whole parsing scheme with your own. One, for example, that expects the first argument to be a 'command'. What about other 'positionals'? Who or what handles the 'options'?
Do you realize that the argparse subparser scheme is built on top of the more basic optionals and positionals parsing scheme. The parser.add_subparsers command is a specialized form of add_argument. The subparsers object is a positional argument, with a special Action class. {foo,bar} is actually a list of the choices values that you defined for this argument (names or aliases of the subcommands). The subcommands themselves are parsers.
Custom front end command parser
If the sys.argv[1] item will always be a command name, you could set up something like this:
if sys.argv[1:]:
cmd = sys.argv[1]
rest = sys.argv[2:]
parser = parser_dict.get(cmd, None)
if parser:
args = parser.parse_args(rest)
else:
print_default_help()
Where parser_dict is a dictionary matching cmd strings to defined parsers. In effect this is just a front end that captures the first argument string, and dispatches the handling of the rest to other defined parsers. They could be a mix of argparse, optparse, and custom parsers. This front end does not have to be fancy if all it handles is the first 'command' string.
print_default_help would be little more than a pretty print of the parser_dict.
On further thought, I realized that the sp.choices attribute of an argparse subparsers object is just such a dictionary - with command strings as keys, and parsers as values.
Custom format_help methods
Here are a couple of custom help formatters.
A simple one that only gets the prog and _choices_actions from the parser. subparsers._choices_actions is a list of objects that contain help and aliases information for the individual sub parsers.
def simple_help(parser, subparsers):
# format a help message with just the subparser choices
usage = "Usage: %s command [options]"%parser.prog
desc = "Available commands:\n"
epilog = '\nUse "%s help" to get full list of supported commands.'%parser.prog
choices = fmt_choices(subparsers._choices_actions)
astr = [usage]
astr.append(desc)
astr.extend(choices)
astr.append(epilog)
return '\n'.join(astr)
def fmt_choices(choices):
# format names and help in 2 columns
x = max(len(k.metavar) for k in choices)
fmt = ' {:<%s} {}'%x
astr = []
for k in choices:
# k.metavar lists aliases as well
astr.append(fmt.format(k.dest, k.help))
return astr
This one is modeled on parser.format_help, and makes uses of the Formatter and all of its wrapping and spacing information. I wrote it to use non-default parameters where possible. It is hard, though, to suppress blank lines.
def special_help(parser, subparsers=None, usage=None, epilog=None):
# format help message using a Formatter
# modeled on parser.format_help
# uses nondefault parameters where possible
if usage is None:
usage = "%(prog)s command [options]"
if epilog is None:
epilog = "Use '%(prog)s help' for command list"
if subparsers is None:
# find the subparsers action in the parser
for action in parser._subparsers._group_actions:
if hasattr(action, '_get_subactions'):
subparsers = action
break
# if none found, subparsers is still None?
if parser._subparsers != parser._positionals:
title = parser._subparsers.title
desc = parser._subparsers.description
else:
title = "Available commands"
desc = None
if subparsers.metavar is None:
subparsers.metavar = '_________'
# restore to None at end?
formatter = parser._get_formatter()
if parser.usage is None:
formatter.add_usage(usage, [], [])
else:
formatter.add_usage(parser.usage,
parser._actions, parser._mutually_exclusive_groups)
# can I get rid of blank line here?
formatter.start_section(title)
formatter.add_text(desc)
formatter.add_arguments([subparsers])
formatter.end_section()
formatter.add_text(epilog)
return formatter.format_help()
These could be invoked in different ways. Either could replace the parser's format_help method, and thus be produced by the -h option, as well as with parser.print_help().
Or you could include a help subcommand. This would fit with the epilog message. -h would still produce the full, ugly help.
sp3 = sp.add_parser('help') # help='optional help message'
and test args:
if args.cmd in ['help']:
print(simple_help(parser, sp))
# print(special_help(parser))
Another option is to check sys.argv before parser.parser_args, and call the help function if that list isn't long enough, or includes a help string. This is roughly what Ipython does to bypass the regular argparse help.
I'm not sure that I understand what's wrong with what you describe.
I use something slightly different though:
parser = argparse.ArgumentParser(description='My description')
parser.add_argument('-i', '--input', type=str, required=True, help='Inputfile')
parser.add_argument('-o', '--output', type=str, required=False, help='Output file')
args = parser.parse_args()
input_filename = args.input
if not args.output:
output_filename = input_filename
else:
output_filename = args.output
You should take a look at Click. From the documentation, Click...
is lazily composable without restrictions
fully follows the Unix command line conventions
supports loading values from environment variables out of the box
supports for prompting of custom values
is fully nestable and composable
works the same in Python 2 and 3
supports file handling out of the box
comes with useful common helpers (getting terminal dimensions, ANSI colors, fetching direct keyboard input, screen clearing, finding config paths, launching apps and editors, etc.)
Arguments and options are pretty intuitive to create with decorators. You can create subcommands by creating groups as shown below.
import click
#click.command()
#click.option('--count', default=1, help='number of greetings')
#click.argument('name')
def hello(count, name):
for i in range(count):
print(f"{i}. Hello {name}")
#click.group()
def cli():
pass
#cli.command()
def initdb():
click.echo('Initialized the database')
#cli.command()
def dropdb():
click.echo('Dropped the database')
if __name__ == "__main__":
cli()
The output from this code is:
$ python click-example.py --help
Usage: click-example.py [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
dropdb
initdb
I'm new to python and currently playing with it.
I have a script which does some API Calls to an appliance. I would like to extend the functionality and call different functions based on the arguments given when calling the script.
Currently I have the following:
parser = argparse.ArgumentParser()
parser.add_argument("--showtop20", help="list top 20 by app",
action="store_true")
parser.add_argument("--listapps", help="list all available apps",
action="store_true")
args = parser.parse_args()
I also have a
def showtop20():
.....
and
def listapps():
....
How can I call the function (and only this) based on the argument given?
I don't want to run
if args.showtop20:
#code here
if args.listapps:
#code here
as I want to move the different functions to a module later on keeping the main executable file clean and tidy.
Since it seems like you want to run one, and only one, function depending on the arguments given, I would suggest you use a mandatory positional argument ./prog command, instead of optional arguments (./prog --command1 or ./prog --command2).
so, something like this should do it:
FUNCTION_MAP = {'top20' : my_top20_func,
'listapps' : my_listapps_func }
parser.add_argument('command', choices=FUNCTION_MAP.keys())
args = parser.parse_args()
func = FUNCTION_MAP[args.command]
func()
At least from what you have described, --showtop20 and --listapps sound more like sub-commands than options. Assuming this is the case, we can use subparsers to achieve your desired result. Here is a proof of concept:
import argparse
import sys
def showtop20():
print('running showtop20')
def listapps():
print('running listapps')
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# Create a showtop20 subcommand
parser_showtop20 = subparsers.add_parser('showtop20', help='list top 20 by app')
parser_showtop20.set_defaults(func=showtop20)
# Create a listapps subcommand
parser_listapps = subparsers.add_parser('listapps', help='list all available apps')
parser_listapps.set_defaults(func=listapps)
# Print usage message if no args are supplied.
# NOTE: Python 2 will error 'too few arguments' if no subcommand is supplied.
# No such error occurs in Python 3, which makes it feasible to check
# whether a subcommand was provided (displaying a help message if not).
# argparse internals vary significantly over the major versions, so it's
# much easier to just override the args passed to it.
if len(sys.argv) <= 1:
sys.argv.append('--help')
options = parser.parse_args()
# Run the appropriate function (in this case showtop20 or listapps)
options.func()
# If you add command-line options, consider passing them to the function,
# e.g. `options.func(options)`
There are lots of ways of skinning this cat. Here's one using action='store_const' (inspired by the documented subparser example):
p=argparse.ArgumentParser()
p.add_argument('--cmd1', action='store_const', const=lambda:'cmd1', dest='cmd')
p.add_argument('--cmd2', action='store_const', const=lambda:'cmd2', dest='cmd')
args = p.parse_args(['--cmd1'])
# Out[21]: Namespace(cmd=<function <lambda> at 0x9abf994>)
p.parse_args(['--cmd2']).cmd()
# Out[19]: 'cmd2'
p.parse_args(['--cmd1']).cmd()
# Out[20]: 'cmd1'
With a shared dest, each action puts its function (const) in the same Namespace attribute. The function is invoked by args.cmd().
And as in the documented subparsers example, those functions could be written so as to use other values from Namespace.
args = parse_args()
args.cmd(args)
For sake of comparison, here's the equivalent subparsers case:
p = argparse.ArgumentParser()
sp = p.add_subparsers(dest='cmdstr')
sp1 = sp.add_parser('cmd1')
sp1.set_defaults(cmd=lambda:'cmd1')
sp2 = sp.add_parser('cmd2')
sp2.set_defaults(cmd=lambda:'cmd2')
p.parse_args(['cmd1']).cmd()
# Out[25]: 'cmd1'
As illustrated in the documentation, subparsers lets you define different parameter arguments for each of the commands.
And of course all of these add argument or parser statements could be created in a loop over some list or dictionary that pairs a key with a function.
Another important consideration - what kind of usage and help do you want? The different approaches generate very different help messages.
If your functions are "simple enough" take adventage of type parameter https://docs.python.org/2.7/library/argparse.html#type
type= can take any callable that takes a single string argument and
returns the converted value:
In your example (even if you don't need a converted value):
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
This simple script:
import argparse
def showtop20(dummy):
print "{0}\n".format(dummy) * 5
parser = argparse.ArgumentParser()
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
args = parser.parse_args()
Will give:
# ./test.py --listapps test
test
test
test
test
test
test
Instead of using your code as your_script --showtop20, make it into a sub-command your_script showtop20 and use the click library instead of argparse. You define functions that are the name of your subcommand and use decorators to specify the arguments:
import click
#click.group()
#click.option('--debug/--no-debug', default=False)
def cli(debug):
print(f'Debug mode is {"on" if debug else "off"}')
#cli.command() # #cli, not #click!
def showtop20():
# ...
#cli.command()
def listapps():
# ...
See https://click.palletsprojects.com/en/master/commands/
# based on parser input to invoke either regression/classification plus other params
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--path", type=str)
parser.add_argument("--target", type=str)
parser.add_argument("--type", type=str)
parser.add_argument("--deviceType", type=str)
args = parser.parse_args()
df = pd.read_csv(args.path)
df = df.loc[:, ~df.columns.str.contains('^Unnamed')]
if args.type == "classification":
classify = AutoML(df, args.target, args.type, args.deviceType)
classify.class_dist()
classify.classification()
elif args.type == "regression":
reg = AutoML(df, args.target, args.type, args.deviceType)
reg.regression()
else:
ValueError("Invalid argument passed")
# Values passed as : python app.py --path C:\Users\Abhishek\Downloads\adult.csv --target income --type classification --deviceType GPU
You can evaluate using evalwhether your argument value is callable:
import argparse
def list_showtop20():
print("Calling from showtop20")
def list_apps():
print("Calling from listapps")
my_funcs = [x for x in dir() if x.startswith('list_')]
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--function", required=True,
choices=my_funcs,
help="function to call", metavar="")
args = parser.parse_args()
eval(args.function)()
I'm writing a wrapper around the ssh command line client. After the first positional argument that's part of command, all further options should also be treated as positional arguments.
Under optparse, I believe this would be done with disable_interspersed_args.
Presently I have something like this:
parser = argparse.ArgumentParser()
parser.add_argument('--parallel', default=False, action='store_true')
# maybe allow no command? this would ssh interactively into each machine...
parser.add_argument('command', nargs='+')
args = parser.parse_args()
But if options are passed as part of the command (such as my_wrapper ls -l), they're instead interpreted by ArgumentParser as unknown options. error: unrecognized arguments: -l
If I use parse_known_args(), the options may be taken out of order.
p = argparse.ArgumentParser()
p.add_argument('-a', action='store_true')
p.add_argument('command', nargs='+')
print(p.parse_known_args())
$ python3 bah.py -b ls -l -a
(Namespace(a=True, command=['ls']), ['-b', '-l'])
Here you can see that -b's position before ls has been lost, and -a has been parsed out from the command, which is not desired.
How can I:
Prevent arguments from being parsed after a certain point?
Disable parsing of interspersed arguments?
Allow arguments with a prefix to be consumed as positional arguments?
I had the same problem. I found the solution on the argparse bug tracker: http://code.google.com/p/argparse/issues/detail?id=52
The solution is simple: replace nargs='+' (or '*') with nargs=argparse.REMAINDER. This special value is not documented, but it does what you want.
I think your best bet to start solving these issues is to try out -- after all your optional args. -- is a pseudo-arg that tells ArgumentParser that everything after is a positional argument. Docs are here
As for prevent arguments from being parsed after a certain point, you can pass part of argv to parse_args. That combined with some introspection can be used to limit what is parsed.
What #dcolish suggested is the universal approach. Here is a sample implementation which also supports the standard -- separator, but its usage is not required for correct parsing.
Result:
# ./parse-pos.py -h
usage: parse-pos.py [-h] [-qa] [-qb] COMMAND [ARGS...]
# ./parse-pos.py -qa ls -q -h aa /bb
try_argv = ['-qa', 'ls']
cmd_rest_argv = ['-q', '-h', 'aa', '/bb']
parsed_args = Namespace(command='ls', qa=True, qb=False)
The code:
#!/usr/bin/python3
import argparse
import sys
from pprint import pprint
class CustomParserError(Exception):
pass
class CustomArgumentParser(argparse.ArgumentParser):
def error(self, message):
raise CustomParserError(message)
def original_error(self, message):
super().error(message)
def parse_argv():
parser = CustomArgumentParser(description='Example')
parser.add_argument('command', metavar='COMMAND [ARGS...]', help='the command to be executed')
parser.add_argument('-qa', action='store_true') # "ambiguous option" if you specify just "-q"
parser.add_argument('-qb', action='store_true') # "ambiguous option" if you specify just "-q"
def parse_until_positional(parser, _sys_argv = None):
if _sys_argv is None:
_sys_argv = sys.argv[1:] # skip the program name
for i in range(0, len(_sys_argv) + 1):
try_argv = _sys_argv[0:i]
try:
parsed_args = parser.parse_args(try_argv)
except CustomParserError as ex:
if len(try_argv) == len(_sys_argv):
# this is our last try and we still couldn't parse anything
parser.original_error(str(ex)) # sys.exit()
continue
# if we are here, we parsed our known optional & dash-prefixed parameters and the COMMAND
cmd_rest_argv = _sys_argv[i:]
break
return (parsed_args, cmd_rest_argv, try_argv)
(parsed_args, cmd_rest_argv, try_argv) = parse_until_positional(parser)
# debug
pprint(try_argv)
pprint(cmd_rest_argv)
pprint(parsed_args)
return (parsed_args, cmd_rest_argv)
def main():
parse_argv()
main()
Another option is to use parse_known_args, which stops parsing when an unknown argument is encountered.
I am using the argparse package of Python 2.7 to write some option-parsing logic for a command-line tool. The tool should accept one of the following arguments:
"ON": Turn a function on.
"OFF": Turn a function off.
[No arguments provided]: Echo the current state of the function.
Looking at the argparse documentation led me to believe that I wanted two--possibly three--subcommands to be defined, since these three states are mutually exclusive and represent different conceptual activities. This is my current attempt at the code:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser.set_defaults(func=print_state) # I think this line is wrong.
parser_on = subparsers.add_parser('ON')
parser_on.set_defaults(func=set_state, newstate='ON')
parser_off = subparsers.add_parser('OFF')
parser_off.set_defaults(func=set_state, newstate='OFF')
args = parser.parse_args()
if(args.func == set_state):
set_state(args.newstate)
elif(args.func == print_state):
print_state()
else:
args.func() # Catchall in case I add more functions later
I was under the impression that if I provided 0 arguments, the main parser would set func=print_state, and if I provided 1 argument, the main parser would use the appropriate subcommand's defaults and call func=set_state. Instead, I get the following error with 0 arguments:
usage: cvsSecure.py [-h] {ON,OFF} ...
cvsSecure.py: error: too few arguments
And if I provide "OFF" or "ON", print_state gets called instead of set_state. If I comment out the parser.set_defaults line, set_state is called correctly.
I'm a journeyman-level programmer, but a rank beginner to Python. Any suggestions about how I can get this working?
Edit: Another reason I was looking at subcommands was a potential fourth function that I am considering for the future:
"FORCE txtval": Set the function's state to txtval.
The defaults of the top-level parser override the defaults on the sub-parsers, so setting the default value of func on the sub-parsers is ignored, but the value of newstate from the sub-parser defaults is correct.
I don't think you want to use subcommands. Subcommands are used when the available options and positional arguments change depending on which subcommand is chosen. However, you have no other options or positional arguments.
The following code seems to do what you require:
import argparse
def print_state():
print "Print state"
def set_state(s):
print "Setting state to " + s
parser = argparse.ArgumentParser()
parser.add_argument('state', choices = ['ON', 'OFF'], nargs='?')
args = parser.parse_args()
if args.state is None:
print_state()
elif args.state in ('ON', 'OFF'):
set_state(args.state)
Note the optional parameters to parser.add_argument. The "choices" parameter specifies the allowable options, while setting "nargs" to "?" specifies that 1 argument should be consumed if available, otherwise none should be consumed.
Edit: If you want to add a FORCE command with an argument and have separate help text for the ON and OFF command then you do need to use subcommands. Unfortunately there doesn't seem to be a way of specifying a default subcommand. However, you can work around the problem by checking for an empty argument list and supplying your own. Here's some sample code illustrating what I mean:
import argparse
import sys
def print_state(ignored):
print "Print state"
def set_state(s):
print "Setting state to " + s
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
on = subparsers.add_parser('ON', help = 'On help here.')
on.set_defaults(func = set_state, newstate = 'ON')
off = subparsers.add_parser('OFF', help = 'Off help here.')
off.set_defaults(func = set_state, newstate = 'OFF')
prt = subparsers.add_parser('PRINT')
prt.set_defaults(func = print_state, newstate = 'N/A')
force = subparsers.add_parser('FORCE' , help = 'Force help here.')
force.add_argument('newstate', choices = [ 'ON', 'OFF' ])
force.set_defaults(func = set_state)
if (len(sys.argv) < 2):
args = parser.parse_args(['PRINT'])
else:
args = parser.parse_args(sys.argv[1:])
args.func(args.newstate)
There are two problems with your approach.
First you probably already noticed that newstate is not some sub_value of the sub parser and needs to be addressed at the top level of args as args.newstate. That should explain that assigning a default to newstate twice will result in the first value being overwritten. Whether you call your programm with 'ON' or 'OFF' as a parameter, each time set_state() will be called with OFF. If you just want to be able to do python cvsSecure ON and
python cvsSecure OFF the following would work:
from __future__ import print_function
import sys
import argparse
def set_state(state):
print("set_state", state)
def do_on(args):
set_state('ON')
def do_off(args):
set_state('OFF')
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_on = subparsers.add_parser('ON')
parser_on.set_defaults(func=do_on)
parser_on.add_argument('--fast', action='store_true')
parser_off = subparsers.add_parser('OFF')
parser_off.set_defaults(func=do_off)
args = parser.parse_args()
args.func(args)
The second problem is that argparse does handle subparsers as single value arguments, so you have to specify one before invoking parser.parse_args(). You can automate insertion of a lacking argument by adding a extra subparser 'PRINT' and automatically inserting
that using set_default_subparser added to argparse.ArgumentParser() (that code is part
of the package ruamel.std.argparse
from __future__ import print_function
import sys
import argparse
def set_default_subparser(self, name, args=None):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if not subparser_found:
# insert default in first position, this implies no
# global options without a sub_parsers specified
if args is None:
sys.argv.insert(1, name)
else:
args.insert(0, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
def print_state(args):
print("print_state")
def set_state(state):
print("set_state", state)
def do_on(args):
set_state('ON')
def do_off(args):
set_state('OFF')
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_print = subparsers.add_parser('PRINT', help='default action')
parser_print.set_defaults(func=print_state)
parser_on = subparsers.add_parser('ON')
parser_on.set_defaults(func=do_on)
parser_on.add_argument('--fast', action='store_true')
parser_off = subparsers.add_parser('OFF')
parser_off.set_defaults(func=do_off)
parser.set_default_subparser('PRINT')
args = parser.parse_args()
args.func(args)
You don't need to handle in args to do_on(), etc., but it comes in handy if you start specifying options to the different subparsers.