I'm trying to build a CLI in Python and I have an argument (--arg) that I want to reuse across multiple subcommands (req and opt).
But one subcommand (req) will have to require --arg and the other (opt) doesn't. How do I resolve this without having to make two versions of the same argument?
import argparse
arg_1 = argparse.ArgumentParser(add_help=False)
arg_1.add_argument('-a', '--arg', required=True,
help='reusable argument')
parser = argparse.ArgumentParser()
subp = parser.add_subparsers()
cmd_require = subp.add_parser('req', parents=[arg_1],
help='this subcommand requires --arg')
cmd_optional = subp.add_parser('opt', parents=[arg_1],
help='this subcommand doesn\'t require --arg')
I don't know any 'native' argparse feature that does that. However, I thought of 2 different approaches to solve your problem.
Validate args in a separate function -
Sometimes CLI application get complicated and by adding a validator function you can 'complete' the missing argparse features you wish for.
import argparse
arg_1 = argparse.ArgumentParser(add_help=False)
arg_1.add_argument('-a', '--arg', required=False,
help='reusable argument')
parser = argparse.ArgumentParser()
subp = parser.add_subparsers(dest='sub_parser')
cmd_require = subp.add_parser('req', parents=[arg_1],
help='this subcommand requires --arg')
cmd_optional = subp.add_parser('opt', parents=[arg_1],
help='this subcommand doesn\'t require --arg')
def validate_args(args):
print(args)
if args.sub_parser == 'req' and not args.arg:
print("Invalid usage! using 'req' requires 'arg'")
exit(1)
if __name__ == '__main__':
args = parser.parse_args()
validate_args(args)
Note:
I used dest for the subparser in order to later identify the chosen
subparser.
Using argparse, if an optional argument is not passed it will be 'None'
"prepared argument" -
Although argparse doesn't support an argument object - you could 'prepare' an argument by unpacking a dict and a tuple (*args, **kwargs)
import argparse
arg_name = ('-a', '--arg')
arg_dict = {'help': 'reusable argument'}
parser = argparse.ArgumentParser()
subp = parser.add_subparsers()
cmd_require = subp.add_parser('req',
help='this subcommand requires --arg')
cmd_optional = subp.add_parser('opt',
help='this subcommand doesn\'t require --arg')
cmd_optional.add_argument(*arg_name, **arg_dict, required=False)
cmd_require.add_argument(*arg_name, **arg_dict, required=True)
if __name__ == '__main__':
args = parser.parse_args()
validate_args(args)
I like the first approach better.
Hope you find that useful
import argparse
arg_1 = argparse.ArgumentParser(add_help=False)
foobar = arg_1.add_argument('-a', '--arg', required=True,
help='reusable argument')
arg_1 is a parser object. When you use the add_argument command, it creates an Action object and adds it to the args_1._actions list. I just saved a reference to that in the foobar variable.
parser = argparse.ArgumentParser()
subp = parser.add_subparsers()
The parents mechanism adds the args_1._actions list to the cmd_require._actions list. So foobar will appear in both subparsers. It's copy by reference, which is common in python.
cmd_require = subp.add_parser('req', parents=[arg_1],
help='this subcommand requires --arg')
cmd_optional = subp.add_parser('opt', parents=[arg_1],
help='this subcommand doesn\'t require --arg')
foobar.required=False will turn off at attribute, but will do so for both parsers. I've seen this issue come up when people wanted to assign different default attributes.
The parents mechanism just a shortcut, that occasionally is useful, but not always. It doesn't do anything special; just saves a bit of typing. There are plenty other ways of defining an Action with the same flags in both subparsers.
typing it twice (horror of horrors!)
copy-n-paste
writing a utility function to add arguments to subparsers (see Larry Wall's Three Virtues)
Related
Imagine you've got common arguments for several subparsers:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
"--learn_rate",
type=float,
)
subparsers = parser.add_subparsers(help='task', dest='lib')
spacy_parser = subparsers.add_parser(
"spacy",
)
args = vars(parser.parse_args())
From the command line, you have to run python test.py --learn_rate 2 spacy. Is it possible to make it so that python test.py spacy --learn_rate 2 also works?
learn_rate isn't an option common to your subparsers; it's an option on the main command, which is available to your code regardless of which subparser gets invoked. If you truly want to share an option among multiple subparsers as in your second use case, you need to define it in a parent parser.
import argparse
parser = argparse.ArgumentParser()
shared_parent = argparse.ArgumentParser(add_help=False)
shared_parent.add_argument(
"--learn_rate",
type=float,
)
subparsers = parser.add_subparsers(help='task', dest='lib')
spacy_parser = subparsers.add_parser(
"spacy",
parents=[shared_parent]
)
args = vars(parser.parse_args())
Allowing --learn_rate to be used in either position is trickier. While you could define the shared_parent parser first, then add it to the main parser as well with parser = argparse.ArgumentParser(parents=[shared_parent]), the subcommand will overwrite whatever value you specify from the main parser with the default if you don't use the option from the subparser. Working around this behavior of argparse would probably require a custom action at the very least.
This works:
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
subparsers = parser.add_subparsers(help='task', dest='lib')
spacy_parser = subparsers.add_parser(
"spacy",
help="Spacy's Textcat",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
ulmfit_parser = subparsers.add_parser(
"ulmfit",
help="Fastai's ULMFiT",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
for subparser in subparsers.choices.values():
subparser.add_argument(...)
Structure of my parser:
parser = argparse.ArgumentParser()
parser.add_argument('-s', '--search')
parser.add_argument('-t', '--status' action='store_true')
args = parser.parse_args()
if args.search:
func(args.search)
if args.status:
func1()
Right now the parser can accept both options, -s query -t is valid.
I have two questions:
How to take actions on the first argument only if multiple are passed.
-s query -t have to result only
if args.search:
func(args.search)
to be done.
How to throw an error if multiple args are passed?
The appropriate tool here is a subparser; idiomatic usage is different than what your currently proposed command line looks like.
def your_search_function(options):
pass # do a search here
def your_status_function(options):
pass # collect status here
def main():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparser(dest='action')
search_parser = subparsers.add_parser('search')
search_parser.set_defaults(func=your_search_function)
status_parser = subparsers.add_parser('status')
status_parser.set_defaults(func=your_status_function)
results = parser.parse_args()
results.func(options=results)
if __name__ == '__main__':
main()
Usage then looks like:
./yourcommand <global options> search <search options>
or
./yourcommand <global options> status <status options>
...which, since it takes the subcommand to run as an argument rather than an option, does not allow multiple subcommands to be passed, thus mooting the parts of your question only applicable when usage is ambiguous.
Global options can be added to parser as usual; search-specific options to search_parser, and status-specific options to status_parser.
I'm new to python and currently playing with it.
I have a script which does some API Calls to an appliance. I would like to extend the functionality and call different functions based on the arguments given when calling the script.
Currently I have the following:
parser = argparse.ArgumentParser()
parser.add_argument("--showtop20", help="list top 20 by app",
action="store_true")
parser.add_argument("--listapps", help="list all available apps",
action="store_true")
args = parser.parse_args()
I also have a
def showtop20():
.....
and
def listapps():
....
How can I call the function (and only this) based on the argument given?
I don't want to run
if args.showtop20:
#code here
if args.listapps:
#code here
as I want to move the different functions to a module later on keeping the main executable file clean and tidy.
Since it seems like you want to run one, and only one, function depending on the arguments given, I would suggest you use a mandatory positional argument ./prog command, instead of optional arguments (./prog --command1 or ./prog --command2).
so, something like this should do it:
FUNCTION_MAP = {'top20' : my_top20_func,
'listapps' : my_listapps_func }
parser.add_argument('command', choices=FUNCTION_MAP.keys())
args = parser.parse_args()
func = FUNCTION_MAP[args.command]
func()
At least from what you have described, --showtop20 and --listapps sound more like sub-commands than options. Assuming this is the case, we can use subparsers to achieve your desired result. Here is a proof of concept:
import argparse
import sys
def showtop20():
print('running showtop20')
def listapps():
print('running listapps')
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# Create a showtop20 subcommand
parser_showtop20 = subparsers.add_parser('showtop20', help='list top 20 by app')
parser_showtop20.set_defaults(func=showtop20)
# Create a listapps subcommand
parser_listapps = subparsers.add_parser('listapps', help='list all available apps')
parser_listapps.set_defaults(func=listapps)
# Print usage message if no args are supplied.
# NOTE: Python 2 will error 'too few arguments' if no subcommand is supplied.
# No such error occurs in Python 3, which makes it feasible to check
# whether a subcommand was provided (displaying a help message if not).
# argparse internals vary significantly over the major versions, so it's
# much easier to just override the args passed to it.
if len(sys.argv) <= 1:
sys.argv.append('--help')
options = parser.parse_args()
# Run the appropriate function (in this case showtop20 or listapps)
options.func()
# If you add command-line options, consider passing them to the function,
# e.g. `options.func(options)`
There are lots of ways of skinning this cat. Here's one using action='store_const' (inspired by the documented subparser example):
p=argparse.ArgumentParser()
p.add_argument('--cmd1', action='store_const', const=lambda:'cmd1', dest='cmd')
p.add_argument('--cmd2', action='store_const', const=lambda:'cmd2', dest='cmd')
args = p.parse_args(['--cmd1'])
# Out[21]: Namespace(cmd=<function <lambda> at 0x9abf994>)
p.parse_args(['--cmd2']).cmd()
# Out[19]: 'cmd2'
p.parse_args(['--cmd1']).cmd()
# Out[20]: 'cmd1'
With a shared dest, each action puts its function (const) in the same Namespace attribute. The function is invoked by args.cmd().
And as in the documented subparsers example, those functions could be written so as to use other values from Namespace.
args = parse_args()
args.cmd(args)
For sake of comparison, here's the equivalent subparsers case:
p = argparse.ArgumentParser()
sp = p.add_subparsers(dest='cmdstr')
sp1 = sp.add_parser('cmd1')
sp1.set_defaults(cmd=lambda:'cmd1')
sp2 = sp.add_parser('cmd2')
sp2.set_defaults(cmd=lambda:'cmd2')
p.parse_args(['cmd1']).cmd()
# Out[25]: 'cmd1'
As illustrated in the documentation, subparsers lets you define different parameter arguments for each of the commands.
And of course all of these add argument or parser statements could be created in a loop over some list or dictionary that pairs a key with a function.
Another important consideration - what kind of usage and help do you want? The different approaches generate very different help messages.
If your functions are "simple enough" take adventage of type parameter https://docs.python.org/2.7/library/argparse.html#type
type= can take any callable that takes a single string argument and
returns the converted value:
In your example (even if you don't need a converted value):
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
This simple script:
import argparse
def showtop20(dummy):
print "{0}\n".format(dummy) * 5
parser = argparse.ArgumentParser()
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
args = parser.parse_args()
Will give:
# ./test.py --listapps test
test
test
test
test
test
test
Instead of using your code as your_script --showtop20, make it into a sub-command your_script showtop20 and use the click library instead of argparse. You define functions that are the name of your subcommand and use decorators to specify the arguments:
import click
#click.group()
#click.option('--debug/--no-debug', default=False)
def cli(debug):
print(f'Debug mode is {"on" if debug else "off"}')
#cli.command() # #cli, not #click!
def showtop20():
# ...
#cli.command()
def listapps():
# ...
See https://click.palletsprojects.com/en/master/commands/
# based on parser input to invoke either regression/classification plus other params
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--path", type=str)
parser.add_argument("--target", type=str)
parser.add_argument("--type", type=str)
parser.add_argument("--deviceType", type=str)
args = parser.parse_args()
df = pd.read_csv(args.path)
df = df.loc[:, ~df.columns.str.contains('^Unnamed')]
if args.type == "classification":
classify = AutoML(df, args.target, args.type, args.deviceType)
classify.class_dist()
classify.classification()
elif args.type == "regression":
reg = AutoML(df, args.target, args.type, args.deviceType)
reg.regression()
else:
ValueError("Invalid argument passed")
# Values passed as : python app.py --path C:\Users\Abhishek\Downloads\adult.csv --target income --type classification --deviceType GPU
You can evaluate using evalwhether your argument value is callable:
import argparse
def list_showtop20():
print("Calling from showtop20")
def list_apps():
print("Calling from listapps")
my_funcs = [x for x in dir() if x.startswith('list_')]
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--function", required=True,
choices=my_funcs,
help="function to call", metavar="")
args = parser.parse_args()
eval(args.function)()
I'm trying to write a argparse system where it takes in 3 arguments, with 1 of them being optional. Depending on the presence of the 3rd argument, the program will decide how to proceed.
I current have this:
parser = argparse.ArgumentParser(description='Some description')
parser.add_argument('-f', '--first', help='Filename for f', required=True)
parser.add_argument('-s', '--second', help='Filename for s', required=True)
parser.add_argument('-t', '--third', help='Filename for t')
args = parser.parse_args()
if args.third:
parse.classify(args['first'], args['second'], args['third'])
else:
parse.classify(args['first'], args['second'], None)
I'm getting the following error: TypeError: 'Namespace' object is not subscriptable
Namespace objects aren't subscriptable; they have attributes.
parse.classify(args.first, args.second, args.third)
You can use vars to create a dict, though.
args = vars(args)
Then subscripting will work.
In my application, I have a parser like this:
description = ("Cluster a matrix using bootstrap resampling or "
"Bayesian hierarchical clustering.")
sub_description = ("Use these commands to cluster data depending on "
"the algorithm.")
parser = argparse.ArgumentParser(description=description, add_help=False)
subparsers = parser.add_subparsers(title="Sub-commands",
description=sub_description)
parser.add_argument("--no-logfile", action="store_true", default=False,
help="Don't log to file, use stdout")
parser.add_argument("source", metavar="FILE",
help="Source data to cluster")
parser.add_argument("destination", metavar="FILE",
help="File name for clustering results")
Then I add a series of sub parsers like this (using functions because they're long):
setup_pvclust_parser(subparsers, parser)
setup_native_parser(subparsers, parser)
These call (example with one):
def setup_pvclust_parser(subparser, parent=None):
pvclust_description = ("Perform multiscale bootstrap resampling "
"(Shimodaira et al., 2002)")
pvclust_commands = subparser.add_parser("bootstrap",
description=pvclust_description, parents=[parent])
pvclust_commands.add_argument("-b", "--boot", type=int,
metavar="BOOT",
help="Number of permutations",
default=100)
# Other long list of options...
pvclust_commands.set_defaults(func=cluster_pvclust) # The function doing the processing
The issue is that somehow the parsing of the command line fails, and I'm sure it's my fault, somewhere. Example when run:
my_program.py bootstrap --boot 10 --no-logfile test.txt test.pdf
my_program.py bootstrap: error: too few arguments
As if the parsing is somehow wrong. This behavior disappears if I remove parents=[] in the subparser call, but I'd rather avoid it as it creates massive duplication.
EDIT: Moving the subparsers call after the add_argument calls fixes part of the problem. However, now the parser cannot parse properly the subcommands:
my_program.py bootstrap --boot 100 --no-logfile test.txt test.pdf
my_program.py: error: invalid choice: 'test.txt' (choose from 'bootstrap', 'native')
The fundamental problem is that you are confusing the arguments that the parser is supposed to handle, with those that the subparsers should handle. In fact by passing the parser as parent to the subparser you end up defining those arguments in both places.
In addition source and destination are positional arguments, as is the subparsers. If they are all defined in the base parser, their order matters.
I'd suggest defining a separate parent parser
parent = argparse.ArgumentParser(add_help=False)
parent.add_argument("--no-logfile", action="store_true". help="...")
parent.add_argument("source", metavar="FILE", help="...")
parent.add_argument("destination", metavar="FILE", help="...")
parser = argparse.ArgumentParser(description=description)
subparsers = parser.add_subparsers(title="Sub-commands", description=sub_description)
setup_pvclust_parser(subparsers, parent)
setup_native_parser(subparsers, parent)