Develop Reference

Make functions importable when using argparse

I have the following CLI program which adds two numbers:
import argparse
def foo(args):
print('X + Y:', args.x + args.y)
return args.x + args.y
if __name__ == '__main__':
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
foo_parser = subparsers.add_parser('foo')
foo_parser.add_argument('-x', type=int, default=1)
foo_parser.add_argument('-y', type=int, default=2)
foo_parser.set_defaults(func=foo)
parser.add_argument('--debug', action='store_true', default=False)
args = parser.parse_args()
args.func(args)
Suppose now I want my users to also be able to import foo and call it directly with arguments x and y. I.e. I want foo to look like this:
def foo(x, y):
print('X + Y:', x + y)
return x + y
How can I adapt args.func(args) to handle this new foo?

The use of args.func(**vars(args)) is not a best fit for use-cases which require import as well as a CLI view
When users import a function and call it, they expect a return value for further processing (nothing printed on a console. The caller can decide to print based on the result obtained)
When they use a CLI, they expect to see an output printed on the console and a proper exit code (0 or 1 based on the return value)
The ideal way is to separate the parsing/CLI management/sum-function into separate functions and delegate processing once the parsing is complete (below is a sample example)
from __future__ import print_function # for python2
import argparse
def mysum(x, y=5):
return x+y
def delegator(input_args):
func_map = {
"mysum" : {
"func" : mysum,
"args": (input_args.get("x"),),
"kwargs" : {
"y" : input_args.get("y")
},
"result_renderer": print
}
}
func_data = func_map.get(input_args.get("action_to_perform"))
func = func_data.get("func")
args = func_data.get("args")
kwargs = func_data.get("kwargs")
renderer = func_data.get("result_renderer")
renderer(func(*args, **kwargs))
if __name__ == '__main__':
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="action_to_perform")
foo_parser = subparsers.add_parser('mysum')
foo_parser.add_argument('-x', metavar='PATH', type=int, default=1)
foo_parser.add_argument('-y', metavar='PATH', type=int, default=3)
delegator(vars(parser.parse_args()))
Above example would also remove *args, **kwargs from your original function and lets the delegator send only what is needed by the function based on the command
You can extend it to support multiple commands
Hope this helps!

This is the cleanest way I've found so far:
import argparse
def foo(*, x, y, **kwargs):
print('X + Y:', x + y)
return x + y
if __name__ == '__main__':
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
foo_parser = subparsers.add_parser('foo')
foo_parser.add_argument('-x', metavar='PATH', type=int, default=1)
foo_parser.add_argument('-y', metavar='PATH', type=int, default=2)
foo_parser.set_defaults(func=foo)
parser.add_argument('--debug', action='store_true', default=False)
args = parser.parse_args()
args.func(**vars(args))
Maybe someone else can find something better.

Conditional passing of arguments to methods in python

I have many possible arguments from argparse that I want to pass to a function. If the variable hasn't been set, I want the method to use its default variable. However, handling which arguments have been set and which haven't is tedious:
import argparse
def my_func(a = 1, b = 2):
return a+b
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Get the numeric values.')
parser.add_argument('-a', type=int)
parser.add_argument('-b', type=int)
args = parser.parse_args()
if not args.a is None and not args.b is None:
result = my_func(a = args.a, b = args.b)
elif not args.a is None and args.b is None:
result = my_func(a = args.a)
elif not args.b is None and args.a is None:
result = my_func(b = args.b)
else:
result = my_func()
It seems like I should be able to do something like this:
result = my_func(a = args.a if not args.a is None, b = args.b if not args.b is None)
But this gives a syntax error on the comma.
I could set default values in the argparser, but I want to use the defaults set in the method definition.

Use a dictionary with the kwargs unpacking syntax.
args = parser.parse_args()
result = my_func(**vars(args))
Edit
Use the SUPPRESS argument to ArgumentParser to remove empty values:
parser = argparse.ArgumentParser(description='Get the numeric values.',
argument_default=argparse.SUPPRESS)

The first solution that comes to me seems kind of hacky...but here it is.
Use inspect to write a function that looks at the arguments of a function and only passes it those arguments from args which it accepts and are not None. My guess is that this would be widely considered bad practice...
import inspect
def call_function(fn, args):
argspec = inspect.getargspec(fn)
arglist = {}
for arg in argspec.args:
if arg in args.__dict__.keys() and args.__dict__[arg] is not None:
arglist[arg] = args.__dict__[arg]
return fn(**arglist)
Here it is in action:
import argparse
def my_func(a=1, c=2):
return a,c
a=None
b=2
c=3
args=argparse.Namespace(a=a,b=b,c=c)
call_function(my_func, args)
>> (1, 3)
This solution is quite under-tested and might need work to make it more robust, but the idea is there and should work in simple cases.

How to have sub-parser arguments in separate namespace with argparse?

I have the following test-code
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", default = 0, type=int)
subparsers = parser.add_subparsers(dest = "parser_name")
parser_lan = subparsers.add_parser('car')
parser_lan.add_argument("--boo")
parser_lan.add_argument("--foo")
parser_serial = subparsers.add_parser('bus')
parser_serial.add_argument("--fun")
print parser.parse_args()
which defines two sub-parsers, having a different set of arguments. When I call the testcode as
tester.py --verbose 3 car --boo 1 --foo 2
I get the expected result
Namespace(boo='1', foo='2', parser_name='car', verbose=3)
What I want to have instead is the values from each subparser in a separate namespace or dict, something like
Namespace(subparseargs={boo:'1', foo:'2'}, parser_name='car', verbose=3)
so that the arguments from each subparser are logical separated from the arguments from the main parser (as verbose in this example).
How can I achieve this, with the arguments for each subparser in the same namespace (subparseargs in the example).

You need to go into the bowels of argparse a bit but changing your script to the following should do the trick:
import argparse
from argparse import _HelpAction, _SubParsersAction
class MyArgumentParser(argparse.ArgumentParser):
def parse_args(self, *args, **kw):
res = argparse.ArgumentParser.parse_args(self, *args, **kw)
from argparse import _HelpAction, _SubParsersAction
for x in parser._subparsers._actions:
if not isinstance(x, _SubParsersAction):
continue
v = x.choices[res.parser_name] # select the subparser name
subparseargs = {}
for x1 in v._optionals._actions: # loop over the actions
if isinstance(x1, _HelpAction): # skip help
continue
n = x1.dest
if hasattr(res, n): # pop the argument
subparseargs[n] = getattr(res, n)
delattr(res, n)
res.subparseargs = subparseargs
return res
parser = MyArgumentParser()
parser.add_argument("--verbose", default = 0, type=int)
subparsers = parser.add_subparsers(dest = "parser_name")
parser_lan = subparsers.add_parser('car')
parser_lan.add_argument("--boo")
parser_lan.add_argument("--foo")
parser_serial = subparsers.add_parser('bus')
parser_serial.add_argument("--fun")
print parser.parse_args()

I have started to develop a different approach (but similar to the suggestion by Anthon) and come up with a much shorter code. However, I am not sure my approach is a general solution for the problem.
To similar what Anthon is proposing, I define a new method which creates a list of 'top-level' arguments which are kept in args, while all the other arguments are returned as an additional dictionary:
class MyArgumentParser(argparse.ArgumentParser):
def parse_subargs(self, *args, **kw):
# parse as usual
args = argparse.ArgumentParser.parse_args(self, *args, **kw)
# extract the destination names for top-level arguments
topdest = [action.dest for action in parser._actions]
# loop over all arguments given in args
subargs = {}
for key, value in args.__dict__.items():
# if sub-parser argument found ...
if key not in topdest:
# ... remove from args and add to dictionary
delattr(args,key)
subargs[key] = value
return args, subargs
Comments on this approach welcome, especially any loopholes I overlooked.

Or manually, you could parse the args and create a dict with details:
# parse args
args = parser.parse_args()
args_dict = {}
for group in parser._action_groups:
# split into groups based on title
args_dict[group.title] = {}
for arg in group._group_actions:
if hasattr(args, arg.dest):
args_dict[group.title][arg.dest] = getattr(args, arg.dest)
# or args_dict[arg.dest] = getattr(args, arg.dest)
delattr(args, arg.dest)
# add remaining items into subparser options
args_dict["subparser"] |= vars(args)
return args_dict

Python - Pass Arguments to Different Methods from Argparse

I'm writing a relatively simple Python script which supports a couple of different commands. The different commands support different options and I want to be able to pass the options parsed by argparse to the correct method for the specified command.
The usage string looks like so:
usage: script.py [-h]
{a, b, c}
...
script.py: error: too few arguments
I can easily call the appropriate method:
def a():
...
def b():
...
def c():
...
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.set_defaults(method = a)
...
arguments = parser.parse_args()
arguments.method()
However, I have to pass arguments to these methods and they all accept different sets of arguments.
Currently, I just pass the Namespace object returned by argparse, like so:
def a(arguments):
arg1 = getattr(arguments, 'arg1', None)
...
This seems a little awkward, and makes the methods a little harder to reuse as I have to pass arguments as a dict or namespace rather than as usual parameters.
I would like someway of defining the methods with parameters (as you would a normal function) and still be able to call them dynamically while passing appropriate parameters. Like so:
def a(arg1, arg2):
...
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.set_defaults(method = a)
...
arguments = parser.parse_args()
arguments.method() # <<<< Arguments passed here somehow
Any ideas?

I found quite a nice solution:
import argparse
def a(arg1, arg2, **kwargs):
print arg1
print arg2
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.set_defaults(method = a)
parser.add_argument('arg1', type = str)
parser.add_argument('arg2', type = str)
arguments = parser.parse_args()
arguments.method(**vars(arguments))
Of course there's a minor problem if the arguments of the method clash with the names of the arguments argparse uses, though I think this is preferable to passing the Namespace object around and using getattr.

You're probably trying to achieve the functionality that sub-commands provide:
http://docs.python.org/dev/library/argparse.html#sub-commands

Not sure how practical this is, but by using inspect you can leave out the extraneous **kwargs parameter on your functions, like so:
import argparse
import inspect
def sleep(seconds=0):
print "sleeping", seconds, "seconds"
def foo(a, b=2, **kwargs):
print "a=",a
print "b=",b
print "kwargs=",kwargs
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title="subcommand")
parser_sleep = subparsers.add_parser('sleep')
parser_sleep.add_argument("seconds", type=int, default=0)
parser_sleep.set_defaults(func=sleep)
parser_foo = subparsers.add_parser('foo')
parser_foo.add_argument("-a", type=int, default=101)
parser_foo.add_argument("-b", type=int, default=201)
parser_foo.add_argument("--wacky", default=argparse.SUPPRESS)
parser_foo.set_defaults(func=foo)
args = parser.parse_args()
arg_spec = inspect.getargspec(args.func)
if arg_spec.keywords:
## convert args to a dictionary
args_for_func = vars(args)
else:
## get a subset of the dictionary containing just the arguments of func
args_for_func = {k:getattr(args, k) for k in arg_spec.args}
args.func(**args_for_func)
Examples:
$ python test.py sleep 23
sleeping 23 seconds
$ python test.py foo -a 333 -b 444
a= 333
b= 444
kwargs= {'func': <function foo at 0x10993dd70>}
$ python test.py foo -a 333 -b 444 --wacky "this is wacky"
a= 333
b= 444
kwargs= {'func': <function foo at 0x10a321d70>, 'wacky': 'this is wacky'}
Have fun!

Default sub-command, or handling no sub-command with argparse

How can I have a default sub-command, or handle the case where no sub-command is given using argparse?
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
a.parse_args()
Here I'd like a command to be selected, or the arguments to be handled based only on the next highest level of parser (in this case the top-level parser).
joiner#X:~/src> python3 default_subcommand.py
usage: default_subcommand.py [-h] {hi} ...
default_subcommand.py: error: too few arguments

On Python 3.2 (and 2.7) you will get that error, but not on 3.3 and 3.4 (no response). Therefore on 3.3/3.4 you could test for parsed_args to be an empty Namespace.
A more general solution is to add a method set_default_subparser() (taken from the ruamel.std.argparse package) and call that method just before parse_args():
import argparse
import sys
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if not subparser_found:
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
def do_hi():
print('inside hi')
a = argparse.ArgumentParser()
b = a.add_subparsers()
sp = b.add_parser('hi')
sp.set_defaults(func=do_hi)
a.set_default_subparser('hi')
parsed_args = a.parse_args()
if hasattr(parsed_args, 'func'):
parsed_args.func()
This will work with 2.6 (if argparse is installed from PyPI), 2.7, 3.2, 3.3, 3.4. And allows you to do both
python3 default_subcommand.py
and
python3 default_subcommand.py hi
with the same effect.
Allowing to chose a new subparser for default, instead of one of the existing ones.
The first version of the code allows setting one of the previously-defined subparsers as a default one. The following modification allows adding a new default subparser, which could then be used to specifically process the case when no subparser was selected by user (different lines marked in the code)
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
existing_default = False # check if default parser previously defined
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if sp_name == name: # check existance of default parser
existing_default = True
if not subparser_found:
# If the default subparser is not among the existing ones,
# create a new parser.
# As this is called just before 'parse_args', the default
# parser created here will not pollute the help output.
if not existing_default:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
x.add_parser(name)
break # this works OK, but should I check further?
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
a = argparse.ArgumentParser()
b = a.add_subparsers(dest ='cmd')
sp = b.add_parser('hi')
sp2 = b.add_parser('hai')
a.set_default_subparser('hey')
parsed_args = a.parse_args()
print(parsed_args)
The "default" option will still not show up in the help:
python test_parser.py -h
usage: test_parser.py [-h] {hi,hai} ...
positional arguments:
{hi,hai}
optional arguments:
-h, --help show this help message and exit
However, it is now possible to differentiate between and separately handle calling one of the provided subparsers, and calling the default subparser when no argument was provided:
$ python test_parser.py hi
Namespace(cmd='hi')
$ python test_parser.py
Namespace(cmd='hey')

It seems I've stumbled on the solution eventually myself.
If the command is optional, then this makes the command an option. In my original parser configuration, I had a package command that could take a range of possible steps, or it would perform all steps if none was given. This makes the step a choice:
parser = argparse.ArgumentParser()
command_parser = subparsers.add_parser('command')
command_parser.add_argument('--step', choices=['prepare', 'configure', 'compile', 'stage', 'package'])
...other command parsers
parsed_args = parser.parse_args()
if parsed_args.step is None:
do all the steps...

Here's a nicer way of adding a set_default_subparser method:
class DefaultSubcommandArgParse(argparse.ArgumentParser):
__default_subparser = None
def set_default_subparser(self, name):
self.__default_subparser = name
def _parse_known_args(self, arg_strings, *args, **kwargs):
in_args = set(arg_strings)
d_sp = self.__default_subparser
if d_sp is not None and not {'-h', '--help'}.intersection(in_args):
for x in self._subparsers._actions:
subparser_found = (
isinstance(x, argparse._SubParsersAction) and
in_args.intersection(x._name_parser_map.keys())
)
if subparser_found:
break
else:
# insert default in first position, this implies no
# global options without a sub_parsers specified
arg_strings = [d_sp] + arg_strings
return super(DefaultSubcommandArgParse, self)._parse_known_args(
arg_strings, *args, **kwargs
)

Maybe what you're looking for is the dest argument of add_subparsers:
(Warning: works in Python 3.4, but not in 2.7)
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='cmd')
parser_hi = subparsers.add_parser('hi')
parser.parse_args([]) # Namespace(cmd=None)
Now you can just use the value of cmd:
if cmd in [None, 'hi']:
print('command "hi"')

You can duplicate the default action of a specific subparser on the main parser, effectively making it the default.
import argparse
p = argparse.ArgumentParser()
sp = p.add_subparsers()
a = sp.add_parser('a')
a.set_defaults(func=do_a)
b = sp.add_parser('b')
b.set_defaults(func=do_b)
p.set_defaults(func=do_b)
args = p.parse_args()
if args.func:
args.func()
else:
parser.print_help()
Does not work with add_subparsers(required=True), which is why the if args.func is down there.

In my case I found it easiest to explicitly provide the subcommand name to parse_args() when argv was empty.
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='commands')
runParser = subparsers.add_parser('run', help='[DEFAULT ACTION]')
altParser = subparsers.add_parser('alt', help='Alternate command')
altParser.add_argument('alt_val', type=str, help='Value required for alt command.')
# Here's my shortcut: If `argv` only contains the script name,
# manually inject our "default" command.
args = parser.parse_args(['run'] if len(sys.argv) == 1 else None)
print args
Example runs:
$ ./test.py
Namespace()
$ ./test.py alt blah
Namespace(alt_val='blah')
$ ./test.py blah
usage: test.py [-h] {run,alt} ...
test.py: error: invalid choice: 'blah' (choose from 'run', 'alt')

In python 2.7, you can override the error behaviour in a subclass (a shame there isn't a nicer way to differentiate the error):
import argparse
class ExceptionArgParser(argparse.ArgumentParser):
def error(self, message):
if "invalid choice" in message:
# throw exception (of your choice) to catch
raise RuntimeError(message)
else:
# restore normal behaviour
super(ExceptionArgParser, self).error(message)
parser = ExceptionArgParser()
subparsers = parser.add_subparsers(title='Modes', dest='mode')
default_parser = subparsers.add_parser('default')
default_parser.add_argument('a', nargs="+")
other_parser = subparsers.add_parser('other')
other_parser.add_argument('b', nargs="+")
try:
args = parser.parse_args()
except RuntimeError:
args = default_parser.parse_args()
# force the mode into namespace
setattr(args, 'mode', 'default')
print args

Here's another solution using a helper function to build a list of known subcommands:
import argparse
def parse_args(argv):
parser = argparse.ArgumentParser()
commands = []
subparsers = parser.add_subparsers(dest='command')
def add_command(name, *args, **kwargs):
commands.append(name)
return subparsers.add_parser(name, *args, **kwargs)
hi = add_command("hi")
hi.add_argument('--name')
add_command("hola")
# check for default command
if not argv or argv[0] not in commands:
argv.insert(0, "hi")
return parser.parse_args(argv)
assert parse_args([]).command == 'hi'
assert parse_args(['hi']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).name == 'John'
assert parse_args(['--name', 'John']).command == 'hi'
assert parse_args(['hola']).command == 'hola'

You can add an argument with a default value that will be used when nothing is set I believe.
See this: http://docs.python.org/dev/library/argparse.html#default
Edit:
Sorry, I read your question a bit fast.
I do not think you would have a direct way of doing what you want via argparse. But you could check the length of sys.argv and if its length is 1 (only script name) then you could manually pass the default parameters for parsing, doing something like this:
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
if len(sys.argv) == 1:
a.parse_args(['hi'])
else:
a.parse_args()
I think that should do what you want, but I agree it would be nice to have this out of the box.

For later reference:
...
b = a.add_subparsers(dest='cmd')
b.set_defaults(cmd='hey') # <-- this makes hey as default
b.add_parser('hi')
so, these two will be same:
python main.py hey
python main.py