Develop Reference

Python unittest and argparse with required value not given

I am having trouble unit-testing a function I made using argparse library. I would like to test the case when no argument is given. This is the function I would like to test:
def parse_arguments():
parser = argparse.ArgumentParser(description='Testing parsing')
parser.add_argument('--reqVal', required=True, help='Required value')
parser.add_argument('--notReq', required=False, help='Value not required')
return parser.parse_args()
Here are my tests:
import argparse
from parse_gen import parse_arguments
from unittest.mock import patch
def test_parse_arguments_only_required_args(capsys):
args_namespace = argparse.Namespace()
args_namespace.reqVal = 'Required'
args_namespace.notReq = None
with patch('sys.argv', ['parse_arguments', '--reqVal', 'Required']):
parsed_args = parse_arguments()
assert parsed_args == args_namespace
def test_parse_arguments_no_args(capsys):
with patch('sys.argv', ['parse_arguments']):
parsed_args = parse_arguments()
assert parsed_args == argparse.ArgumentTypeError()
First test case is working as expected. My problem is that when I want to test no_args case I get an error but it is very long and I do not understand how to fix it. Could you please help me?

Is there a way to control the order the arguments are evaluated in Python argparse

Let's take the example below. The parser contains two arguments --inputfile and verbosity. The Set_verbosity_level() function is used to controls the value of a module-level/global variable (in my real life a package-level variable) to 0-4. The CheckFile() function performs tests inside input file (in the real life depending on type).
I would like to print messages in CheckFile() depending on verbosity. The problem is that argparse calls CheckFile() before Set_verbosity_level() so the verbosity level is always 0/default in CheckFile...
So my question is whether there is any solution to force argparse to evaluate some arguments before others...
import argparse
VERBOSITY = 0
def Set_verbosity_level():
"""Set the verbosity level.
"""
def type_func(value):
a_value = int(value)
globals()['VERBOSITY'] = value
print("Verbosity inside Set_verbosity_level(): " + str(globals()['VERBOSITY']))
return value
return type_func
class CheckFile(argparse.FileType):
"""
Check whatever in the file
"""
def __init__(self, mode='r', **kwargs):
super(CheckFile, self).__init__(mode, **kwargs)
def __call__(self, string):
# Do whatever processing/checking/transformation
# e.g print some message according to verbosity
print("Verbosity inside CheckFile(): " + str(globals()['VERBOSITY']))
return super(CheckFile, self).__call__(string)
def make_parser():
"""The main argument parser."""
parser = argparse.ArgumentParser(add_help=True)
parser.add_argument("-V",
"--verbosity",
default=0,
type=Set_verbosity_level(),
help="Increase output verbosity.",
required=False)
parser.add_argument('-i', '--inputfile',
help="Input file",
type=CheckFile(mode='r'),
required=True)
return parser
if __name__ == '__main__':
myparser = make_parser()
args = myparser.parse_args()
print("Verbosity in Main: " + str(VERBOSITY))
Calling this script gives:
$python test.py -i test.bed -V 2
Verbosity inside CheckFile(): 0
Verbosity inside Set_verbosity_level(): 2
Verbosity in Main: 2

argparse processes the command-line arguments in the order that they are listed, so if you simply swap the order of the given options, it would output in the verbosity you want:
python test.py -V 2 -i test.bed
This outputs:
Verbosity inside Set_verbosity_level(): 2
Verbosity inside CheckFile(): 2
There's no way otherwise to tell argparse to process the command-line arguments in a different order than how they're listed.

I don't know that you can force an argparse variable to be read first, but you can use pythons built in command line parser in your main function:
import sys
# Your classes here #
if __name__ == '__main__':
verbosity = 0
for i, sysarg in enumerate(sys.argv):
if str(sysarg).strip().lower().replace('-','') in ['v', 'verbose']:
try:
verbosity = sys.argv[i + 1]
except IndexError:
print("No verbosity level specified")
# more code
Its not very elegant and it's not argparse, but it's one way to ensure you get the verbosity first.
You could also update your CheckFile class to include a verbosity checking function:
class CheckFile(argparse.FileType):
"""
Check whatever in the file
"""
def __init__(self, mode='r', **kwargs):
super(CheckFile, self).__init__(mode, **kwargs)
def _check_verbosity(self):
verbosity = 0
for i, sysarg in enumerate(sys.argv):
if str(sysarg).strip().lower().replace('-','') in ['v', 'verbose']:
try:
verbosity = sys.argv[i + 1]
except IndexError:
print("No verbosity level specified")
return verbosity
def __call__(self, string):
# Do whatever processing/checking/transformation
# e.g print some message according to verbosity
print("Verbosity inside CheckFile(): {}".format(self._check_verbosity()))
return super(CheckFile, self).__call__(string)
Again, I know it's not really an answer to your argparse question, but it is a solution for your problem

Adding common parameters to groups with Click

I am trying to use the Python library Click, but struggle to get an example working. I defined two groups, one of which (group2) is meant to handle common parameters for this group of commands. What I want to achieve is that those common parameters get processed by the group function (group2) and assigned to the context variable, so they can be used by the actual commands.
A use case would be a number of commands that require username and password, while some others don't (not even optionally).
This is the code
import click
#click.group()
#click.pass_context
def group1(ctx):
pass
#click.group()
#click.option('--optparam', default=None, type=str)
#click.option('--optparam2', default=None, type=str)
#click.pass_context
def group2(ctx, optparam):
print 'in group2', optparam
ctx['foo'] = create_foo_by_processing_params(optparam, optparam2)
#group2.command()
#click.pass_context
def command2a(ctx):
print 'command2a', ctx['foo']
#group2.command()
#click.option('--another-param', default=None, type=str)
#click.pass_context
def command2b(ctx, another_param):
print 'command2b', ctx['foo'], another_param
# many more more commands here...
# #group2.command()
# def command2x():
# ...
#group1.command()
#click.argument('argument1')
#click.option('--option1')
def command1(argument1, option1):
print 'In command2', argument1, option1
cli = click.CommandCollection(sources=[group1, group2])
if __name__ == '__main__':
cli(obj={})
And this is the result when using command2:
$ python cli-test.py command2 --optparam=123
> Error: no such option: --optparam`
What's wrong with this example. I tried to follow the docs closely, but opt-param doesn't seem to be recognised.

The basic issue with the desired scheme is that click.CommandCollection does not call the group function. It skips directly to the command. In addition it is desired to apply options to the group via decorator, but have the options parsed by the command. That is:
> my_prog my_command --group-option
instead of:
> my_prog --group-option my_command
How?
This click.Group derived class hooks the command invocation for the commands to intercept the group parameters, and pass them to the group command.
In Group.add_command, add the params to the command
In Group.add_command, override command.invoke
In overridden command.invoke, take the special args inserted from the group and put them into ctx.obj and remove them from params
In overridden command.invoke, invoke the group command, and then the command itself
Code:
import click
class GroupWithCommandOptions(click.Group):
""" Allow application of options to group with multi command """
def add_command(self, cmd, name=None):
click.Group.add_command(self, cmd, name=name)
# add the group parameters to the command
for param in self.params:
cmd.params.append(param)
# hook the commands invoke with our own
cmd.invoke = self.build_command_invoke(cmd.invoke)
self.invoke_without_command = True
def build_command_invoke(self, original_invoke):
def command_invoke(ctx):
""" insert invocation of group function """
# separate the group parameters
ctx.obj = dict(_params=dict())
for param in self.params:
name = param.name
ctx.obj['_params'][name] = ctx.params[name]
del ctx.params[name]
# call the group function with its parameters
params = ctx.params
ctx.params = ctx.obj['_params']
self.invoke(ctx)
ctx.params = params
# now call the original invoke (the command)
original_invoke(ctx)
return command_invoke
Test Code:
#click.group()
#click.pass_context
def group1(ctx):
pass
#group1.command()
#click.argument('argument1')
#click.option('--option1')
def command1(argument1, option1):
click.echo('In command2 %s %s' % (argument1, option1))
#click.group(cls=GroupWithCommandOptions)
#click.option('--optparam', default=None, type=str)
#click.option('--optparam2', default=None, type=str)
#click.pass_context
def group2(ctx, optparam, optparam2):
# create_foo_by_processing_params(optparam, optparam2)
ctx.obj['foo'] = 'from group2 %s %s' % (optparam, optparam2)
#group2.command()
#click.pass_context
def command2a(ctx):
click.echo('command2a foo:%s' % ctx.obj['foo'])
#group2.command()
#click.option('--another-param', default=None, type=str)
#click.pass_context
def command2b(ctx, another_param):
click.echo('command2b %s %s' % (ctx['foo'], another_param))
cli = click.CommandCollection(sources=[group1, group2])
if __name__ == '__main__':
cli('command2a --optparam OP'.split())
Results:
command2a foo:from group2 OP None

This isn't the answer I am looking for, but a step towards it. Essentially a new kind of group is introduced (GroupExt) and the option added to the group is now being added to the command.
$ python cli-test.py command2 --optparam=12
cli
command2 12
import click
class GroupExt(click.Group):
def add_command(self, cmd, name=None):
click.Group.add_command(self, cmd, name=name)
for param in self.params:
cmd.params.append(param)
#click.group()
def group1():
pass
#group1.command()
#click.argument('argument1')
#click.option('--option1')
def command1(argument1, option1):
print 'In command2', argument1, option1
# Equivalent to #click.group() with special group
#click.command(cls=GroupExt)
#click.option('--optparam', default=None, type=str)
def group2():
print 'in group2'
#group2.command()
def command2(optparam):
print 'command2', optparam
#click.command(cls=click.CommandCollection, sources=[group1, group2])
def cli():
print 'cli'
if __name__ == '__main__':
cli(obj={})
This is not quite what I am looking for. Ideally, the optparam would be handled by group2 and the results placed into the context, but currently it's processed in the command2. Perhaps someone knows how to extend this.

How to pass values to python function by executing instructions input as string?

I have a function:
def foo(a=0, b=0, c=0, val=0, someotherval=0):
print val + someotherval
This function is called inside a file bar.py. When I run bar.py from the console, I want to pass the arguments to the function as a string:
>>python bar.py "val=3"
So the function foo interprets it as:
foo(val=3)
I have attempted to use the exec command. In my bar.py file:
import sys
cmdlinearg = sys.argv[1] # capturing commandline argument
foo(exec(cmdlinearg))
But I get a syntax error.
I understand that I can just pass the argument values themselves, but with a function with many arguments, I do not want the end user to enter 0s for unneeded arguments:
>>python bar.py "0" "0" "0" "3"
Is there a way to accomplish this?

I would rather do this the proper way and use argparse.
Your command line interface would look such as:
bar.py --a 0 --b 0 --c 0 --val 0 --someotherval 0
And the code something along:
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument('a', type=int, default=0)
...
args = parser.parse_args()
foo(a=args.a, b=args.b, c=args.c, var=args.val, someotherval=args.someotherval)
if __name__ == '__main__':
main()

If you have no concern for safety, you can just do
exec('foo(%s)' % sys.argv[1])
Or this:
def getdict(**vals):
return vals
kwargs = exec('getdict(%s)' % sys.argv[1])
foo(**kwargs)
However, if your concern is the user's ease of use, maybe you should take a look at argparse.

How about using the argparse for parsing the command line arguments?
Example -
import argparse
def foo(a=0, b=0, c=0, val=0, someotherval=0):
print(val + someotherval)
parser = argparse.ArgumentParser(description='Some Parser')
parser.add_argument('-a','--a',default=0,type=int,help="value for a")
parser.add_argument('-b','--b',default=0,type=int,help="value for a")
parser.add_argument('-c','--c',default=0,type=int,help="value for a")
parser.add_argument('-v','--val',default=0,type=int,help="value for a")
parser.add_argument('-s','--someotherval',default=0,type=int,help="value for a")
args = parser.parse_args()
foo(a=args.a,b=args.b,c=args.c,val=args.val,someotherval=args.someotherval)
Then you can call and get results like -
>python a.py
0
>python a.py --val 10
10
>python a.py --val 10 --someotherval 100
110

Take a look at this library:
http://click.pocoo.org
It may be useful in your case.
import click
#click.command()
#click.option('-a', default=0)
#click.option('-b', default=0)
#click.option('-c', default=0)
#click.option('--val', default=0)
#click.option('--someotherval', default=0)
def foo(a, b, c, val, someotherval):
print val + someotherval
if __name__ == '__main__':
foo()
However, you have to add use options as arguments, not strings:
>> python bar.py --val=3

You could just default the values for the other parameters if you know that they are never going to enter them.
foo(0,0,exec(cmdlinearg),0)

Default sub-command, or handling no sub-command with argparse

How can I have a default sub-command, or handle the case where no sub-command is given using argparse?
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
a.parse_args()
Here I'd like a command to be selected, or the arguments to be handled based only on the next highest level of parser (in this case the top-level parser).
joiner#X:~/src> python3 default_subcommand.py
usage: default_subcommand.py [-h] {hi} ...
default_subcommand.py: error: too few arguments

On Python 3.2 (and 2.7) you will get that error, but not on 3.3 and 3.4 (no response). Therefore on 3.3/3.4 you could test for parsed_args to be an empty Namespace.
A more general solution is to add a method set_default_subparser() (taken from the ruamel.std.argparse package) and call that method just before parse_args():
import argparse
import sys
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if not subparser_found:
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
def do_hi():
print('inside hi')
a = argparse.ArgumentParser()
b = a.add_subparsers()
sp = b.add_parser('hi')
sp.set_defaults(func=do_hi)
a.set_default_subparser('hi')
parsed_args = a.parse_args()
if hasattr(parsed_args, 'func'):
parsed_args.func()
This will work with 2.6 (if argparse is installed from PyPI), 2.7, 3.2, 3.3, 3.4. And allows you to do both
python3 default_subcommand.py
and
python3 default_subcommand.py hi
with the same effect.
Allowing to chose a new subparser for default, instead of one of the existing ones.
The first version of the code allows setting one of the previously-defined subparsers as a default one. The following modification allows adding a new default subparser, which could then be used to specifically process the case when no subparser was selected by user (different lines marked in the code)
def set_default_subparser(self, name, args=None, positional_args=0):
"""default subparser selection. Call after setup, just before parse_args()
name: is the name of the subparser to call by default
args: if set is the argument list handed to parse_args()
, tested with 2.7, 3.2, 3.3, 3.4
it works with 2.6 assuming argparse is installed
"""
subparser_found = False
existing_default = False # check if default parser previously defined
for arg in sys.argv[1:]:
if arg in ['-h', '--help']: # global help if no subparser
break
else:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
for sp_name in x._name_parser_map.keys():
if sp_name in sys.argv[1:]:
subparser_found = True
if sp_name == name: # check existance of default parser
existing_default = True
if not subparser_found:
# If the default subparser is not among the existing ones,
# create a new parser.
# As this is called just before 'parse_args', the default
# parser created here will not pollute the help output.
if not existing_default:
for x in self._subparsers._actions:
if not isinstance(x, argparse._SubParsersAction):
continue
x.add_parser(name)
break # this works OK, but should I check further?
# insert default in last position before global positional
# arguments, this implies no global options are specified after
# first positional argument
if args is None:
sys.argv.insert(len(sys.argv) - positional_args, name)
else:
args.insert(len(args) - positional_args, name)
argparse.ArgumentParser.set_default_subparser = set_default_subparser
a = argparse.ArgumentParser()
b = a.add_subparsers(dest ='cmd')
sp = b.add_parser('hi')
sp2 = b.add_parser('hai')
a.set_default_subparser('hey')
parsed_args = a.parse_args()
print(parsed_args)
The "default" option will still not show up in the help:
python test_parser.py -h
usage: test_parser.py [-h] {hi,hai} ...
positional arguments:
{hi,hai}
optional arguments:
-h, --help show this help message and exit
However, it is now possible to differentiate between and separately handle calling one of the provided subparsers, and calling the default subparser when no argument was provided:
$ python test_parser.py hi
Namespace(cmd='hi')
$ python test_parser.py
Namespace(cmd='hey')

It seems I've stumbled on the solution eventually myself.
If the command is optional, then this makes the command an option. In my original parser configuration, I had a package command that could take a range of possible steps, or it would perform all steps if none was given. This makes the step a choice:
parser = argparse.ArgumentParser()
command_parser = subparsers.add_parser('command')
command_parser.add_argument('--step', choices=['prepare', 'configure', 'compile', 'stage', 'package'])
...other command parsers
parsed_args = parser.parse_args()
if parsed_args.step is None:
do all the steps...

Here's a nicer way of adding a set_default_subparser method:
class DefaultSubcommandArgParse(argparse.ArgumentParser):
__default_subparser = None
def set_default_subparser(self, name):
self.__default_subparser = name
def _parse_known_args(self, arg_strings, *args, **kwargs):
in_args = set(arg_strings)
d_sp = self.__default_subparser
if d_sp is not None and not {'-h', '--help'}.intersection(in_args):
for x in self._subparsers._actions:
subparser_found = (
isinstance(x, argparse._SubParsersAction) and
in_args.intersection(x._name_parser_map.keys())
)
if subparser_found:
break
else:
# insert default in first position, this implies no
# global options without a sub_parsers specified
arg_strings = [d_sp] + arg_strings
return super(DefaultSubcommandArgParse, self)._parse_known_args(
arg_strings, *args, **kwargs
)

Maybe what you're looking for is the dest argument of add_subparsers:
(Warning: works in Python 3.4, but not in 2.7)
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='cmd')
parser_hi = subparsers.add_parser('hi')
parser.parse_args([]) # Namespace(cmd=None)
Now you can just use the value of cmd:
if cmd in [None, 'hi']:
print('command "hi"')

You can duplicate the default action of a specific subparser on the main parser, effectively making it the default.
import argparse
p = argparse.ArgumentParser()
sp = p.add_subparsers()
a = sp.add_parser('a')
a.set_defaults(func=do_a)
b = sp.add_parser('b')
b.set_defaults(func=do_b)
p.set_defaults(func=do_b)
args = p.parse_args()
if args.func:
args.func()
else:
parser.print_help()
Does not work with add_subparsers(required=True), which is why the if args.func is down there.

In my case I found it easiest to explicitly provide the subcommand name to parse_args() when argv was empty.
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='commands')
runParser = subparsers.add_parser('run', help='[DEFAULT ACTION]')
altParser = subparsers.add_parser('alt', help='Alternate command')
altParser.add_argument('alt_val', type=str, help='Value required for alt command.')
# Here's my shortcut: If `argv` only contains the script name,
# manually inject our "default" command.
args = parser.parse_args(['run'] if len(sys.argv) == 1 else None)
print args
Example runs:
$ ./test.py
Namespace()
$ ./test.py alt blah
Namespace(alt_val='blah')
$ ./test.py blah
usage: test.py [-h] {run,alt} ...
test.py: error: invalid choice: 'blah' (choose from 'run', 'alt')

In python 2.7, you can override the error behaviour in a subclass (a shame there isn't a nicer way to differentiate the error):
import argparse
class ExceptionArgParser(argparse.ArgumentParser):
def error(self, message):
if "invalid choice" in message:
# throw exception (of your choice) to catch
raise RuntimeError(message)
else:
# restore normal behaviour
super(ExceptionArgParser, self).error(message)
parser = ExceptionArgParser()
subparsers = parser.add_subparsers(title='Modes', dest='mode')
default_parser = subparsers.add_parser('default')
default_parser.add_argument('a', nargs="+")
other_parser = subparsers.add_parser('other')
other_parser.add_argument('b', nargs="+")
try:
args = parser.parse_args()
except RuntimeError:
args = default_parser.parse_args()
# force the mode into namespace
setattr(args, 'mode', 'default')
print args

Here's another solution using a helper function to build a list of known subcommands:
import argparse
def parse_args(argv):
parser = argparse.ArgumentParser()
commands = []
subparsers = parser.add_subparsers(dest='command')
def add_command(name, *args, **kwargs):
commands.append(name)
return subparsers.add_parser(name, *args, **kwargs)
hi = add_command("hi")
hi.add_argument('--name')
add_command("hola")
# check for default command
if not argv or argv[0] not in commands:
argv.insert(0, "hi")
return parser.parse_args(argv)
assert parse_args([]).command == 'hi'
assert parse_args(['hi']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).command == 'hi'
assert parse_args(['hi', '--name', 'John']).name == 'John'
assert parse_args(['--name', 'John']).command == 'hi'
assert parse_args(['hola']).command == 'hola'

You can add an argument with a default value that will be used when nothing is set I believe.
See this: http://docs.python.org/dev/library/argparse.html#default
Edit:
Sorry, I read your question a bit fast.
I do not think you would have a direct way of doing what you want via argparse. But you could check the length of sys.argv and if its length is 1 (only script name) then you could manually pass the default parameters for parsing, doing something like this:
import argparse
a = argparse.ArgumentParser()
b = a.add_subparsers()
b.add_parser('hi')
if len(sys.argv) == 1:
a.parse_args(['hi'])
else:
a.parse_args()
I think that should do what you want, but I agree it would be nice to have this out of the box.

For later reference:
...
b = a.add_subparsers(dest='cmd')
b.set_defaults(cmd='hey') # <-- this makes hey as default
b.add_parser('hi')
so, these two will be same:
python main.py hey
python main.py