Writing test cases to handle successful and failed python subprocess calls, I need to capture subprocess.call returning code.
Using python unittest.mock module, is it possible to patch the subprocess.call function and capture its real system exit code?
Consider an external library with the following code:
## <somemodule.py> file
import subprocess
def run_hook(cmd):
# ...
subprocess.call(cmd, shell=True)
sum_one = 1+1
return sum_one
I can't modify the function run_hook. It is not part of my code. But, the fact is that subprocess.call is being called among other statements.
Here we have a snippet of code returning a forced system exit error code 1:
## <somemodule.py> tests file
import subprocess
from somemodule import run_hook
try:
from unittest.mock import patch
except ImportError:
from mock import patch
#patch("subprocess.call", side_effect=subprocess.call)
def test_run_hook_systemexit_not_0(call_mock):
python_exec = sys.executable
cmd_parts = [python_exec, "-c", "'exit(1)'"] # Forced error code 1
cmd = " ".join(cmd_parts)
run_hook(cmd)
call_mock.assert_called_once_with(cmd, shell=True)
# I need to change the following assertion to
# catch real return code "1"
assert "1" == call_mock.return_value(), \
"Forced system exit(1) should return 1. Just for example purpose"
How can I improve this test to capture the expected real value for any subprocess.call return code?
For compatibility purposes, new subprocess.run (3.5+) can't be used. This library is still broadly used by python 2.7+ environments.
About subprocess.call, the documentation says:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
All you need to do is to modify your run_hook function and return the exit code:
def run_hook(cmd):
# ...
return subprocess.call(cmd, shell=True)
This will simply your test code.
def test_run_hook_systemexit_not_0():
python_exec = sys.executable
args = [python_exec, "-c", "'exit(1)'"]
assert run_hook(args) == 1
My advice: use subprocess.run instead
Edit
If you want to check the exit code of subprocess.call you need to patch it with your own version, like this:
import subprocess
_original_call = subprocess.call
def assert_call(*args, **kwargs):
assert _original_call(*args, **kwargs) == 0
Then, you use assert_call as a side effect function for your patch:
from unittest.mock import patch
#patch('subprocess.call', side_effect=assert_call)
def test(call):
python_exec = sys.executable
args = [python_exec, "-c", "'exit(1)'"]
run_hook(args)
A wrapper around subprocess.call can handle the assertion verification.
In this case, I declare this wrapper as the side_effect argument in #patch definition.
In this case, the following implementation worked well.
import sys
import unittest
try:
from unittest.mock import patch
except ImportError:
from mock import patch
def subprocess_call_assert_wrap(expected, message=None):
from subprocess import call as _subcall
def _wrapped(*args, **kwargs):
if message:
assert expected == _subcall(*args, **kwargs), message
else:
assert expected == _subcall(*args, **kwargs)
return _wrapped
class TestCallIsBeingCalled(unittest.TestCase):
#patch("subprocess.call", side_effect=subprocess_call_assert_wrap(expected=0))
def test_run_hook_systemexit_0(self, call_mock):
python_exec = sys.executable
cmd_parts = [python_exec, "-c", "'exit(0)'"]
cmd = " ".join(cmd_parts)
run_hook(cmd)
call_mock.assert_called_once_with(cmd, shell=True)
#patch("subprocess.call", side_effect=subprocess_call_assert_wrap(expected=1))
def test_run_hook_systemexit_not_0(self, call_mock):
python_exec = sys.executable
cmd_parts = [python_exec, "-c", "'exit(1)'"]
cmd = " ".join(cmd_parts)
run_hook(cmd)
call_mock.assert_called_once_with(cmd, shell=True)
After some tests with taking this approach, it seems possible to use for a more general purpose calls, like:
def assert_wrapper(expected, callable, message=None):
def _wrapped(*args, **kwargs):
if message:
assert expected == callable(*args, **kwargs), message
else:
assert expected == callable(*args, **kwargs)
return _wrapped
This is not the best approach, but it seems reasonable.
There is some best known lib with similar behavior that I can use in this project?
Related
The following is a test. How to make this test fail if the subprocess.run results in an error?
import pytest
import subprocess
#pytest.mark.parametrize("input_json", ["input.json"])
def test_main(input_json):
subprocess.run(['python', 'main.py', input_json]
subprocess.run returns a CompletedProcess instance, which you can inspect.
I'm not sure what exactly you mean by "results in an error"—if it's the return code of the process being non-zero, you can check returncode.
If it's a specific error message being output, check stdout or stderr instead.
For example:
import pytest
import subprocess
#pytest.mark.parametrize("input_json", ["input.json"])
def test_main(input_json):
completed_process = subprocess.run(['python', 'main.py', 'input_json']
assert completed_process.returncode == 0
NB. You don't seem to use the parametrized argument input_json within your test function. Just an observation.
I have some code that parses command line options using argparse.
For example:
# mycode.py
import argparse
def parse_args():
parser = argparse.ArgumentParser('my code')
# list of arguments
# ...
# ...
return vars(parser.parse_args())
if __name__ == "__main__":
parse_args()
I would like to use unittest to check the output of the help function. I also don't want to change the actual code unless there is no other solution.
The help action has a SystemExit call built into it after printing to stdout, so I have had to try and catch it in the unittest.
Here is my unittest code with the following steps:
1) Set the sys.argv list to include the -h flag.
2) Wrap the function call in a context manager to prevent the SystemExit being viewed as an error.
3) Switch the sys.stdout temporarily to an io.StringIO object so I can inspect it without having it print to screen.
4) Call the function in a try...finally block so the SystemExit isn't fatal.
5) Switch sys.stdout back to the real stdout.
6) Open a file to which I had previously saved the help text (by entering python mycode.py -h > help_out.txt in the terminal) to verify it is the same as the captured output from the StringIO.
import unittest
import mycode
import sys
import io
class TestParams(unittest.TestCase):
def setUp(self):
pass
def test_help(self):
args = ["-h"]
sys.argv[1:] = args
with self.assertRaises(SystemExit):
captured_output = io.StringIO()
sys.stdout = captured_output
try:
mycode.parse_args()
finally:
sys.stdout = sys.__stdout__
with open("help_out.txt", "r") as f:
help_text = f.read()
self.assertEqual(captured_output, help_text)
def tearDown(self):
pass
This code works, but the captured_output StringIO object is empty, so the test fails.
I am looking for an explanation as to what is going wrong with the captured output and/or an alternative solution.
I was very close. The captured_output wasn't actually empty - I just wasn't accessing the contents correctly.
Substitute captured_output.get_value() for captured_value in my example code and it works perfectly.
I've been reading the Python documentation about the subprocess module (see here) and it talks about a subprocess.check_output() command which seems to be exactly what I need.
However, when I try and use it I get an error that it doesn't exist, and when I run dir(subprocess) it is not listed.
I am running Python 2.6.5, and the code I have used is below:
import subprocess
subprocess.check_output(["ls", "-l", "/dev/null"])
Does anyone have any idea why this is happening?
It was introduced in 2.7 See the docs.
Use subprocess.Popen if you want the output:
>>> import subprocess
>>> output = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE).communicate()[0]
IF it's used heavily in the code you want to run but that code doesn't have to be maintained long-term (or you need a quick fix irrespective of potential maintenance headaches in the future) then you could duck punch (aka monkey patch) it in wherever subprocess is imported...
Just lift the code from 2.7 and insert it thusly...
import subprocess
if "check_output" not in dir( subprocess ): # duck punch it in!
def f(*popenargs, **kwargs):
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd)
return output
subprocess.check_output = f
Minor fidgeting may be required.
Do bear in mind though the onus is on you to maintain dirty little backports like this. If bugs are discovered and corrected in the latest python then you a) have to notice that and b) update your version if you want to stay secure. Also, overriding & defining internal functions yourself is the next guy's worst nightmare, especially when the next guy is YOU several years down the line and you've forgot all about the grody hacks you did last time! In summary: it's very rarely a good idea.
Thanks to the monkey patch suggestion (and my attempts failing - but we were consuming CalledProcessError output, so needed to monkey patch that)
found a working 2.6 patch here:
http://pydoc.net/Python/pep8radius/0.9.0/pep8radius.shell/
"""Note: We also monkey-patch subprocess for python 2.6 to
give feature parity with later versions.
"""
try:
from subprocess import STDOUT, check_output, CalledProcessError
except ImportError: # pragma: no cover
# python 2.6 doesn't include check_output
# monkey patch it in!
import subprocess
STDOUT = subprocess.STDOUT
def check_output(*popenargs, **kwargs):
if 'stdout' in kwargs: # pragma: no cover
raise ValueError('stdout argument not allowed, '
'it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE,
*popenargs, **kwargs)
output, _ = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd,
output=output)
return output
subprocess.check_output = check_output
# overwrite CalledProcessError due to `output`
# keyword not being available (in 2.6)
class CalledProcessError(Exception):
def __init__(self, returncode, cmd, output=None):
self.returncode = returncode
self.cmd = cmd
self.output = output
def __str__(self):
return "Command '%s' returned non-zero exit status %d" % (
self.cmd, self.returncode)
subprocess.CalledProcessError = CalledProcessError
I have a method - run_script() - I would like to test. Specifically, I want to test that a call to subprocess.Popen occurs. It would be even better to test that subprocess.Popen is called with certain parameters. When I run the test however I get TypeError: 'tuple' object is not callable.
How can I test my method to ensure that subprocess is actually being called using mocks?
#mock.patch("subprocess.Popen")
def run_script(file_path):
process = subprocess.Popen(["myscript", -M, file_path], stdout=subprocess.PIPE)
output, err = process.communicate()
return process.returncode
def test_run_script(self, mock_subproc_popen):
mock_subproc_popen.return_value = mock.Mock(
communicate=("ouput", "error"), returncode=0
)
am.account_manager("path")
self.assertTrue(mock_subproc_popen.called)
It seems unusual to me that you use the patch decorator over the run_script function, since you don't pass a mock argument there.
How about this:
from unittest import mock
import subprocess
def run_script(file_path):
process = subprocess.Popen(["myscript", -M, file_path], stdout=subprocess.PIPE)
output, err = process.communicate()
return process.returncode
#mock.patch("subprocess.Popen")
def test_run_script(self, mock_subproc_popen):
process_mock = mock.Mock()
attrs = {"communicate.return_value": ("output", "error")}
process_mock.configure_mock(**attrs)
mock_subproc_popen.return_value = process_mock
am.account_manager("path") # this calls run_script somewhere, is that right?
self.assertTrue(mock_subproc_popen.called)
Right now, your mocked subprocess.Popen seems to return a tuple, causeing process.communicate() to raise TypeError: 'tuple' object is not callable.. Therefore it's most important to get the return_value on mock_subproc_popen just right.
The testfixtures library (docs, github) can mock the subprocess package.
Here's an example on using the mock subprocess.Popen:
from unittest import TestCase
from testfixtures.mock import call
from testfixtures import Replacer, ShouldRaise, compare
from testfixtures.popen import MockPopen, PopenBehaviour
class TestMyFunc(TestCase):
def setUp(self):
self.Popen = MockPopen()
self.r = Replacer()
self.r.replace(dotted_path, self.Popen)
self.addCleanup(self.r.restore)
def test_example(self):
# set up
self.Popen.set_command("svn ls -R foo", stdout=b"o", stderr=b"e")
# testing of results
compare(my_func(), b"o")
# testing calls were in the right order and with the correct parameters:
process = call.Popen(["svn", "ls", "-R", "foo"], stderr=PIPE, stdout=PIPE)
compare(Popen.all_calls, expected=[process, process.communicate()])
def test_example_bad_returncode(self):
# set up
Popen.set_command("svn ls -R foo", stdout=b"o", stderr=b"e", returncode=1)
# testing of error
with ShouldRaise(RuntimeError("something bad happened")):
my_func()
If you want to check that the mocked object was called with a certain parameter, you can add the side_effect argument to the mock.patch decorator.
The return value of the side_effect function determines the return value of subprocess.Popen. If the side_effect_func returns DEFAULT, subprocess.Popen will be called in a normal way.
from unittest import mock, TestCase
from unittest.mock import DEFAULT
import subprocess
def run_script(script_path, my_arg):
process = subprocess.Popen([script_path, my_arg])
return process
def side_effect_func(*args, **kwargs):
# Print the arguments
print(args)
# If 'bar' is contained within the arguments, return 'foo'
if any(['bar' in arg for arg in args]):
return 'foo'
# If 'bar' is not contained within the arguments, run subprocess.Popen
else:
return DEFAULT
class TestRunScriptClass(TestCase):
#mock.patch("subprocess.Popen", side_effect=side_effect_func)
def test_run_script(self, mock):
# Run the function
process = run_script(script_path='my_script.py', my_arg='bar')
# Assert if the mock object was called
self.assertTrue(mock.called)
# Assert if the mock object returned 'foo' when providing 'bar'
self.assertEqual(process, 'foo')
There is no need for anything complex. You can simply move the call to subprocess to a function and pass a mock function in your test.
import subprocess
import unittest
import logging
from unittest.mock import Mock
def _exec(cmd):
p = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
stdout, stderr = p.communicate()
return p, stdout, stderr
def uses_exec(execute=_exec):
cmd = ['ls']
p, stdout, stderr = execute(cmd)
if p.returncode != 0:
logging.error(
'%s returned Error Code: %s',
cmd,
p.returncode
)
logging.error(stderr)
else:
logging.info(stdout)
class TestUsesExecute(unittest.TestCase):
def test_get_aws_creds_from_namespace(self):
p = lambda:None
p.returncode = 0
stdout = 'file1 file2'
mock_execute = Mock(return_value=(p, stdout, ''))
uses_exec(execute=mock_execute)
mock_execute.assert_called_with(['ls'])
if __name__ == '__main__':
unittest.main()
This pattern works in all programming languages that allow functions to be passed as parameters. It can be used with other python system calls for file and network reads just as easily.
I used this for a test suite where there were many subprocess.run calls to check.
import contextlib
import re
import subprocess
from typing import NamedTuple
from unittest.mock import MagicMock
class CmdMatch(NamedTuple):
cmd: str
match: str = ".*"
result: str = ""
side_effect: Callable = None
#contextlib.contextmanager
def mock_run(*cmd_match: Union[str, CmdMatch], **kws):
sub_run = subprocess.run
mock = MagicMock()
if isinstance(cmd_match[0], str):
cmd_match = [CmdMatch(*cmd_match, **kws)]
def new_run(cmd, **_kws):
check_cmd = " ".join(cmd[1:])
mock(*cmd[1:])
for m in cmd_match:
if m.cmd in cmd[0].lower() and re.match(m.match, check_cmd):
if m.side_effect:
m.side_effect()
return subprocess.CompletedProcess(cmd, 0, m.result, "")
assert False, "No matching call for %s" % check_cmd
subprocess.run = new_run
yield mock
subprocess.run = sub_run
So now you can write a test like this:
def test_call_git():
with mock_run("git", "describe.*", "v2.5") as cmd:
do_something()
cmd.assert_called_once()
def test_other_calls():
with mock_run(
CmdMatch("git", "describe.*", "v2.5"),
CmdMatch("aws", "s3.*links.*", side_effect=subprocess.CalledProcessError),
) as cmd:
do_something()
Couple funky things you might want to change, but I liked them:
the resulting mock calls ignore the first argument (so which/full-paths aren't part of the tests)
if nothing matches any calls, it asserts (you have to have a match)
the first positional is considered the "name of the command"
kwargs to run are ignored (_call assertions are easy but loose)
I've been reading the Python documentation about the subprocess module (see here) and it talks about a subprocess.check_output() command which seems to be exactly what I need.
However, when I try and use it I get an error that it doesn't exist, and when I run dir(subprocess) it is not listed.
I am running Python 2.6.5, and the code I have used is below:
import subprocess
subprocess.check_output(["ls", "-l", "/dev/null"])
Does anyone have any idea why this is happening?
It was introduced in 2.7 See the docs.
Use subprocess.Popen if you want the output:
>>> import subprocess
>>> output = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE).communicate()[0]
IF it's used heavily in the code you want to run but that code doesn't have to be maintained long-term (or you need a quick fix irrespective of potential maintenance headaches in the future) then you could duck punch (aka monkey patch) it in wherever subprocess is imported...
Just lift the code from 2.7 and insert it thusly...
import subprocess
if "check_output" not in dir( subprocess ): # duck punch it in!
def f(*popenargs, **kwargs):
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd)
return output
subprocess.check_output = f
Minor fidgeting may be required.
Do bear in mind though the onus is on you to maintain dirty little backports like this. If bugs are discovered and corrected in the latest python then you a) have to notice that and b) update your version if you want to stay secure. Also, overriding & defining internal functions yourself is the next guy's worst nightmare, especially when the next guy is YOU several years down the line and you've forgot all about the grody hacks you did last time! In summary: it's very rarely a good idea.
Thanks to the monkey patch suggestion (and my attempts failing - but we were consuming CalledProcessError output, so needed to monkey patch that)
found a working 2.6 patch here:
http://pydoc.net/Python/pep8radius/0.9.0/pep8radius.shell/
"""Note: We also monkey-patch subprocess for python 2.6 to
give feature parity with later versions.
"""
try:
from subprocess import STDOUT, check_output, CalledProcessError
except ImportError: # pragma: no cover
# python 2.6 doesn't include check_output
# monkey patch it in!
import subprocess
STDOUT = subprocess.STDOUT
def check_output(*popenargs, **kwargs):
if 'stdout' in kwargs: # pragma: no cover
raise ValueError('stdout argument not allowed, '
'it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE,
*popenargs, **kwargs)
output, _ = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd,
output=output)
return output
subprocess.check_output = check_output
# overwrite CalledProcessError due to `output`
# keyword not being available (in 2.6)
class CalledProcessError(Exception):
def __init__(self, returncode, cmd, output=None):
self.returncode = returncode
self.cmd = cmd
self.output = output
def __str__(self):
return "Command '%s' returned non-zero exit status %d" % (
self.cmd, self.returncode)
subprocess.CalledProcessError = CalledProcessError