I'm trying to mock subprocess.Popen. When I run the following code however, the mock is completely ignored and I'm not sure why
Test Code:
def test_bring_connection_up(self):
# All settings should either overload the update or the run method
mock_popen = MagicMock()
mock_popen.return_value = {'communicate': (lambda: 'hello','world')}
with patch('subprocess.Popen', mock_popen):
self.assertEqual(network_manager.bring_connection_up("test"), "Error: Unknown connection: test.\n")
Module Code:
from subprocess import Popen, PIPE
# ........
def list_connections():
process = Popen(["nmcli", "-t", "-fields", "NAME,TYPE", "con", "list"], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate() # <--- Here's the failure
return stdout
You're not patching in the right place. You patch where Popen is defined:
with patch('subprocess.Popen', mock_popen):
You need to patch where Popen is imported, i.e. in the "Module Code" where you have written this line:
from subprocess import Popen, PIPE
I.e., it should look something like:
with patch('myapp.mymodule.Popen', mock_popen):
For a quick guide, read the section in the docs: Where to patch.
Related
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 4 years ago.
How can I get the output of a process run using subprocess.call()?
Passing a StringIO.StringIO object to stdout gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 588, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 945, in _get_handles
c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
>>>
If you have Python version >= 2.7, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as string).
Simple example (linux version, see note):
import subprocess
print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])
Note that the ping command is using linux notation (-c for count). If you try this on Windows remember to change it to -n for same result.
As commented below you can find a more detailed explanation in this other answer.
Output from subprocess.call() should only be redirected to files.
You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.
See here for more info.
For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.
from subprocess import PIPE, run
command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)
I have the following solution. It captures the exit code, the stdout, and the stderr too of the executed external command:
import shlex
from subprocess import Popen, PIPE
def get_exitcode_stdout_stderr(cmd):
"""
Execute the external command and get its exitcode, stdout and stderr.
"""
args = shlex.split(cmd)
proc = Popen(args, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
exitcode = proc.returncode
#
return exitcode, out, err
cmd = "..." # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)
I also have a blog post on it here.
Edit: the solution was updated to a newer one that doesn't need to write to temp. files.
I recently just figured out how to do this, and here's some example code from a current project of mine:
#Getting the random picture.
#First find all pictures:
import shlex, subprocess
cmd = 'find ../Pictures/ -regex ".*\(JPG\|NEF\|jpg\)" '
#cmd = raw_input("shell:")
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
#Another way to get output
#output = subprocess.Popen(args,stdout = subprocess.PIPE).stdout
ber = raw_input("search complete, display results?")
print output
#... and on to the selection process ...
You now have the output of the command stored in the variable "output". "stdout = subprocess.PIPE" tells the class to create a file object named 'stdout' from within Popen. The communicate() method, from what I can tell, just acts as a convenient way to return a tuple of the output and the errors from the process you've run. Also, the process is run when instantiating Popen.
The key is to use the function subprocess.check_output
For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:
from subprocess import check_output, CalledProcessError, STDOUT
def system_call(command):
"""
params:
command: list of strings, ex. `["ls", "-l"]`
returns: output, success
"""
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call(["ls", "-l"])
If you want to pass commands as strings rather than arrays, use this version:
from subprocess import check_output, CalledProcessError, STDOUT
import shlex
def system_call(command):
"""
params:
command: string, ex. `"ls -l"`
returns: output, success
"""
command = shlex.split(command)
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call("ls -l")
In Ipython shell:
In [8]: import subprocess
In [9]: s=subprocess.check_output(["echo", "Hello World!"])
In [10]: s
Out[10]: 'Hello World!\n'
Based on sargue's answer. Credit to sargue.
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 4 years ago.
How can I get the output of a process run using subprocess.call()?
Passing a StringIO.StringIO object to stdout gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 588, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 945, in _get_handles
c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
>>>
If you have Python version >= 2.7, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as string).
Simple example (linux version, see note):
import subprocess
print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])
Note that the ping command is using linux notation (-c for count). If you try this on Windows remember to change it to -n for same result.
As commented below you can find a more detailed explanation in this other answer.
Output from subprocess.call() should only be redirected to files.
You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.
See here for more info.
For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.
from subprocess import PIPE, run
command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)
I have the following solution. It captures the exit code, the stdout, and the stderr too of the executed external command:
import shlex
from subprocess import Popen, PIPE
def get_exitcode_stdout_stderr(cmd):
"""
Execute the external command and get its exitcode, stdout and stderr.
"""
args = shlex.split(cmd)
proc = Popen(args, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
exitcode = proc.returncode
#
return exitcode, out, err
cmd = "..." # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)
I also have a blog post on it here.
Edit: the solution was updated to a newer one that doesn't need to write to temp. files.
I recently just figured out how to do this, and here's some example code from a current project of mine:
#Getting the random picture.
#First find all pictures:
import shlex, subprocess
cmd = 'find ../Pictures/ -regex ".*\(JPG\|NEF\|jpg\)" '
#cmd = raw_input("shell:")
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
#Another way to get output
#output = subprocess.Popen(args,stdout = subprocess.PIPE).stdout
ber = raw_input("search complete, display results?")
print output
#... and on to the selection process ...
You now have the output of the command stored in the variable "output". "stdout = subprocess.PIPE" tells the class to create a file object named 'stdout' from within Popen. The communicate() method, from what I can tell, just acts as a convenient way to return a tuple of the output and the errors from the process you've run. Also, the process is run when instantiating Popen.
The key is to use the function subprocess.check_output
For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:
from subprocess import check_output, CalledProcessError, STDOUT
def system_call(command):
"""
params:
command: list of strings, ex. `["ls", "-l"]`
returns: output, success
"""
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call(["ls", "-l"])
If you want to pass commands as strings rather than arrays, use this version:
from subprocess import check_output, CalledProcessError, STDOUT
import shlex
def system_call(command):
"""
params:
command: string, ex. `"ls -l"`
returns: output, success
"""
command = shlex.split(command)
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call("ls -l")
In Ipython shell:
In [8]: import subprocess
In [9]: s=subprocess.check_output(["echo", "Hello World!"])
In [10]: s
Out[10]: 'Hello World!\n'
Based on sargue's answer. Credit to sargue.
I'm trying to mock the subprocess module in my application
from subprocess import PIPE
import sys
from mock import MagicMock
from django.test import TestCase
def mockNMCLI(command_list, stdout, stderr):
return {'communicate': (lambda: ("hello", ""))}
def mockPipe():
return PIPE
sys.modules['subprocess'] = MagicMock()
sys.modules['subprocess.Popen'] = mockNMCLI
sys.modules['subprocess.PIPE'] = mockPipe
from robot_configuration_interface.helpers import network_manager # NOQA
class NetworkManagerTestCase(TestCase):
def test_list_available_networks(self):
self.assertIsInstance(network_manager.list_connections(), (list))
This is the pertinant function and the imports from the module being tested.
from subprocess import Popen, PIPE
# ........
def list_connections():
process = Popen(["nmcli", "-t", "-fields", "NAME,TYPE", "con", "list"], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate() # <--- Here's the failure
return stdout
However, this is failing with:
ValueError: need more than 0 values to unpack in process.communicate()
This clearly because Popen is not being mocked correctly. But why wouldn't it be.
You're not using the package properly, in several ways: don't patch by direct assignment in sys.modules, because that patch will not get undone afterwards and could break all sorts of things.
First of all, make a mock correctly, so that it knows how to *unpack:
>>> mock_popen = MagicMock()
>>> mock_popen.communicate.return_value = 'hello', 'world'
>>> stdout, stderr = mock_popen.communicate()
>>> stdout, stderr
('hello', 'world')
Then patch with a context-manager, something like this:
with mock.patch('path.where.you.import.Popen', mock_popen):
the_code_that_uses_popen()
mock_popen.assert_called_with(...)
I'm trying to mock subprocess.Popen. When I run the following code however, the mock is completely ignored and I'm not sure why
Test Code:
def test_bring_connection_up(self):
# All settings should either overload the update or the run method
mock_popen = MagicMock()
mock_popen.return_value = {'communicate': (lambda: 'hello','world')}
with patch('subprocess.Popen', mock_popen):
self.assertEqual(network_manager.bring_connection_up("test"), "Error: Unknown connection: test.\n")
Module Code:
from subprocess import Popen, PIPE
# ........
def list_connections():
process = Popen(["nmcli", "-t", "-fields", "NAME,TYPE", "con", "list"], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate() # <--- Here's the failure
return stdout
You're not patching in the right place. You patch where Popen is defined:
with patch('subprocess.Popen', mock_popen):
You need to patch where Popen is imported, i.e. in the "Module Code" where you have written this line:
from subprocess import Popen, PIPE
I.e., it should look something like:
with patch('myapp.mymodule.Popen', mock_popen):
For a quick guide, read the section in the docs: Where to patch.
I'm trying to use LocalPath.sysexec() from the py library, but the documentation is not clear enough for me to understand how to invoke it with the right syntax.
Here is some fake code that mimics my syntax:
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Testing pylib implementation of subprocess.Popen
from py.path import local
import sys
path = local(sys.argv[1])
path.sysexec("/usr/bin/foo", ["arg1", "arg2"])
You can look the source code:
def sysexec(self, *argv, **popen_opts):
""" return stdout text from executing a system child process,
where the 'self' path points to executable.
The process is directly invoked and not through a system shell.
"""
from subprocess import Popen, PIPE
argv = map_as_list(str, argv)
popen_opts['stdout'] = popen_opts['stderr'] = PIPE
proc = Popen([str(self)] + argv, **popen_opts)
stdout, stderr = proc.communicate()
ret = proc.wait()
if py.builtin._isbytes(stdout):
stdout = py.builtin._totext(stdout, sys.getdefaultencoding())
if ret != 0:
if py.builtin._isbytes(stderr):
stderr = py.builtin._totext(stderr, sys.getdefaultencoding())
raise py.process.cmdexec.Error(ret, ret, str(self),
stdout, stderr,)
return stdout
clearly, It use the python subprocess module. If you have not been used subprocess, you can click the above link to raed the docs.
In this function, It use subprocess construct a Popen object, and use the argv you pass.
Then call the Popen.wait(), block until the command execute finish. And return the stdout of the command.
Example:
local_path = py._path.local.LocalPath('/usr/bin/ls')
print(local_path.sysexec())
# out: file1\nfile2\nfile3...
print(local_path.sysexec('-l'))
# out likes "ls -l" out