I want to mock a function present inside a list and check whether it has been called at least once. Below is a similar implementation I tried:-
In fun_list.py (funA and funB are two functions in other_module)
import other_module
FUN_LIST = [
other_module.funA,
other_module.funB,
]
def run_funs():
for fun in FUN_LIST:
fun()
In demo.py
from fun_list import run_funs
def run_demo():
...
run_funs()
...
In test_demo.py
from demo import run_demo
#patch('other_module.funB')
def test_demo_funs(mocked_funB):
mocked_funB.return_value = {}
run_demo()
assert mocked_funB.called
In above case, I'm trying to mock funB in other_module but the function doesn't get mocked and the cursor gets inside the actual funB in other_module. Thus, the assert mocked_funB.called returns false.
Any lead on how I can mock other_module.funB ?
I have found a similar question on StackOverflow but that went unanswered, so decided to post my version of it.
Any help will be appreciated, thank you in advance.
You need to mock before importing the module under test. The code in the module scope will be executed when import the module. It is too late to mock through the decorator when the test case is executed.
E.g.
other_module.py:
def funA():
pass
def funB():
pass
fun_list.py:
import other_module
print('execute module scope code')
FUN_LIST = [
other_module.funA,
other_module.funB,
]
def run_funs():
for fun in FUN_LIST:
fun()
demo.py:
from fun_list import run_funs
def run_demo():
run_funs()
test_demo.py:
import unittest
from unittest.mock import patch
class TestDemo(unittest.TestCase):
#patch('other_module.funB')
def test_demo_funs(self, mocked_funB):
print('mock before import the module')
from demo import run_demo
mocked_funB.return_value = {}
run_demo()
assert mocked_funB.called
if __name__ == '__main__':
unittest.main()
test result:
mock before import the module
execute module scope code
.
----------------------------------------------------------------------
Ran 1 test in 0.002s
OK
Name Stmts Miss Cover Missing
--------------------------------------------------------------------------
src/stackoverflow/67563601/demo.py 3 0 100%
src/stackoverflow/67563601/fun_list.py 6 0 100%
src/stackoverflow/67563601/other_module.py 4 1 75% 6
src/stackoverflow/67563601/test_demo.py 12 0 100%
--------------------------------------------------------------------------
TOTAL 25 1 96%
I took the lead from #slideshowp2's answer and have modified things a bit differently. In my case I was having multiple such test functions that is mocking funB and calling run_demo (originally a client.post() call from Django.test). If an earlier function calls it successfully, the other subsequent patches were failing (because of the same reason stated by #slideshowp2). So, I changed the approach to this :-
In fun_list.py (funA and funB are two functions in other_module)
import other_module
FUN_LIST = [
'funA',
'funB',
]
def run_funs():
for fun in FUN_LIST:
getattr(other_module, fun)()
In demo.py
from fun_list import run_funs
def run_demo():
...
run_funs()
...
In test_demo.py
from demo import run_demo
#patch('other_module.funB')
def test_demo_funs(mocked_funB):
mocked_funB.return_value = {}
run_demo()
assert mocked_funB.called
Related
Let's say that I have a package qaz:
qaz/
__init__.py
qaz.py
tests/
test_qaz.py
setup.py
Now, I want to test some internal qaz.py functions:
def _abc():
return 3
def def():
return _abc() + 2
but when I run pytest with tests like this:
from qaz.qaz import *
def test_abc():
assert _abc() == 3
def test_def():
assert def() == 5
My question is how to test _abc()?
Now I'm getting:
E NameError: name '_abc' is not defined
Oh, figured it out. You need to be specific, when it comes to the internal functions:
from qaz.qaz import _abc
I'm simply trying to import a function from another script. But despite import running successfully, the functions never enter my local environment.
The paths and files look like this:
project/__main__.py
project/script_a.py
from setup import script_b
x = ABC() # NameError: name 'ABC' is not defined
print(x)
project/setup/__init__.py
project/setup/script_b.py
def ABC():
return "ABC"
I've done this before and the documentation (officials and on here) is quite straightforward but I cannot grasp what I am failing to understand. Is the function running but never entering my environment?
I also tried using...
if __name__ == '__main__':
def ABC():
return "ABC"
...as script_b.
Import the functions inside the module:
from setup.script_b import ABC
Or call the function on the modules name like said in the comments
x = script_b.ABC()
I am having what I believe to be a common issue in using mock patching in that I can not figure out the right thing to patch.
I have two questions that I am hoping for help with.
Thoughts on how to fix the specific issue in the below example
And possibly most-importantly pro-tips/pointers/thoughts/suggestions on how to best troubleshoot the "which thing do I patch" question. The problem I'm having is, without a full understanding of how patching works, I really dont even know what I should be looking for and find myself playing a guessing game.
An example using pyarrow that is currently causing me pain:
mymodule.py
import pyarrow
class HdfsSearch:
def __init__(self):
self.fs = self._connect()
def _connect(self) -> object:
return pyarrow.hdfs.connect(driver="libhdfs")
def search(self, path: str):
return self.fs.ls(path=path)
test_module.py
import pyarrow
import pytest
from mymodule import HdfsSearch
#pytest.fixture()
def hdfs_connection_fixture(mocker):
mocker.patch("pyarrow.hdfs.connect")
yield HdfsSearch()
def test_hdfs_connection(hdfs_connection_fixture):
pyarrow.hdfs.connect.assert_called_once() # <-- succeeds
def test_hdfs_search(hdfs_connection_fixture):
hdfs_connection_fixture.search(".")
pyarrow.hdfs.HadoopFileSystem.ls.assert_called_once() # <-- fails
pytest output:
$ python -m pytest --verbose test_module.py
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/bbaur/miniconda3/envs/dev/bin/python
cachedir: .pytest_cache
rootdir: /home/user1/work/app
plugins: cov-2.7.1, mock-1.10.4
collected 2 items
test_module.py::test_hdfs_connection PASSED [ 50%]
test_module.py::test_hdfs_search FAILED [100%]
================================================================================================================= FAILURES =================================================================================================================
_____________________________________________________________________________________________________________ test_hdfs_search _____________________________________________________________________________________________________________
hdfs_connection_fixture = <mymodule.HdfsSearch object at 0x7fdb4ec2a610>
def test_hdfs_search(hdfs_connection_fixture):
hdfs_connection_fixture.search(".")
> pyarrow.hdfs.HadoopFileSystem.ls.assert_called_once()
E AttributeError: 'function' object has no attribute 'assert_called_once'
test_module.py:16: AttributeError
You're not calling the assert on the Mock object, this is the correct assert:
hdfs_connection_fixture.fs.ls.assert_called_once()
Explanation:
When you access any attribute in a Mock object it will return another Mock object.
Since you patched "pyarrow.hdfs.connect" you've replaced it with a Mock, let's call it Mock A. Your _connect method will return that Mock A and you'll assign it to self.fs.
Now let's break down what's happening in the search method when you call self.fs.ls.
self.fs returns your Mock A object, then the .ls will return a different Mock object, let's call it Mock B. In this Mock B object you're doing the call passing (path=path).
In your assert you're trying to access pyarrow.hdfs.HadoopFileSystem, but it was never patched. You'll need do the assert on the Mock B object, which is at hdfs_connection_fixture.fs.ls
What to Patch
If you change your import in mymodule.py to this from pyarrow.hdfs import connect your patch will stop working.
Why is that?
When you patch something you're changing what a name points to, not the actual object.
Your current patch is patching the name pyarrow.hdfs.connect and in mymodule you're using the same name pyarrow.hdfs.connect so everything is fine.
However, if you use from pyarrow.hdfs import connect mymodule will have imported the real pyarrow.hdfs.connect and created a reference for it with the name mymodule.connect.
So when you call connect inside mymodule you're accessing the name mymodule.connect, which is not patched.
That is why you would need to patch mymodule.connect when using from import.
I'd recommend using from x import y when doing this kind of patching. It makes it more explicit what you're trying to mock and the patch will be limited to that module only, which can prevent unforeseen side-effects.
Source, this section in the Python documentation: Where to patch
To understand how patching works in python let's first understand the import statement.
When we use import pyarrow in a module (mymodule.py in this case) it does two operations :
It searches for the pyarrow module in sys.modules
It binds the results of that search to a name(pyarrow) in the local scope.
By doing something like: pyarrow = sys.modules['pyarrow']
NOTE: import statements in python doesn't execute code. The import statement brings a name into local scope. The execution of code happens as a side-effect only when python can't find a module in sys.modules
So, to patch pyarrow imported in mymodule.py we need to patch the pyarrow name present in the local scope of mymodule.py
patch('mymodule.pyarrow', autospec=True)
test_module.py
import pytest
from mock import Mock, sentinel
from pyarrow import hdfs
from mymodule import HdfsSearch
class TestHdfsSearch(object):
#pytest.fixture(autouse=True, scope='function')
def setup(self, mocker):
self.hdfs_mock = Mock(name='HadoopFileSystem', spec=hdfs.HadoopFileSystem)
self.connect_mock = mocker.patch("mymodule.pyarrow.hdfs.connect", return_value=self.hdfs_mock)
def test_initialize_HdfsSearch_should_connect_pyarrow_hdfs_file_system(self):
HdfsSearch()
self.connect_mock.assert_called_once_with(driver="libhdfs")
def test_initialize_HdfsSearch_should_set_pyarrow_hdfs_as_file_system(self):
hdfs_search = HdfsSearch()
assert self.hdfs_mock == hdfs_search.fs
def test_search_should_retrieve_directory_contents(self):
hdfs_search = HdfsSearch()
self.hdfs_mock.ls.return_value = sentinel.contents
result = hdfs_search.search(".")
self.hdfs_mock.ls.assert_called_once_with(path=".")
assert sentinel.contents == result
Use context managers to patch built-ins
def test_patch_built_ins():
with patch('os.curdir') as curdir_mock: # curdir_mock lives only inside with block. Doesn't lives outside
assert curdir_mock == os.curdir
assert os.curdir == '.'
I have a module called learning that uses random.uniform(). I have a file called test_learning.py containing unit tests. When I run a unit test, I would like the code in learning to see the patched version of random.uniform(). How can I do this? Here is what I have currently.
import random
import unittest
import unittest.mock as mock
class TestLearning(unittest.TestCase):
def test_get_random_belief_bit(self):
with mock.patch('learning.random.uniform', mock_uniform):
bit = learning.get_random_belief_bit(0.4)
self.assertEqual(bit, 0)
But the test (sometimes) fails because learning.get_random_belief_bit() seems to be using the real random.uniform().
Unit test solution:
learning.py:
import random
def get_random_belief_bit(f):
return random.uniform()
test_learning.py:
import random
import unittest
import unittest.mock as mock
import learning
class TestLearning(unittest.TestCase):
def test_get_random_belief_bit(self):
with mock.patch('random.uniform', mock.Mock()) as mock_uniform:
mock_uniform.return_value = 0
bit = learning.get_random_belief_bit(0.4)
self.assertEqual(bit, 0)
mock_uniform.assert_called_once()
if __name__ == '__main__':
unittest.main()
unit test result with coverage report:
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------
src/stackoverflow/57874971/learning.py 3 0 100%
src/stackoverflow/57874971/test_learning.py 13 0 100%
---------------------------------------------------------------------------
TOTAL 16 0 100%
I have a function I want to unit test contains calls two other functions. I am unsure how can I mock both functions at the same time properly using patch. I have provided an example of what I mean below. When I run nosetests, the tests pass but I feel that there must be a cleaner way to do this and I do not really Understand the piece regarding f.close()...
The directory structure looks like this:
program/
program/
data.py
tests/
data_test.py
data.py:
import cPickle
def write_out(file_path, data):
f = open(file_path, 'wb')
cPickle.dump(data, f)
f.close()
data_test.py:
from mock import MagicMock, patch
def test_write_out():
path = '~/collection'
mock_open = MagicMock()
mock_pickle = MagicMock()
f_mock = MagicMock()
with patch('__builtin__.open', mock_open):
f = mock_open.return_value
f.method.return_value = path
with patch('cPickle.dump', mock_pickle):
write_out(path, 'data')
mock_open.assert_called_once_with('~/collection', 'wb')
f.close.assert_any_call()
mock_pickle.assert_called_once_with('data', f)
Results:
$ nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.008s
OK
You can simplify your test by using the patch decorator and nesting them like so (they are MagicMock objects by default):
from unittest.mock import patch
#patch('cPickle.dump')
#patch('__builtin__.open')
def test_write_out(mock_open, mock_pickle):
path = '~/collection'
f = mock_open.return_value
f.method.return_value = path
write_out(path, 'data')
mock_open.assert_called_once_with('~/collection', 'wb')
mock_pickle.assert_called_once_with('data', f)
f.close.assert_any_call()
Calls to a MagicMock instance return a new MagicMock instance, so you can check that the returned value was called just like any other mocked object. In this case f is a MagicMock named 'open()' (try printing f).
In addition to the response #Matti John you can also use patch inside function test_write_out:
from mock import MagicMock, patch
def test_write_out():
path = '~/collection'
with patch('__builtin__.open') as mock_open, \
patch('cPickle.dump') as mock_pickle:
f = mock_open.return_value
...
As of Python 3.10 you can do use Parenthesized Context Managers like this
from unittest.mock import patch
def test_write_out():
with (
patch('cPickle.dump'),
patch('__builtin__.open') as open_mock, # example of using `as`
):
yield
Here's a simple example on how to test raising ConflictError in create_collection function using mock:
import os
from unittest import TestCase
from mock import patch
from ..program.data import ConflictError, create_collection
class TestCreateCollection(TestCase):
def test_path_exists(self):
with patch.object(os.path, 'exists') as mock_method:
mock_method.return_value = True
self.assertRaises(ConflictError, create_collection, 'test')
Please, also see mock docs and Michael Foord's awesome introduction to mock.