Using Python's Unittest framework how do you mock or replace a module that has code that is run as the module is loaded?
I understand this is poorly written code, but this is similar to what I have to test. (See example)
I understand that once a module is imported it can be patched to use mocks. But what if there is code that is run immediately?
I have a file that I need to put under test. One of the files it imports runs code immediately, see example.
file_under_test.py
from somewhere.something.worker import a_func as f
class ClassToTest():
__init__(self):
...
the somewhere.something.worker module
import os
import redis
REDIS_HOST = os.environ.get('redishost', '') #<-- Mock this
connection = redis.Redis(host=REDIS_HOST) #<--- Mock this
class AClass():
...
def a_func():
connection.doSomething()
...
Defer creating the connection until you are really ready for it to happen. As a bonus, you can have init_connection take an optional pre-allocated connection rather than always creating it on-demand. This makes it easier to migrate towards avoiding the global connection altogether.
import os
import redis
connection = None
def init_connection(c=None):
global connection
if connection is None:
if c is None:
c = redis.Redis(host=os.environ.get('redishost', ''))
connection = c
...
Then, in your test module, you can call init_connection from inside setupModule, with the option of passing in the desired connection-like object
instead of having to patch anything.
def setupModule():
init_connection()
# or
# conn = Mock()
# ... configure the mock ...
# init_connection(conn)
class ClassToTest():
__init__(self):
...
Related
I have something like this in my source file
# code.py
def some_func():
# doing some connections and stuff
return {'someKey': 'someVal'}
class ClassToTest:
var = some_func()
My test file looks like this... I am trying to mock some_func as I want to avoid creating connections.
# test_code.py
from src.code import ClassToTest
def mock_function():
return {"someOtherKey": "someOtherValue"}
class Test_Code(unittest.TestCase):
#mock.patch('src.code.some_func', new=mock_function)
def test_ClassToTest(self):
self.assertEqual(ClassToTest.var, {"someOtherKey": "someOtherValue"})
But this doesn't work. On the other hand if var is an instant variable mock works fine. I guess this is due to class variables getting initialized during imports. How do I properly mock some_func before var even gets initialized?
When you imported code.py, there is no active patch yet, so when ClassToTest.var was initialized, it used the original some_func. Only then would the patch for src.code.some_func would take effect which obviously is too late now.
Solution 1
What you can do is to patch some_func and then reload code.py so that it re-initializes the ClassToTest including its attribute var. Thus since we already have an active patch by the time we reload code.py, then ClassToTest.var would be set with the patched value.
But we can't do it if both the class and the patched function lives in the same file, so to make it testable move some_func to another file and then just import it.
src/code.py
from src.other import some_func
class ClassToTest:
var = some_func()
src/other.py
def some_func():
# doing some connections and stuff
return {'realKey': 'realValue'}
test_code.py
from importlib import reload
import sys
import unittest
from unittest import mock
from src.code import ClassToTest # This will always refer to the unpatched version
def mock_function():
return {"someOtherKey": "someOtherValue"}
class Test_Code(unittest.TestCase):
def test_real_first(self):
self.assertEqual(ClassToTest.var, {"realKey": "realValue"})
#mock.patch('src.other.some_func', new=mock_function)
def test_mock_then_reload(self):
# Option 1:
# import src
# reload(src.code)
# Option 2
reload(sys.modules['src.code'])
from src.code import ClassToTest # This will be the patched version
self.assertEqual(ClassToTest.var, {"someOtherKey": "someOtherValue"})
def test_real_last(self):
self.assertEqual(ClassToTest.var, {"realKey": "realValue"})
Output
$ pytest -q
... [100%]
3 passed in 0.04s
Solution 2
If you don't want the real some_func to be ever called during test, then just reloading isn't enough. What needs to be done is to never import the file containing ClassToTest nor any file that would import it indirectly. Only import it once an active patch for some_func is already established.
from importlib import reload
import sys
import unittest
from unittest import mock
# from src.code import ClassToTest # Remove this import!!!
def mock_function():
return {"someOtherKey": "someOtherValue"}
class Test_Code(unittest.TestCase):
#mock.patch('src.other.some_func', new=mock_function)
def test_mock_then_reload(self):
from src.code import ClassToTest # Move the import here once the patch has taken effect already
self.assertEqual(ClassToTest.var, {"someOtherKey": "someOtherValue"})
I'm new to testing and testing in python. I have a python class that looks like this :
File name : my_hive.py
from pyhive import hive
class Hive:
def __init__(self, hive_ip):
self.cursor = hive.connect(hive_ip).cursor()
def execute(self, command):
self.cursor.execute(command)
I want to mock these functions : pyhive.hive.connect, pyhive.Connection.cursor(used by my class as hive.connect(hive_ip).cursor()) and pyhive.Cursor.execute (used by my class as self.cursor.execute(command) in execute method).
I'm able to mock function call hive.connect and also I have been able to assert that it has been called with hive_ip given by me as follows.
import unittest
import mock
from my_hive import Hive
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
def test_workflow(self, mock_connect):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
But how do I make sure that subsequent function calls like .cursor() and self.cursor.execute() have also been called? hive.connect(hive_ip) returns an instance of pyhive.hive.Connection, which has method called cursor
I have tried to add mocks like this :
import unittest
import mock
from hive_schema_processor import HiveSchemaProcessor
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
#mock.patch('pyhive.hive.Connection.cursor')
def test_workflow(self, mock_connect, mock_cursor):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
mock_cursor.assert_called()
But the tests are failed with complain :
AssertionError: expected call not found.
Expected: cursor('localhost')
Actual: not called.
Your problem is that you have already mocked connect, so the subsequent calls on the result of connect will be made on the mock, not on the real object.
To check that call, you have to make the check on the returned mock object instead:
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
def test_workflow(self, mock_connect):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
mock_cursor = mock_connect.return_value.cursor
mock_cursor.assert_called()
Each call on a mock produces another mock object.
mock_connect.return_value gives you the mock that is returned by calling mock_connect, and mock_connect.return_value.cursor contains another mock that will actually be called.
TL;DR
How can I patch or mock "any functions that are not being called/used directly"?
Sceneario
I have a simple unit-test snippet as
# utils/functions.py
def get_user_agents():
# sends requests to a private network and pulls data
return pulled_data
# my_module/tasks.py
def create_foo():
from utils.functions import get_user_agents
value = get_user_agents()
# do something with value
return some_value
# test.py
class TestFooKlass(unittest.TestCase):
def setUp(self):
create_foo()
def test_foo(self):
...
Here in setUp() method I am calling get_user_agents() function indirectly by calling create_foo(). During this execution I have got socket.timeout exception since get_user_agents() tried to access a private network. So, how can I manipulate the return data or the entire get_user_agents function during the test?
Also, is there any way to persists this mock/patch during the whole test suite execution?
It does not matter that you call the function indirectly - important is to patch it as it is imported. In your example you import the function to be patched locally inside the tested function, so it will only be imported at function run time. In this case you have to patch the function as imported from its module (e.g. 'utils.functions.get_user_agents'):
class TestFooKlass(unittest.TestCase):
def setUp(self):
self.patcher = mock.patch('utils.functions.get_user_agents',
return_value='foo') # whatever it shall return in the test
self.patcher.start() # this returns the patched object, i case you need it
create_foo()
def tearDown(self):
self.patcher.stop()
def test_foo(self):
...
If you had imported the function instead at module level like:
from utils.functions import get_user_agents
def create_foo():
value = get_user_agents()
...
you should have patched the imported instance instead:
self.patcher = mock.patch('my_module.tasks.get_user_agents',
return_value='foo')
As for patching the module for all tests: you can start patching in setUp as shown above, and stop it in tearDown.
Suppose we have the following Flask view function to test. Specifically, we want to mock create_foo() as it writes to the filesystem.
# proj_root/some_project/views.py
from some_project import APP
from some_project.libraries.foo import create_foo
#APP.route('/run', methods=['POST']
def run():
bar = create_foo()
return 'Running'
Now we want to write a unit test for run() with the call to create_foo() mocked to avoid creating unnecessary files.
# proj_root/tests/test_views.py
from some_project import some_project
#pytest.fixture
def client():
some_project.APP.config['TESTING'] = True
with some_project.APP.test_client() as client:
yield client
def test_run(client, monkeypatch):
monkeypatch.setattr('some_project.libraries.foo.create_foo', lambda: None)
response = client.post('/run')
assert b'Running' in response.data
It seems that this approach should work even with the named create_foo import. The tests all pass, however the original code of create_foo is clearly being executed as a new file in the filesystem is created each time the test suite is run. What am I missing? I suspect it has something to do with the named imports based on some related questions but I'm not sure.
The correct monkeypatch is:
monkeypatch.setattr('some_project.views.create_foo', lambda: None)
The reason for this is pretty well explained here.
So,
consider I have a simple library that I am trying to write unit-tests for. This library talks to a database and then uses that data to call an SOAP API. I have three modules, and a testfile for each module.
dir structure:
./mypkg
../__init__.py
../main.py
../db.py
../api.py
./tests
../test_main
../test_db
../test_api
Code:
#db.py
import mysqlclient
class Db(object):
def __init__(self):
self._client = mysqlclient.Client()
#property
def data(self):
return self._client.some_query()
#api.py
import soapclient
class Api(object):
def __init__(self):
self._client = soapclient.Client()
#property
def call(self):
return self._client.some_external_call()
#main.py
from db import Db
from api import Api
class MyLib(object):
def __init__(self):
self.db = Db()
self.api = Api()
def caller(self):
return self.api.call(self.db.data)
Unit-Tests:
#test_db.py
import mock
from mypkg.db import Db
#mock.patch('mypkg.db.mysqlclient')
def test_db(mysqlclient_mock):
mysqlclient_mock.Client.return_value.some_query = {'data':'data'}
db = Db()
assert db.data == {'data':'data'}
#test_api.py
import mock
from mypkg.api import Api
#mock.patch('mypkg.db.soapclient')
def test_db(soap_mock):
soap_mock.Client.return_value.some_external_call = 'foo'
api = Api()
assert api.call == 'foo'
In the above example, mypkg.main.MyLib calls mypkg.db.Db() (uses third-party mysqlclient) and then mypkg.api.Api() (uses third-party soapclient)
I am using mock.patch to patch the third-party libraries to mock my db and api calls in test_db and test_api separately.
Now my question is, is it recommended to patch these external calls again in test_main OR simply patch db.Db and api.Api? (this example is pretty simple, but in larger libraries, the code becomes cumbersome when patching the external calls again or even using test helper functions that patch internal libraries).
Option1: patch external libraries in main again
#test_main.py
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.db.mysqlclient')
#mock.patch('mypkg.api.soapclient')
def test_main(soap_mock, mysqlcient_mock):
ml = MyLib()
soap_mock.Client.return_value.some_external_call = 'foo'
assert ml.caller() == 'foo'
Option2: patch internal libraries
#test_main.py
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.db.Db')
#mock.patch('mypkg.api.Api')
def test_main(api_mock, db_mock):
ml = MyLib()
api_mock.return_value = 'foo'
assert ml.caller() == 'foo'
mock.patch creates a mock version of something where it's imported, not where it lives. This means the string passed to mock.patch has to be a path to an imported module in the module under test. Here's what the patch decorators should look like in test_main.py:
#mock.patch('mypkg.main.Db')
#mock.patch('mypkg.main.Api')
Also, the handles you have on your patched modules (api_mock and db_mock) refer to the classes, not instances of those classes. When you write api_mock.return_value = 'foo', you're telling api_mock to return 'foo' when it gets called, not when an instance of it has a method called on it. Here are the objects in main.py and how they relate to api_mock and db_mock in your test:
Api is a class : api_mock
Api() is an instance : api_mock.return_value
Api().call is an instance method : api_mock.return_value.call
Api().call() is a return value : api_mock.return_value.call.return_value
Db is a class : db_mock
Db() is an instance : db_mock.return_value
Db().data is an attribute : db_mock.return_value.data
test_main.py should therefore look like this:
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.main.Db')
#mock.patch('mypkg.main.Api')
def test_main(api_mock, db_mock):
ml = MyLib()
api_mock.return_value.call.return_value = 'foo'
db_mock.return_value.data = 'some data' # we need this to test that the call to api_mock had the correct arguments.
assert ml.caller() == 'foo'
api_mock.return_value.call.assert_called_once_with('some data')
The first patch in Option 1 would work great for unit-testing db.py, because it gives the db module a mock version of mysqlclient. Similarly, #mock.patch('mypkg.api.soapclient') belongs in test_api.py.
I can't think of a way Option 2 could help you unit-test anything.
Edited: I was incorrectly referring to classes as modules. db.py and api.py are modules