I want to test an evolving SQLite database application, which is in parallel used "productively". In fact I am investigating a pile of large text files by importing them to the database and fiddling around with it. I am used to develop test-driven, and I do not want to drop that for this investigation. But running tests against the "production" database feels somewhat strange. So my objective is to run the tests against a test database (a real SQLite database, not a mock) containing a controlled, but considerable amount of real data showing all kinds of variability I have met during the investigation.
To support this approach, I have a central module myconst.py containing a function returning the name of the database that is used like so:
import myconst
conn = sqlite3.connect(myconst.get_db_path())
Now in the unittest TestCases, I thought about mocking like so:
#patch("myconst.get_db_name", return_value="../test.sqlite")
def test_this_and_that(self, mo):
...
where the test calls functions that will, in nested functions, access the database using myconst.get_db_path().
I have tried to do a little mocking for myself first, but it tends to be clumsy and error prone so I decided to dive into the python mock module as shown before.
Unfortunately, I found warnings all over, that I am supposed to "mock where it's used and not where it's defined" like so:
#patch("mymodule.myconst.get_db_name", return_value="../test.sqlite")
def test_this_and_that(self, mo):
self.assertEqual(mymodule.func_to_be_tested(), 1)
But mymodule will likely not call database functions itself but delegate that to another module. This in turn would imply that my unit tests have to know the call tree where the database is actually access – something I really want to avoid because it would lead to unnecessary test refactoring when the code is refactored.
So I tried to create a minimal example to understand the behavior of mock and where it fails to allow me to mock "at the source". Because a multi module setup is clumsy here, I have provided the original code also on github for everybody's convenience. See this:
myconst.py
----------
# global definition of the database name
def get_db_name():
return "../production.sqlite"
# this will replace get_db_name()
TEST_VALUE = "../test.sqlite"
def fun():
return TEST_VALUE
inner.py
--------
import myconst
def functio():
return myconst.get_db_name()
print "inner:", functio()
test_inner.py
-------------
from mock import patch
import unittest
import myconst, inner
class Tests(unittest.TestCase):
#patch("inner.myconst.get_db_name", side_effect=myconst.fun)
def test_inner(self, mo):
"""mocking where used"""
self.assertEqual(inner.functio(), myconst.TEST_VALUE)
self.assertTrue(mo.called)
outer.py
--------
import inner
def functio():
return inner.functio()
print "outer:", functio()
test_outer.py
-------------
from mock import patch
import unittest
import myconst, outer
class Tests(unittest.TestCase):
#patch("myconst.get_db_name", side_effect=myconst.fun)
def test_outer(self, mo):
"""mocking where it comes from"""
self.assertEqual(outer.functio(), myconst.TEST_VALUE)
self.assertTrue(mo.called)
unittests.py
------------
"""Deeply mocking a database name..."""
import unittest
print(__doc__)
suite = unittest.TestLoader().discover('.', pattern='test_*.py')
unittest.TextTestRunner(verbosity=2).run(suite)
test_inner.py works like the sources linked above say, and so I was expecting it to pass. test_outer.py should fail when I understand the caveats right. But all the tests pass without complaint! So my mock is drawn all the time, even when the mocked function is called from down the callstack like in test_outer.py. From that example I would conclude that my approach is safe, but on the other hand the warnings are consistent across quite some sources and I do not want to recklessly risk my "production" database by using concepts that I do not grok.
So my question is: Do I misunderstand the warnings or are these warnings just over-cautious?
Finally I sorted it out. Maybe this will help future visitors, so I will share my findings:
When changing the code like so:
inner.py
--------
from myconst import get_db_name
def functio():
return get_db_name()
test_inner.py
-------------
#patch("inner.get_db_name", side_effect=myconst.fun)
def test_inner(self, mo):
self.assertEqual(inner.functio(), myconst.TEST_VALUE)
test_inner will succeed, but test_outer will break with
AssertionError: '../production.sqlite' != '../test.sqlite'
This is because mock.patch will not replace the referenced object, which is function get_db_name in module myconst in both cases. mock will instead replace the usages of the name "myconst.get_db_name" by the Mock object passed as the second parameter to the test.
test_outer.py
-------------
#patch("myconst.get_db_name", side_effect=myconst.fun)
def test_outer(self, mo):
self.assertEqual(outer.functio(), myconst.TEST_VALUE)
Since I mock only "myconst.getdb_name" here and inner.py accesses get_db_name via "inner.get_db_name", the test will fail.
By using the proper name, however, this can be fixed:
#patch("outer.inner.get_db_name", return_value=myconst.TEST_VALUE)
def test_outer(self, mo):
self.assertEqual(outer.functio(), myconst.TEST_VALUE)
So the conclusion is that my approach will be safe when I make sure that all modules accessing the database include myconst and use myconst.get_db_name. Alternatively all modules could from myconst import get_db_name and use get_db_name. But I have to draw this decision globally.
Because I control all code accessing get_db_name I am safe. One can argue whether this is good style or not (assumingly the latter), but technically it's safe. Would I mock a library function instead, I could hardly control access to that function and so mocking "where it's defined" becomes risky. This is why the sources cited are warning.
Related
I have a complex function that calls many other 3rd party methods. I monkeypatched them out one by one:
import ThirdParty as tp
def my_method():
tp.func_3rd_party_1()
...
tp.func_3rd_party_5()
return "some_value"
In my test:
import pytest
def test_my_method(monkeypatch):
monkeypatch.setattr(ThirdParty, 'func_3rd_party_1', some_mock_1())
...
monkeypatch.setattr(ThirdParty, 'func_3rd_party_5', some_mock_5())
return_value = my_method()
assert return value
This runs just fine but the test feels too implicit for me in this form. I'd like to explicitly state that the monkeypatched methods were indeed called.
For the record, my mocked methods are not using any inbuilt Mock library resource. They are just redefined methods (smart stubs).
Is there any way to assert for that?
So the pytest monkeypatching fixture is specifically provided so you can change some global attributes like environment variables, stuff in third party libraries, etc, to provide some controlled and easy behavior for your test.
The Mock objects, on the other hand, are meant to provide all sorts of tracking and inspection on the object.
The two go hand in hand: You use patching to replace some third party function with a Mock object, then execute your code, and then ask the Mock object if it has indeed been invoked with the right arguments, for the right number of times.
Note that even though the mock module is part of unittest, it works perfectly fine with pytest.
Now as for the patching itself, it's up to your personal preference, and depends a bit on what exactly you want to patch, whether using unittest.mock.patch is more compact or pytest's monkeypatch fixture.
import pytest
from unittest.mock import Mock
def test_my_method(monkeypatch):
# refer to the mock module documentation for more complex
# set ups, where the mock object _also_ exhibits some behavior.
# as is, calling the function doesn't actually _do_ anything.
some_mock_1 = Mock()
...
some_mock_5 = Mock(return_value=66)
monkeypatch.setattr(ThirdParty, 'func_3rd_party_1', some_mock_1)
...
monkeypatch.setattr(ThirdParty, 'func_3rd_party_5', some_mock_5)
some_mock_1.assert_called_once()
some_mock_5.assert_called_with(42)
...
Now a note on this type of testing: Don't go overboard! It can quite easily lead to what's called brittle tests: Tests that break with the slightest change to your code. It can make refactoring an impossible neightmare.
These types of assertions are best when you use them in a message-focused object-oriented approach. If the whole point of the class or method under test is to invoke, in a particular way, the method or class of another object, then Mock away. If the calls to third party functions on the other hand are merely a means to an end, then go a level higher with your test and test for the desired behavior instead.
We have a Python 3.7 application that has a declared constants.py file that has this form:
APP_CONSTANT_1 = os.environ.get('app-constant-1-value')
In a test.py we were hoping to test the setting of these constants using something like this (this is highly simplified but represents the core issue):
class TestConfig:
"""General config tests"""
#pytest.fixture
def mock_os_environ(self, monkeypatch):
""" """
def mock_get(*args, **kwargs):
return 'test_config_value'
monkeypatch.setattr(os.environ, "get", mock_get)
def test_mock_env_vars(self, mock_os_environ):
import constants
assert os.environ.get('app-constant-1-value') == 'test_config_value' #passes
assert constants.APP_CONSTANT_1 == 'test_config_value' #fails
The second assertion fails as constants.constants.APP_CONSTANT_1 is None. Turns out that the constants.py seems to be loaded during pytest's 'collecting' phase and thus is already set by the time the test is run.
What are we missing here? I feel like there is a simple way to resolve this in pytest but haven't yet discovered the secret. Is there some way to avoid loading the constants file prior to the tests being run? Any ideas are appreciated.
The problem is most likely that constants has been loaded before. To make sure it gets the patched value, you have to reload it:
import os
from importlib import reload
import pytest
import constants
class TestConfig:
"""General config tests"""
#pytest.fixture
def mock_os_environ(self, monkeypatch):
""" """
monkeypatch.setenv('app-constant-1-value', 'test_config_value')
reload(constants)
def test_mock_env_vars(self, mock_os_environ):
assert os.environ.get('app-constant-1-value') == 'test_config_value'
assert app.APP_CONSTANT_1 == 'test_config_value'
Note that I used monkeypatch.setenv to specifically set the variable you need. If you don't need to change all environment variables, this is easier to use.
Erm, I would avoid using constants. You can subclass os.environment for a start, and then use a mocked subclass for your unit tests, so you can have my_env.unique_env as a member variable. You can then use eg. import json to use a json configuration file without getting involved with hard coded python.
The subclass can then hold the relevant variables (or methods if you prefer)
Being able to add a facade to os.environment provides you with the abstraction you are looking for, without any of the problems.
Even is one is using a legacy/larger project, the advantage of using an adapter for access to the environment must be apparent.
Since you are writing unit tests, there is an opportunity to use an adapter class in both the tests and the functions being tested.
I am trying to understand the mock/monkeypatch/pytest-mock capabilities.
Let me know if this is possible. If not could you please suggest how I can test this code.
My code structure:
/
./app
../__init__.py
../some_module1
.../__init__.py
../some_module2
.../__init__.py
./tests
../test_db.py
The /app/__init__.py is where my application (a Flask application if it helps) is started along with initializing a database connection object to a MongoDB database:
# ...
def create_app():
# ...
return app
db_conn = DB()
The some_module1 and some_module import the db_conn object and use it as part of their functions:
## some_module1/__init__.py
from app import db_conn
...
db = db_conn.db_name2.db_collection2
def some_func1():
data = db.find()
# check and do something with data
return boolean_result
...
## some_module2/__init__.py
from app import db_conn
...
db = db_conn.db_name1.db_collection1
def some_func2():
data = db.find()
# check and do something with data
return boolean_result
...
In my tests, I want to test if my code works properly based on the data received from the database.
I want to mock the database, more specifically the db_conn object since I don't want to use a real database (which would be a lot of work setting up the environment and maintaining it).
Any suggestions on how I can emulate the db_conn?
I've been exploring pytest-mock and magicmock but I don't think or know how to mock the db_conn in my test.
To answer the intial question "How to mock a imported object with pytest-mock or magicmock" you can do:
from unittest import mock # because unittest's mock works great with pytest
def test_some_func1():
with mock.patch('some_module1.db', mock.MagicMock(return_value=...)) as magicmock:
result = some_func1(...)
assert ... e.g. different fields of magicmock
assert expected == result
# or alternatively use annotations
#mock.patch('some_module2.db', mock.MagicMock(return_value=...))
def test_some_func2():
result = some_func2(...)
note that you do not patch the actual source of db
For your other use case
I want to mock the database (using a mongo database), more specifically the "db_conn" object
you similarly follow the hints of the link above:
mock.patch('some_module1.db_conn', mock.MagicMock(return_value=...))
Given that, you will notice in your tests that db from `db = db_conn.db_name2.db_collection2' will create another mock object. Calls to that object will be recorded as well. In such a way you will be able to trace history of calls and values assignments as well.
Furthermore, see an example how to pach mongo db.
For testing of Flask apps see the documentation of flask. Also this is a nice explanation as well, and uses DB connections.
As a general hint, like #MikeMajara mentioned - separate your code more into smaller functions that are also easy to test. In tradition to TDD: write tests first, implement later, and refactor (especially DRY!)
I believe you are right not testing cases on a real database because it's not unit testing anymore if you are using external dependencies.
There is a possibility to specify return-value and customize it (different return values on each iteration even) for Mock or MagicMock objects.
from unittest.mock import Mock, patch
from app import db_conn
#patch('app.db_conn.find')
def test_some_func1(db_con_mock):
...
assert ...
Keep in mind that in each patch you should specify the import path of db_conn - the path where db_conn **is used (I assume it's a different path in each test), not where it is defined.
Separation of concerns. Build methods that do one, and only one thing. Even more if you are going with TDD. In your example some_func2 does more than one. You could refactor as follows:
def get_object_from_db():
return db.find()
def check_condition_on_object(obj):
check something to do with object
return true or false
def some_func2():
obj = get_object_from_db()
check_condition_on_object(obj)
With this approach you could easily test get_object_from_db and check_condition_on_object separately. This will improve readability, avoid bugs, and help detecting these if they appear at some point.
About "mocking an imported object". You might be trying to mock an object with a library that is meant for a more advance case than yours. These libraries provide you with a bunch of methods surrounding test environment out of the box that you might not need. From the looks, you just want to populate an object with mock data, and/or interact with a mocked db_connection instance. So...
To populate, I would simplify: You know the condition you want to test and you want to check if the result for a given object is the expected one. Just build yourself a test_object_provider.py that returns your known cases for true|false. No need to make things more complex.
To use a fake MongoDB connection you can try with mongomock. (although ideally you would test mongo with a real instance in a separate test).
Let's assume that we have a system of modules that exists only on production stage. At the moment of testing these modules do not exist. But still I would like to write tests for the code that uses those modules. Let's also assume that I know how to mock all the necessary objects from those modules. The question is: how do I conveniently add module stubs into current hierarchy?
Here is a small example. The functionality I want to test is placed in a file called actual.py:
actual.py:
def coolfunc():
from level1.level2.level3_1 import thing1
from level1.level2.level3_2 import thing2
do_something(thing1)
do_something_else(thing2)
In my test suite I already have everything I need: I have thing1_mock and thing2_mock. Also I have a testing function. What I need is to add level1.level2... into current module system. Like this:
tests.py
import sys
import actual
class SomeTestCase(TestCase):
thing1_mock = mock1()
thing2_mock = mock2()
def setUp(self):
sys.modules['level1'] = what should I do here?
#patch('level1.level2.level3_1.thing1', thing1_mock)
#patch('level1.level2.level3_1.thing1', thing2_mock)
def test_some_case(self):
actual.coolfunc()
I know that I can substitute sys.modules['level1'] with an object containing another object and so on. But it seems like a lot of code for me. I assume that there must be much simpler and prettier solution. I just cannot find it.
So, no one helped me with my problem and I decided to solve it by myself. Here is a micro-lib called surrogate which allows one to create stubs for non-existing modules.
Lib can be used with mock like this:
from surrogate import surrogate
from mock import patch
#surrogate('this.module.doesnt.exist')
#patch('this.module.doesnt.exist', whatever)
def test_something():
from this.module.doesnt import exist
do_something()
Firstly #surrogate decorator creates stubs for non-existing modules, then #patch decorator can alter them. Just as #patch, #surrogate decorators can be used "in plural", thus stubbing more than one module path. All stubs exist only at the lifetime of decorated function.
If anyone gets any use of this lib, that would be great :)
I'd like to have several of the doctests in a file share test data and/or functions. Is there a way to do this without locating them in an external file or within the code of the file being tested?
update
"""This is the docstring for the module ``fish``.
I've discovered that I can access the module under test
from within the doctest, and make changes to it, eg
>>> import fish
>>> fish.data = {1: 'red', 2: 'blue'}
"""
def jef():
"""
Modifications made to the module will persist across subsequent tests:
>>> import fish
>>> fish.data[1]
'red'
"""
pass
def seuss():
"""
Although the doctest documentation claims that
"each time doctest finds a docstring to test,
it uses a shallow copy of M‘s globals",
modifications made to the module by a doctest
are not imported into the context of subsequent docstrings:
>>> data
Traceback (most recent call last):
...
NameError: name 'data' is not defined
"""
pass
So I guess that doctest copies the module once, and then copies the copy for each docstring?
In any case, importing the module into each docstring seems usable, if awkward.
I'd prefer to use a separate namespace for this, to avoid accidentally trampling on actual module data that will or will not be imported into subsequent tests in an possibly undocumented manner.
It's occurred to me that it's (theoretically) possible to dynamically create a module in order to contain this namespace. However as yet I've not gotten any direction on how to do that from the question I asked about that a while back. Any information is quite welcome! (as a response to the appropriate question)
In any case I'd prefer to have the changes be propagated directly into the namespace of subsequent docstrings. So my original question still stands, with that as a qualifier.
This is the sort of thing that causes people to turn away from doctests: as your tests grow in complexity, you need real programming tools to be able to engineer your tests just as you would engineer your product code.
I don't think there's a way to include shared data or functions in doctests other than defining them in your product code and then using them in the doctests.
You are going to need to use real code to define some of your test infrastructure. If you like doctests, you can use that infrastructure from your doctests.
This is possible, albeit perhaps not advertised as loudly.
To obtain literate modules with tests that all use a shared execution context (i.e. individual tests can share and re-use their results), one has to look at the relevant part of documentation which says:
... each time doctest finds a docstring to test, it uses a shallow copy of M‘s globals, so that running tests doesn’t change the module’s real globals, and so that one test in M can’t leave behind crumbs that accidentally allow another test to work.
...
You can force use of your own dict as the execution context by passing globs=your_dict to testmod() or testfile() instead.
Given this, I managed to reverse-engineer from doctest module that besides using copies (i.e. the dict's copy() method), it also clears the globals dict (using clear()) after each test.
Thus, one can patch their own globals dictionary with something like:
class Context(dict):
def clear(self):
pass
def copy(self):
return self
and then use it as:
import doctest
from importlib import import_module
module = import_module('some.module')
doctest.testmod(module,
# Make a copy of globals so tests in this
# module don't affect the tests in another
glob=Context(module.__dict__.copy()))