Develop Reference

Is it possible to unit test the parameters that I pass to the function in python?

In this scenario:
In FileA:
from FileB import BJob
class EasyJob
def __init__():
self.job = BJob()
def launch_job_in_A():
self.job.launch_job_in_B(cpu=100, memory=200)
In FileB:
class BJob():
def __init__():
pass
def launch_job_in_B(cpu=0, memory=0):
do_some_thing(cpu, memory)
...
Is there a way to write a unit test for launch_job_in_A() function and also verify whether we do pass cpu=100 and memory=200 to launch_job_in_B() function successfully?
Broadly question is like, is it possible to check the parameter we passed to the function?

You can create a Mock via unittest.mock.patch and then use the assert_called_once_with method to verify the function call:
import unittest
from unittest import TestCase
from unittest.mock import patch
from FileA import EasyJob
class TestEasyJob(TestCase):
def test_launch_job_in_A(self):
with patch('FileB.BJob.launch_job_in_B') as mock:
job = EasyJob()
job.launch_job_in_A()
mock.assert_called_once_with(cpu=100, memory=200)
unittest.main()

Yes, you would replace self.job with a mock object and its launch_job_in_B with a mock method.
This would be easier if there was an option to pass in the BJob, rather than hard coding it. This avoids having to mock every call to launch_job_in_B which might have unexpected side effects. This also makes the class more flexible.
from FileB import BJob
class EasyJob
def __init__(bjob=BJob()):
self.job = bjob
def launch_job_in_A():
self.job.launch_job_in_B(cpu=100, memory=200)
Now we can mock the method just for a single object.
from FileA import EasyJob
from FileB import BJob
from unittest.mock import MagicMock
# Set up the mocked object and method.
mock_bjob = BJob()
# If it needs to return something, configure that.
mock_bjob.launch_job_in_B() = MagicMock()
# Make the EasyJob with the mocked BJob
ejob = EasyJob(bjob=mock_bjob)
# Call the method in question.
ejob.launch_job_in_A()
# Check if the method was called with the correct arguments.
mock_bjob.method.assert_called_once_with(cpu=100, memory=200)
BJob.launch_job_in_B would be tested in its own unit test.

How to mock subsequent function calls in python?

I'm new to testing and testing in python. I have a python class that looks like this :
File name : my_hive.py
from pyhive import hive
class Hive:
def __init__(self, hive_ip):
self.cursor = hive.connect(hive_ip).cursor()
def execute(self, command):
self.cursor.execute(command)
I want to mock these functions : pyhive.hive.connect, pyhive.Connection.cursor(used by my class as hive.connect(hive_ip).cursor()) and pyhive.Cursor.execute (used by my class as self.cursor.execute(command) in execute method).
I'm able to mock function call hive.connect and also I have been able to assert that it has been called with hive_ip given by me as follows.
import unittest
import mock
from my_hive import Hive
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
def test_workflow(self, mock_connect):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
But how do I make sure that subsequent function calls like .cursor() and self.cursor.execute() have also been called? hive.connect(hive_ip) returns an instance of pyhive.hive.Connection, which has method called cursor
I have tried to add mocks like this :
import unittest
import mock
from hive_schema_processor import HiveSchemaProcessor
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
#mock.patch('pyhive.hive.Connection.cursor')
def test_workflow(self, mock_connect, mock_cursor):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
mock_cursor.assert_called()
But the tests are failed with complain :
AssertionError: expected call not found.
Expected: cursor('localhost')
Actual: not called.

Your problem is that you have already mocked connect, so the subsequent calls on the result of connect will be made on the mock, not on the real object.
To check that call, you have to make the check on the returned mock object instead:
class TestHive(unittest.TestCase):
#mock.patch('pyhive.hive.connect')
def test_workflow(self, mock_connect):
hive_ip = "localhost"
processor = Hive(hive_ip)
mock_connect.assert_called_with(hive_ip)
mock_cursor = mock_connect.return_value.cursor
mock_cursor.assert_called()
Each call on a mock produces another mock object.
mock_connect.return_value gives you the mock that is returned by calling mock_connect, and mock_connect.return_value.cursor contains another mock that will actually be called.

Using unittests and moto to mock AWS

I am used to pytest approach for unit testing, without using classes. Today I wanted to give a try to unittest and I wanted to encapsulate my tests inside a TestCase.
Then consider this sample test class:
import unittest
import moto
import boto3
class TestMyClass(unittest.TestCase):
#classmethod
#moto.mock_ssm
def setUpClass(cls) -> None:
cls.ssm_client = boto3.client('ssm')
cls.ssm_client.put_parameter(Name='test', Value='foo', Type='String')
#moto.mock_ssm
def test_something(self):
value = self.ssm_client.get_parameter(Name='test').get('Parameter').get('Value')
self.assertEqual(value, 'foo')
Why is not the parameter placed in setUpClass visible from the test? I could imagine that by using the #moto.mock_ssm decorator there it would all have been done in the mocked context.
I can, however, place the parameter within test_something as just:
#moto.mock_ssm
def test_something(self):
self.ssm_client.put_parameter(Name='test', Value='foo', Type='String')
value = self.ssm_client.get_parameter(Name='test').get('Parameter').get('Value')
self.assertEqual(value, 'foo')
And then it (obviously) works. Why not with my first approach? I do not want to be populating the fake ssm parameter for each test that will rely on it. What is the best way of doing so here?
The reason why I am asking this is because the class I want to test requires this parameter when it is initialised.

The mock-decorators are scoped for that particular function, so data created inside that function will only be available there.
Instead, you can use the class-decorator:
import unittest
import moto
import boto3
#moto.mock_ssm
class TestMyClass(unittest.TestCase):
client = None
def setUp(self) -> None:
self.client = boto3.client('ssm')
self.client.put_parameter(Name='test', Value='foo', Type='String')
def test_something(self):
value = self.client.get_parameter(Name='test').get('Parameter').get('Value')
self.assertEqual(value, 'foo')
Note that I've switched to setUp, instead of setUpClass. Because setUpClass is a classmethod, it is executed before the decorator is applied, and it will try to execute this method against AWS itself.

How to mock a base_class from an external module?

I'm trying to unit test a class which is derived from a base_class in an external module. In my dev/test environment I have not access to this external module, which means I have to somehow mock this base_class.
My test-code resides in a different file from the code I'm trying to test.
The problem can be summarized as follows:
my_class.py
import external_module
class MyClass(external_module.ExternalClass):
def test_method(self):
return "successful"
test_my_class.py
import sys
import unittest
from unittest.mock import MagicMock
sys.modules['external_module'] = MagicMock()
from my_class import MyClass
class TestMyClass(unittest.TestCase):
def test_first(self):
my_class = MyClass()
result = my_class.test_method()
self.assertEqual(result, "successful")
if __name__ == '__main__':
unittest.main()
Results
When running test_my_class.py the result are the following.
AssertionError: <MagicMock name='mock.ExternalClass.test_method()' id='140272215184664'> != 'successful'
Clearly since the external_module is mocked, even MyClass becomes an instance of a mock-object.
Similar posts
The problem is similar to as described in Python mock: mocking base class for inheritance, but has the difference that the base_class is from an external module.
Even How to mock a base class with python mock library show som similarities to my problem, though the solutions can not be directly applied.
Tries and failures
To get the import
import external_module
to work in my_class.py
sys.modules['external_module'] = MagicMock()
need to be set in test_my_class.py.
Though, this leads to that external_module.* becomes a Mock-instance.

You could create a helper module mocked_external_module, which can be imported from your tests and also contains a class base_class. Then you do the following in your test code:
import mocked_external_module
sys.modules['external_module'] = mocked_external_module
Plus, every method of your base_class that you need to mock you can create as a Mock or MagicMock.

Best practices using python mock for testing functions within sub modules

So,
consider I have a simple library that I am trying to write unit-tests for. This library talks to a database and then uses that data to call an SOAP API. I have three modules, and a testfile for each module.
dir structure:
./mypkg
../__init__.py
../main.py
../db.py
../api.py
./tests
../test_main
../test_db
../test_api
Code:
#db.py
import mysqlclient
class Db(object):
def __init__(self):
self._client = mysqlclient.Client()
#property
def data(self):
return self._client.some_query()
#api.py
import soapclient
class Api(object):
def __init__(self):
self._client = soapclient.Client()
#property
def call(self):
return self._client.some_external_call()
#main.py
from db import Db
from api import Api
class MyLib(object):
def __init__(self):
self.db = Db()
self.api = Api()
def caller(self):
return self.api.call(self.db.data)
Unit-Tests:
#test_db.py
import mock
from mypkg.db import Db
#mock.patch('mypkg.db.mysqlclient')
def test_db(mysqlclient_mock):
mysqlclient_mock.Client.return_value.some_query = {'data':'data'}
db = Db()
assert db.data == {'data':'data'}
#test_api.py
import mock
from mypkg.api import Api
#mock.patch('mypkg.db.soapclient')
def test_db(soap_mock):
soap_mock.Client.return_value.some_external_call = 'foo'
api = Api()
assert api.call == 'foo'
In the above example, mypkg.main.MyLib calls mypkg.db.Db() (uses third-party mysqlclient) and then mypkg.api.Api() (uses third-party soapclient)
I am using mock.patch to patch the third-party libraries to mock my db and api calls in test_db and test_api separately.
Now my question is, is it recommended to patch these external calls again in test_main OR simply patch db.Db and api.Api? (this example is pretty simple, but in larger libraries, the code becomes cumbersome when patching the external calls again or even using test helper functions that patch internal libraries).
Option1: patch external libraries in main again
#test_main.py
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.db.mysqlclient')
#mock.patch('mypkg.api.soapclient')
def test_main(soap_mock, mysqlcient_mock):
ml = MyLib()
soap_mock.Client.return_value.some_external_call = 'foo'
assert ml.caller() == 'foo'
Option2: patch internal libraries
#test_main.py
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.db.Db')
#mock.patch('mypkg.api.Api')
def test_main(api_mock, db_mock):
ml = MyLib()
api_mock.return_value = 'foo'
assert ml.caller() == 'foo'

mock.patch creates a mock version of something where it's imported, not where it lives. This means the string passed to mock.patch has to be a path to an imported module in the module under test. Here's what the patch decorators should look like in test_main.py:
#mock.patch('mypkg.main.Db')
#mock.patch('mypkg.main.Api')
Also, the handles you have on your patched modules (api_mock and db_mock) refer to the classes, not instances of those classes. When you write api_mock.return_value = 'foo', you're telling api_mock to return 'foo' when it gets called, not when an instance of it has a method called on it. Here are the objects in main.py and how they relate to api_mock and db_mock in your test:
Api is a class : api_mock
Api() is an instance : api_mock.return_value
Api().call is an instance method : api_mock.return_value.call
Api().call() is a return value : api_mock.return_value.call.return_value
Db is a class : db_mock
Db() is an instance : db_mock.return_value
Db().data is an attribute : db_mock.return_value.data
test_main.py should therefore look like this:
import mock
from mypkg.main import MyLib
#mock.patch('mypkg.main.Db')
#mock.patch('mypkg.main.Api')
def test_main(api_mock, db_mock):
ml = MyLib()
api_mock.return_value.call.return_value = 'foo'
db_mock.return_value.data = 'some data' # we need this to test that the call to api_mock had the correct arguments.
assert ml.caller() == 'foo'
api_mock.return_value.call.assert_called_once_with('some data')
The first patch in Option 1 would work great for unit-testing db.py, because it gives the db module a mock version of mysqlclient. Similarly, #mock.patch('mypkg.api.soapclient') belongs in test_api.py.
I can't think of a way Option 2 could help you unit-test anything.
Edited: I was incorrectly referring to classes as modules. db.py and api.py are modules