I want to use a mock object to emulate the database on a insert operation.
For example, let's suppose I have a method like insert(1) that calls a db connection object (let's call it db_obj) and performs some insert into mytable (col1) values (%s) where %s is 1.
What I had in mind: create some mock object for db_obj that stores the value of col1, so when db_obj.insert(1) is called, the mocked db_obj stores this col1=1 and, then, I can just get the mocked object col1 value and assert that it's 1 as expected.
Does this approach makes sense? If so, how can I do this using pytest?
Here's an example of what I'm trying
from hub.scripts.tasks.notify_job_module import http_success
from hub.scripts.tasks.notify_job_module import proceed_to
from hub.core.connector.mysql_db import MySql
from unittest.mock import patch
import logging
def proceed_to(conn,a,b,c,d,e):
conn.run_update('insert into table_abc (a,b,c,d,e) values (%s,%s,%s,%s,%s)',[a,b,c,d,e])
class MySql:
# this is the real conn implementation
def connect(self):
# ... create the database connection here
def run_update(self, query, params=None):
# ... perform some insert into the database here
class TestMySql:
# this is the mock implementation I want to get rid of
def connect(self):
pass
def run_update(self, query, params=None):
self.result = params
def get_result(self):
return self.result
def test_proceed_to():
logger = logging.getLogger("")
conn = TestMySql() ## originally, my code uses MySql() here
conn.connect()
proceed_to(conn,1,'2',3,4,5)
assert conn.get_result()[1] == 4
Please notice that I had to replace MySql() with TestMySql(), so what I've done was to implement my own Mock object manually.
This way it works, but I feel like it's obviously not the best approach here. Why?
Because we're talking about mock objects, the definition of proceed_to is irrelevant here :-) the thing is: I had to implement TestMySql.get_result() and store the data I want in self.result to get the result I want, while MySql itself does not has a get_result() itself!
What I'd like to to is to avoid having to create my own mock object here and use some smarter approach here using unittest.mock
What you are testing is basically what arguments run_update is called with. You can just mock the connection and use assert_called_xxx methods on the mock, and if you want to check specific arguments instead of all arguments you can check call_args on the mock. Here is an example that matches your sample code:
#mock.patch("some_module.MySql")
def test_proceed_to(mocked):
# we need the mocked instance, not the class
sql_mock = mocked.return_value
conn = sql_mock.connect() # conn is now a mock - you also could have created a mock manually
proceed_to(conn, 1, '2', 3, 4, 5)
# assert that the function was called with the correct arguments
conn.run_update.assert_called_once()
# conn.run_update.assert_called_once_with() would compare all arguments
assert conn.run_update.call_args[0][1] == [1, '2', 3, 4, 5]
# call_args[0] are the positional arguments, so this checks the second positional argument
Related
I have a function than depends on db connection. This function has a lot of return statements, of this kind:
def do_something_with_data(db: Session, data: DataClass):
db = Session()
if condition1:
db.do_something1()
db.close()
return
if condition2:
db.do_something2()
db.close()
return
if condition3:
db.do_something3()
db.close()
return
...
After executing the function, I need to run db.close(), but because of the structure of the function this entry will have to be duplicated many times for each return as shown above.
So I made a decorator that passes the created session to the function and closes the session at the end of the execution of the function instead.
def db_depends(function_that_depends_on_db):
def inner(*args, **kwargs):
db = Session()
result = function_that_depends_on_db(db, *args, **kwargs)
db.close()
return result
return inner
#db_depends
def do_something_with_data(db: Session, data: DataClass):
if condition1:
db.do_something1()
return
if condition2:
db.do_something2()
return
if condition3:
db.do_something3()
return
...
All works great, but the fact, that user see two required arguments in definition, however there is only one (data) seems kinda dirty.
Is it possible to do the same thing without misleading people who will read the code or IDE hints?
Just have the function accept the Session parameter normally:
def do_something_with_data(db: Session, data: DataClass):
if condition1:
db.do_something1()
return
if condition2:
db.do_something2()
return
if condition3:
db.do_something3()
return
This allows the user to specify a Session explicitly, for example to reuse the same Session to do multiple things.
Yes, that doesn't close the Session. Because that is the responsibility of the calling code, since that's where the Session came from in the first place. After all, if the calling code wants to reuse a Session, then it shouldn't be closed.
If you want a convenience method to open a new, temporary Session for the call, you can easily do that using the existing decorator code:
do_something_in_new_session = db_depends(do_something_with_data)
But if we don't need to apply this logic to multiple functions, then "simple is better than complex" - just write an ordinary wrapper:
def do_something_in_new_session(data: DataClass):
db = Session()
result = do_something_with_data(db, data)
db.close()
return result
Either way, it would be better to write the closing logic using a with block, assuming your library supports it:
def do_something_in_new_session(data: DataClass):
with Session() as db:
return do_something_with_data(db, data)
Among other things, this ensures that .close is called even if an exception is raised in do_something_with_data.
If your DB library doesn't support that (i.e., the Session class isn't defined as a context manager - although that should only be true for very old libraries now), that's easy to fix using contextlib.closing from the standard library:
from contextlib import closing
def do_something_in_new_session(data: DataClass):
with closing(Session()) as db:
return do_something_with_data(db, data)
(And of course, if you don't feel the need to make a wrapper like that, you can easily use such a with block directly at the call site.)
I'm trying to patch a function in Pytest's pytest_sessionstart(). I was expecting the patch function to return {'SENTRY_DSN': "WRONG"}, however. I'm getting back <MagicMoc ... id='4342393248'> object in the test run.
import pytest
from unittest.mock import patch, Mock
def pytest_sessionstart(session):
"""
:type request: _pytest.python.SubRequest
:return:
"""
mock_my_func = patch('core.my_func')
mock_my_func.return_value = {'SENTRY_DSN': "WRONG"}
mock_my_func.__enter__()
def unpatch():
mock_my_func.__exit__()
This has been correctly answered by #gold_cy, so this is just an addition: as already mentioned, you are setting return_value to the patch object, not to the mock itself. The easiest way to correct is is to use instead:
from unittest.mock import patch
def pytest_sessionstart(session):
"""
:type request: _pytest.python.SubRequest
:return:
"""
mock_my_func = patch('core.my_func', return_value = {'SENTRY_DSN': "WRONG"})
mock_my_func.start()
This sets the return value to the mock without the need to create a separate Mock object.
Issue here seems to be that you are not properly configuring the Mock object. Given the code you have shown I am going under the assumption that you are calling some function in the following way:
with foobar() as fb:
# something happens with fb here
That call evaluates to to this essentially:
foobar().__enter__()
However, in the patching that you have shown, you have made a few critical mistakes.
You have not defined the return value of __enter__.
When you initialize a Mock object, it returns a brand new Mock object, therefore when you call __enter__ at the end of your function, it is returning a brand new object, not the one you originally created.
If I understand correctly you probably want something like this:
import pytest
from unittest.mock import patch, Mock
def pytest_sessionstart(session):
"""
:type request: _pytest.python.SubRequest
:return:
"""
mock_my_func = patch('core.my_func')
mock_context = Mock()
mock_context.return_value = {'SENTRY_DSN': "WRONG"}
mock_my_func.return_value.__enter__.return_value = mock_context
# this now returns `mock_context`
mock_my_func().__enter__()
Now mock_my_func().__enter__() returns mock_context which we can see works as expected when we do the following:
with mock_my_func() as mf:
print(mf())
>> {'SENTRY_DSN': 'WRONG'}
I am having issues properly patching an imported function in pytest. The function I want to patch is a function designed to do a large SQL fetch, so for speed I would like to replace this with reading a CSV file. Here is the code I currently have:
from data import postgres_fetch
import pytest
#pytest.fixture
def data_patch_market(monkeypatch):
test_data_path = os.path.join(os.path.dirname(__file__), 'test_data')
if os.path.exists(test_data_path):
mock_data_path = os.path.join(test_data_path, 'test_data_market.csv')
mock_data = pd.read_csv(mock_data_path)
monkeypatch.setattr(postgres_fetch, 'get_data_for_market', mock_data)
def test_mase(data_patch_market):
data = postgres_fetch.get_data_for_market(market_name=market,
market_level=market_level,
backtest_log_ids=log_ids,
connection=conn)
test_result= build_features.MASE(data)
However when I run this test I am getting a type error about calling a DataFrame:
TypeError: 'DataFrame' object is not callable
I know the csv can be read properly as I've tested that separately, so I assume something is wrong with how I am implementing the patch fixture, but I can't seem to work it out
Here, your call to monkeypatch.setattr is replacing any call to postgres_fetch.get_data_for_market with a call to mock_data.
This can't work since mock_data is not a function - its a DataFrame object.
Instead, in your call to monkeypatch.setattr, you need to pass in a function that returns the mocked data (i.e. the DataFrame object).
Hence, something like this should work:
#pytest.fixture
def data_patch_market(monkeypatch):
test_data_path = os.path.join(os.path.dirname(__file__), 'test_data')
if os.path.exists(test_data_path):
mock_data_path = os.path.join(test_data_path, 'test_data_market.csv')
mock_data = pd.read_csv(mock_data_path)
# The lines below are new - here, we define a function that will return the data we have mocked
def return_mocked(*args, **kwargs):
return mock_data
monkeypatch.setattr(postgres_fetch, 'get_data_for_market', return_mocked)
I am writing unit tests. I would like to mock the result of a function called on a mock object.
I have a class called OwnerAnalyzer which accepts an object called client in its constructor. Using this client, I can get owner details.
In my unit test, I want to pass a mock for this client and mock results from its get_owners method.
Here is what I have so far:
def test_get_owner_details(mock_datetime, monkeypatch):
mock_datetime.now.return_value.isoformat.return_value = MOCK_NOW
mock_client = mock.MagicMock()
mock_client.return_value.get_owners.return_value = ListOwnerDetails(
main_owner=OwnerDetails(name='test_owner', type='User'), secondary_owners=[])
owner_analyzer = OwnerAnalyzer(OWNER_NAME, client=mock_client)
owner_analyzer.analyze_owner(OWNER_NAME)
assert classUnderTest.owner_name == 'test_owner'
I don't think the mock value is being returned in the get_owners call because I get something like for main_owner
owner is : <MagicMock name='mock.get_owners().main_owner' id='140420863948896'>.
Thanks to #jonrsharpe for pointing me in the right direction.
I was able to get this working by updating my mock setup to -
mock_client.get_owners.return_value = ListOwnerDetails(
main_owner=OwnerDetails(name='test_owner', type='User'), secondary_owners=[])
I am in the process of learning unit testing, however I am struggling to understand how to mock functions for unit testing. I have reviewed many how-to's and examples but the concept is not transferring enough for me to use it on my code. I am hoping getting this to work on a actual code example I have will help.
In this case I am trying to mock isTokenValid.
Here is example code of what I want to mock.
<in library file>
import xmlrpc.client as xmlrpclib
class Library(object):
def function:
#...
AuthURL = 'https://example.com/xmlrpc/Auth'
auth_server = xmlrpclib.ServerProxy(AuthURL)
socket.setdefaulttimeout(20)
try:
if pull == 0:
valid = auth_server.isTokenValid(token)
#...
in my unit test file I have
import library
class Tester(unittest.TestCase):
#patch('library.xmlrpclib.ServerProxy')
def test_xmlrpclib(self, fake_xmlrpclib):
assert 'something'
How would I mock the code listed in 'function'? Token can be any number as a string and valid would be a int(1)
First of all, you can and should mock xmlrpc.client.ServerProxy; your library imports xmlrpc.client as a new name, but it is still the same module object so both xmlrpclib.ServerProxy in your library and xmlrpc.client.ServerProxy lead to the same object.
Next, look at how the object is used, and look for calls, the (..) syntax. Your library uses the server proxy like this:
# a call to create an instance
auth_server = xmlrpclib.ServerProxy(AuthURL)
# on the instance, a call to another method
valid = auth_server.isTokenValid(token)
So there is a chain here, where the mock is called, and the return value is then used to find another attribute that is also called. When mocking, you need to look for that same chain; use the Mock.return_value attribute for this. By default a new mock instance is returned when you call a mock, but you can also set test values.
So to test your code, you'd want to influence what auth_server.isTokenValid(token) returns, and test if your code works correctly. You may also want to assert that the correct URL is passed to the ServerProxy instance.
Create separate tests for different outcomes. Perhaps the token is valid in one case, not valid in another, and you'd want to test both cases:
class Tester(unittest.TestCase):
#patch('xmlrpc.client.ServerProxy')
def test_valid_token(self, mock_serverproxy):
# the ServerProxy(AuthURL) return value
mock_auth_server = mock_serverproxy.return_value
# configure a response for a valid token
mock_auth_server.isTokenValid.return_value = 1
# now run your library code
return_value = library.Library().function()
# and make test assertions
# about the server proxy
mock_serverproxy.assert_called_with('some_url')
# and about the auth_server.isTokenValid call
mock_auth_server.isTokenValid.assert_called_once()
# and if the result of the function is expected
self.assertEqual(return_value, 'expected return value')
#patch('xmlrpc.client.ServerProxy')
def test_invalid_token(self, mock_serverproxy):
# the ServerProxy(AuthURL) return value
mock_auth_server = mock_serverproxy.return_value
# configure a response; now testing for an invalid token instead
mock_auth_server.isTokenValid.return_value = 0
# now run your library code
return_value = library.Library().function()
# and make test assertions
# about the server proxy
mock_serverproxy.assert_called_with('some_url')
# and about the auth_server.isTokenValid call
mock_auth_server.isTokenValid.assert_called_once()
# and if the result of the function is expected
self.assertEqual(return_value, 'expected return value')
There are many mock attributes to use, and you can change your patch decorator usage a little as follows:
class Tester(unittest.TestCase):
def test_xmlrpclib(self):
with patch('library.xmlrpclib.ServerProxy.isTokenValid') as isTokenValid:
self.assertEqual(isTokenValid.call_count, 0)
# your test code calling xmlrpclib
self.assertEqual(isTokenValid.call_count, 1)
token = isTokenValid.call_args[0] # assume this token is valid
self.assertEqual(isTokenValid.return_value, 1)
You can adjust the code above to satisfy your requirements.