I'm writing unit tests, and I need to mock a method call so that in most of the cases it behaved as method itself except when argument gets a special value 'insert into'. Here is a simplified production code:
class CommandServer(object):
def __init__(self):
self.rowcount = None
def runSQL(self, sql):
print "Do something useful"
self.rowcount=5
return self
class Process(object):
def process(self):
cs = CommandServer()
cs.runSQL("create table tbl1(X VARCHAR2(10))")
r = cs.runSQL("insert into tbl1 select * from tbl2")
print "Number of rows: %s" % r.rowcount
p = Process()
p.process()
which prints
Do something useful
Do something useful
Number of rows: 5
I can make a mock version myself using the following code:
runSQL = CommandServer.runSQL
def runSQLPatch(self, sql):
if sql.lstrip().startswith('insert into'):
print "Patched version in use"
class res(object):
rowcount = -1
return res
else:
return runSQL(self, sql)
CommandServer.runSQL = runSQLPatch
p = Process()
p.process()
which prints
Do something useful
Patched version in use
Number of rows: -1
I want to use mock library to accomplish the same (I believe that this is the library included in python 3). How can I do that? (Python 2.6.2)
To be entirely clear, it's only included in python 3.3 (which I'm so happy to have learned, thank you!).
Otherwise, the pattern you could use is
from mock import patch
with patch.object(CommandServer, 'runSQL') as runSQL:
class res(object):
rowcount = -1
runSQL.return_value = res
p = Process()
p.process()
for c in runSQL.call_list:
assert c[1].lstrip().startswith('insert into') is True
But this would cover all cases, not just cases where you are sending 'insert into'. This might give you a hint as to where to look, but otherwise, I don't think what you're looking for is exactly possible with mock.
Related
Most of my unit testing experience is with Java and now I'm turning to Python. I need to test whether a method (from object B) gets called inside another method (in object A).
In Java the test method would have to pass a mock or spy version of B to A's constructor to be used when the B method is invoked. Do I need to do the same in Python? Or is there a simpler way? (I raise the possibility of the latter, because it seems, from what little I know, that Python is relatively relaxed about enforcing isolation between different components.)
Below is how I do this the "Java way." There are two Python files under test (for objects A and B) and a test program. Notice that object A's constructor had to be modified to accommodate testing.
obj_a.py
from obj_b import *
class ObjA:
def __init__(self, *args):
if len(args) > 0:
self.objb = args[0] # for testing
return
self.objb = ObjB()
def methodCallsB(self, x, y):
return self.objb.add(x, y)
obj_b.py
class ObjB:
def add(self, x, y):
return x + y
test.py
import unittest
from unittest.mock import patch, Mock
from obj_a import *
from obj_b import *
class TTest(unittest.TestCase):
#patch("obj_b.ObjB")
def test_shouldCallBThroughA(self, mockB):
# configure mock
mockB.add = Mock(return_value=137)
obja = ObjA(mockB)
# invoke test method
res = obja.methodCallsB(4, 7)
print("result: " + str(res))
# assess results
self.assertEqual(137, res)
mockB.add.assert_called_once()
args = mockB.add.call_args[0] # Python 3.7
print("args: " + str(args))
self.assertEqual((4, 7), args)
if __name__ =='__main__':
unittest.main()
Again, is there a simpler way to test that ObjB::add is called from ObjA?
Apart from the possible problems with the design, mentioned in the comment by #Alex, there is a couple of errors in using the mock.
First, you are mocking the wrong object. As in object_a you do from obj_b import * (which is bad style by the way - only import the objects you need), you need to patch the object reference imported into obj_b, e.g. obj_a.ObjB (see where to patch).
Second, you have to mock the method call on the instance instead of the class, e.g. mock mockB.return_value.add instead of mockB.add.
Your tests actually only work because you are not testing your real function, only your mock. If you do the patching correctly, there is no need to add that test-specific code in __init__.
So, put together, something like this should work:
obj_a.py
class ObjA:
def __init__(self):
self.objb = ObjB()
...
test.py
class TTest(unittest.TestCase):
#patch("obj_a.ObjB")
def test_shouldCallBThroughA(self, mockB):
# for convenience, store the mocked method
mocked_add = mockB.return_value.add
mocked_add.return_value = 137
obja = ObjA()
res = obja.methodCallsB(4, 7)
self.assertEqual(137, res)
mocked_add.assert_called_once()
args = mocked_add.call_args[0]
self.assertEqual((4, 7), args)
I have a python function with a lot of functionality and several inner functions. I want to mock out the return value of one of those functions. Is it possible to use the mock package to mock out the inner function?
Here's an example
def outer(values):
a = 1
def inner():
return np.mean(values)
if inner() == 1:
return None
return inner()
Ok it's a strange example, but what I want is to mock out inner() to return a certain value. I tried to mock with #mock.patch('outer.inner') and I tried #mock.patch.object(outer, 'inner'), but neither works. Is it possible to mock a closure?
As far as I've found so far the answer is "you can't". Disappointing, but actionable.
In my case I was able to mock out some other call such that the closure returned what I wanted. In the example above it would be like
def test_outer_mean_1(self):
with mock.patch('np.mean', return_value=1):
self.assertIsNone(outer(None))
def test_outer_mean_not_1(self):
with mock.patch('np.mean', return_value=2):
self.assertEqual(2, outer(None))
If anybody comes up with a better answer, I'd be eager to hear it.
related question Mocking a local variable of a function in python
Disclaimer: I'm not saying this is the right approach, mocking np.mean is much better.
I have come up with a workaround: the idea is to change the code of the function at run time and execute the new function.
Here is the code:
from _pytest._code import Code
def convert_function_in_function(func):
context = getattr(func, "__globals__", {})
code = Code.from_function(func)
source = code.source()
new_body = ["from unittest import mock", "new_mock = mock.MagicMock()"] + source.lines[0:2] + [
" inner=new_mock"] + source.lines[4:]
compiled = compile("\n".join(new_body), str(code.path), "exec")
exec(compiled, context)
return context['outer'], context['new_mock']
def test_outer_mean_specific_value():
new_outer, mock_inner = convert_function_in_function(outer)
mock_inner.return_value = 2
assert 2 == new_outer(5)
Explanation: convert_function_in_function makes the code to be
from unittest import mock
new_mock = mock.MagicMock()
def outer(values):
a = 1
inner=new_mock
if inner() == 1:
return None
return inner()
Then it returns the new function and the matching mock. You can then change the mock behaviour and call the new function.
I created a class to make my life easier while doing some integration tests involving workers and their contracts. The code looks like this:
class ContractID(str):
contract_counter = 0
contract_list = list()
def __new__(cls):
cls.contract_counter += 1
new_entry = super().__new__(cls, f'Some_internal_name-{cls.contract_counter:10d}')
cls.contract_list.append(new_entry)
return new_entry
#classmethod
def get_contract_no(cls, worker_number):
return cls.contract_list[worker_number-1] # -1 so WORKER1 has contract #1 and not #0 etc.
When I'm unit-testing the class, I'm using the following code:
from test_helpers import ContractID
#pytest.fixture
def get_contract_numbers():
test_string_1 = ContractID()
test_string_2 = ContractID()
test_string_3 = ContractID()
return test_string_1, test_string_2, test_string_3
def test_contract_id(get_contract_numbers):
assert get_contract_ids[0] == 'Some_internal_name-0000000001'
assert get_contract_ids[1] == 'Some_internal_name-0000000002'
assert get_contract_ids[2] == 'Some_internal_name-0000000003'
def test_contract_id_get_contract_no(get_contract_numbers):
assert ContractID.get_contract_no(1) == 'Some_internal_name-0000000001'
assert ContractID.get_contract_no(2) == 'Some_internal_name-0000000002'
assert ContractID.get_contract_no(3) == 'Some_internal_name-0000000003'
with pytest.raises(IndexError) as py_e:
ContractID.get_contract_no(4)
assert py_e.type == IndexError
However, when I try to run these tests, the second one (test_contract_id_get_contract_no) fails, because it does not raise the error as there are more than three values. Furthermore, when I try to run all my tests in my folder test/, it fails even the first test (test_contract_id), which is probably because I'm trying to use this function in other tests that run before this test.
After reading this book, my understanding of fixtures was that it provides objects as if they were never called before, which is obviously not the case here. Is there a way how to tell the tests to use the class as if it hasn't been used before anywhere else?
If I understand that correctly, you want to run the fixture as setup code, so that your class has exactly 3 instances. If the fixture is function-scoped (the default) it is indeed run before each test, which will each time create 3 new instances for your class. If you want to reset your class after the test, you have to do this yourself - there is no way pytest can guess what you want to do here.
So, a working solution would be something like this:
#pytest.fixture(autouse=True)
def get_contract_numbers():
test_string_1 = ContractID()
test_string_2 = ContractID()
test_string_3 = ContractID()
yield
ContractID.contract_counter = 0
ContractID.contract_list.clear()
def test_contract_id():
...
Note that I did not yield the test strings, as you don't need them in the shown tests - if you need them, you can yield them, of course. I also added autouse=True, which makes sense if you need this for all tests, so you don't have to reference the fixture in each test.
Another possibility would be to use a session-scoped fixture. In this case the setup would be done only once. If that is what you need, you can use this instead:
#pytest.fixture(autouse=True, scope="session")
def get_contract_numbers():
test_string_1 = ContractID()
test_string_2 = ContractID()
test_string_3 = ContractID()
yield
I have a Python application which uses both SQLite and Postgresql. It has a connector class for each database:
class PostgresqlDatabase(Database):
...
class SQLite(Database):
...
Both class share the same methods and logic, and the only thing that differentiate them the SQL is the parametrization of SQL queries. Most of the SQL queries are even identical, e.g. both have a method called _get_tag:
# postgresql method with %s
def _get_tag(self, tagcipher):
sql_search = "SELECT ID FROM TAG WHERE DATA = %s"
self._cur.execute(sql_search, ([tagcipher]))
rv = self._cur.fetchone()
return rv
# sqlite method with ?
def _get_tag(self, tagcipher):
sql_search = "SELECT ID FROM TAG WHERE DATA = ?"
self._cur.execute(sql_search, ([tagcipher]))
rv = self._cur.fetchone()
return rv
To really make it clear, the classes have exact identical method names. The SQL queries differ in each method. So what is my problem?
I find maintaining both classes annoying, and I feel a common class would benefit the code in the long run.
However, creating a common class, would create a complex code. The __init__ would probably have to initialize the correct underlying cursor. This would create a small starting overhead, and small performance penalty if for example I would lookup the correct string every time, e.g.
#property:
def sql_search(self):
return "SELECT ID FROM TAG WHERE DATA = {}".format(
'?' if self.db == 'SQLite' else '%s')
def _get_tag(self, tagcipher):
self._cur.execute(self.sql_search, ([tagcipher]))
rv = self._cur.fetchone()
return rv
I am also afraid this approach would be also harder to understand when first looking at it.
Leaving my personal example, I would like to know what is the most acceptable way here.
Should I keep maintaining both classes or write one more complicated class that does it all?
Is there a general rule of thumb?
It seems that inheritance is what you're looking for. It is a key feature of [OOP][1] (Another one in Java, Yes Java, but I like their docs).
As thefourtheye said in the comments, I believe you should move the identical methods into one class (in other words, delete one set of the identical methods).
Here is a very quick example:
class Connector(Database):
"""This is a super class, common elements go here"""
def __init__(self):
self.sql_search = "SELECT ID FROM TAG WHERE DATA = %s"
self.common_varialbe = None #placeholder
Database.__init__(self) #add necessary arguments
def _get_tag(self, tagcipher, wildcard):
#replace the sql search string with the wildcard.
self._cur.execute(self.sql_search % (wildcard) , ([tagcipher]))
rv = self._cur.fetchone()
return rv
def some_common_method(self, uncommon_value):
self.common_variable = uncommon_value
class Postgresql(Connector):
"""postgresql subclass using %s.
unique postgresql elements go here"""
def __init__(self):
#initialise the superclass
Connector.__init__(self)
self.wildcard = '%s'
self.uncommon_value = 'py hole'
#other unique values go here
class Sqlite(Connector):
"""etc"""
def __init__(self):
#initialise the superclass
Connector.__init__(self)
self.wildcard = '?'
#other unique values go here
#other methods
Even from this example you can see some redundancy, but was included to show how things could be split up if necessary. With this class, i can:
>>>import connector
>>>sqlite = connector.Sqlite()
>>>sqlite.wilcard
`?`
>>>sqlite.sql_search
`SELECT ID FROM TAG WHERE DATA = %s`
>>>sqlite.sql_search % sqlite.wildcard
`SELECT ID FROM TAG WHERE DATA = ?`
If they truly differ only by strings, only one subclass is needed. you can use dict()s to store the unique bits:
class Connector(Database):
def __init__(self,type):
#describe all types in this dict
types = {"sqlite":"?",
"postgre":"%s"}
#Database.__init__(self) as necessary
self.sql_search = "SELECT ID FROM TAG WHERE DATA = %s" % types[type]
def _get_tag(self, tagcipher):
#replace the sql search string with the wildcard.
self._cur.execute(self.sql_search, ([tagcipher]))
rv = self._cur.fetchone()
return rv
So with this class:
>>>c = connector.Connector('sqlite')
>>>c.sql_search
`SELECT ID FROM TAG WHERE DATA = ?`
As long as they are properly inheriting from the Database superclass, subclasses will share its cursor when Database.__init__(*args) is called
I'm creating a program that uses the Twisted module and callbacks.
However, I keep having problems because the asynchronous part goes wrecked.
I have learned (also from previous questions..) that the callbacks will be executed at a certain point, but this is unpredictable.
However, I have a certain program that goes like
j = calc(a)
i = calc2(b)
f = calc3(c)
if s:
combine(i, j, f)
Now the boolean s is set by a callback done by calc3. Obviously, this leads to an undefined error because the callback is not executed before the s is needed.
However, I'm unsure how you SHOULD do if statements with asynchronous programming using Twisted. I've been trying many different things, but can't find anything that works.
Is there some way to use conditionals that require callback values?
Also, I'm using VIFF for secure computations (which uses Twisted): VIFF
Maybe what you're looking for is twisted.internet.defer.gatherResults:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, f)):
if s:
return combine(i, j, f)
d.addCallback(calculated)
However, this still has the problem that s is undefined. I can't quite tell how you expect s to be defined. If it is a local variable in calc3, then you need to return it so the caller can use it.
Perhaps calc3 looks something like this:
def calc3(argument):
s = bool(argument % 2)
return argument + 1
So, instead, consider making it look like this:
Calc3Result = namedtuple("Calc3Result", "condition value")
def calc3(argument):
s = bool(argument % 2)
return Calc3Result(s, argument + 1)
Now you can rewrite the calling code so it actually works:
It's sort of unclear what you're asking here. It sounds like you know what callbacks are, but if so then you should be able to arrive at this answer yourself:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, calc3result)):
if calc3result.condition:
return combine(i, j, calc3result.value)
d.addCallback(calculated)
Or, based on your comment below, maybe calc3 looks more like this (this is the last guess I'm going to make, if it's wrong and you'd like more input, then please actually share the definition of calc3):
def _calc3Result(result, argument):
if result == "250":
# SMTP Success response, yay
return Calc3Result(True, argument)
# Anything else is bad
return Calc3Result(False, argument)
def calc3(argument):
d = emailObserver("The argument was %s" % (argument,))
d.addCallback(_calc3Result)
return d
Fortunately, this definition of calc3 will work just fine with the gatherResults / calculated code block immediately above.
You have to put if in the callback. You may use Deferred to structure your callback.
As stated in previous answer - the preocessing logic should be handled in callback chain, below is simple code demonstration how this could work. C{DelayedTask} is a dummy implementation of a task which happens in the future and fires supplied deferred.
So we first construct a special object - C{ConditionalTask} which takes care of storring the multiple results and servicing callbacks.
calc1, calc2 and calc3 returns the deferreds, which have their callbacks pointed to C{ConditionalTask}.x_callback.
Every C{ConditionalTask}.x_callback does a call to C{ConditionalTask}.process which checks if all of the results have been registered and fires on a full set.
Additionally - C{ConditionalTask}.c_callback sets a flag of wheather or not the data should be processed at all.
from twisted.internet import reactor, defer
class DelayedTask(object):
"""
Delayed async task dummy implementation
"""
def __init__(self,delay,deferred,retVal):
self.deferred = deferred
self.retVal = retVal
reactor.callLater(delay, self.on_completed)
def on_completed(self):
self.deferred.callback(self.retVal)
class ConditionalTask(object):
def __init__(self):
self.resultA=None
self.resultB=None
self.resultC=None
self.should_process=False
def a_callback(self,result):
self.resultA = result
self.process()
def b_callback(self,result):
self.resultB=result
self.process()
def c_callback(self,result):
self.resultC=result
"""
Here is an abstraction for your "s" boolean flag, obviously the logic
normally would go further than just setting the flag, you could
inspect the result variable and do other strange stuff
"""
self.should_process = True
self.process()
def process(self):
if None not in (self.resultA,self.resultB,self.resultC):
if self.should_process:
print 'We will now call the processor function and stop reactor'
reactor.stop()
def calc(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a)
return deferred
def calc2(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*2)
return deferred
def calc3(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*3)
return deferred
def main():
conditional_task = ConditionalTask()
dFA = calc(1)
dFB = calc2(2)
dFC = calc3(3)
dFA.addCallback(conditional_task.a_callback)
dFB.addCallback(conditional_task.b_callback)
dFC.addCallback(conditional_task.c_callback)
reactor.run()