I'm using the following code in my testing framework:
testModules = ["test_foo", "test_bar"]
suite = unittest.TestLoader().loadTestsFromNames(testModules)
runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
results = runner.run(suite)
return results.wasSuccessful()
Is there a way to make the reporting (runner.run?) abort after the first failure to prevent excessive verbosity?
Nine years after the question was asked, this is still one of the top search results for "python unit test fail early" and, as I discovered when looking at the other search results, these answers are no longer correct for more recent versions of the unittest module.
The documentation for the unittest module https://docs.python.org/3/library/unittest.html#command-line-options and https://docs.python.org/2.7/library/unittest.html#command-line-options show that there is an argument, failfast=True, that can be added to unittest.main, or equivalently a command line option, -f, or --failfast, to stop the test run on the first error or failure. This option was added in version 2.7. Using that option is a lot easier than the previously-necessary workarounds suggested in the other answers.
That is, simply change your
unittest.main()
to
unittest.main(failfast=True)
It's a feature. If you want to override this, you'll need to subclass TestCase and/or TestSuite classes and override logic in the run() method.
P.S.:
I think you have to subclass unittest.TestCase and override method run() in your class:
def run(self, result=None):
if result is None: result = self.defaultTestResult()
result.startTest(self)
testMethod = getattr(self, self._testMethodName)
try:
try:
self.setUp()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
return
ok = False
try:
testMethod()
ok = True
except self.failureException:
result.addFailure(self, self._exc_info())
result.stop()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
result.stop()
try:
self.tearDown()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
ok = False
if ok: result.addSuccess(self)
finally:
result.stopTest(self)
(I've added two result.stop() calls to the default run definition).
Then you'll have to modify all your testcases to make them subclasses of this new class, instead of unittest.TestCase.
WARNING: I didn't test this code. :)
Based on Eugene's guidance, I've come up with the following:
class TestCase(unittest.TestCase):
def run(self, result=None):
if result.failures or result.errors:
print "aborted"
else:
super(TestCase, self).run(result)
While this works fairly well, it's a bit annoying that each individual test module has to define whether it wants to use this custom class or the default one (a command-line switch, similar to py.test's --exitfirst, would be ideal)...
Building on AnC's answer, this is what I'm using...
def aborting_run(self, result=None):
if result.failures or result.errors:
print "aborted"
else:
original_run(self, result)
original_run = unittest.TestCase.run
unittest.TestCase.run = aborting_run
Related
I have a function in a python program which does a function call twice:
def add_user(uname,dserver,pwd,dinstance,proc1, query1):
db_conn = db_connect(uname,dserver,pwd,dinstance)
if db_conn is not_conn:
print("Failed to connect")
sys.exit(-1)
db_conn.run_proc(proc1)
if db_conn.error_msg:
print("Failed procedure")
sys.exit(-1)
db_conn.run_query(query1)
if db_conn.err_msg:
print("Failed query")
sys.exit(-1)
Now the unit test is as follows:
#patch('mydir.proj_dir.db_execu.db_connect')
def test_add_user(self, mock_conn):
mock_conn.return_value.not_conn.return_value = True
mock_conn.return_value.run_proc.return_value = True
mock_conn.return_value.run_query.return_value = True
mock_conn.return_value.err_msg.side_effect= [True, False]
with self.assertRaises(SystemExit):
add_user(name,dserver,pwd,dinstance,proc1, query1)
print("failed query")
My objective is to test the second error condition. But after adding side_effect it only going to first condition which is displaying as "Failed procedure". I want to test "Failed Query" condition. First two error conditions I have tested but the third one is failing and calling the second condition always. Please advise.
It's not super clear what you're asking exactly. You're calling add_user only once so you're only testing a single "path", you need to call it multiple times to test multiple "paths" through it.
That aside, there's a huge issue with your understanding of mock return_value and side_effect configure mocks as callables, when accessing attributes but not calling them, you're just getting the next mock in the chain, you're not using any of your configuration. mock also supports configuring some of the "magic methods" of the data model, but it does not support configuring __getattr__ (the "simple" attribute access) as that's used internally by mock.
As a result, you should only use mock to patch / replace db_connect to return a pseudo-connection, but rather than a mock that pseudo-connection should be a fake: an object you build yourself which looks like a connection but behaves however you want it e.g.
class PseudoConnection:
def __init__(self, fail_proc=False, fail_query=False):
self._fail_proc = fail_proc
self._fail_query = fail_query
self.err_msg = None # or error_msg, or both
def run_proc(self, proc):
self.err_msg = self._fail_proc
def run_query(self, query):
self.err_msg = self._fail_query
Then you just configure mock_conn.return_value = PseudoConnection(True, False) to test the first case and mock_conn.return_value = PseudoConnection(False, True) for the second.
Also as hinted by the comment you're accessing both error_msg and err_msg, and only attempting to configure err_msg. I don't know which is right, but I doubt both are.
This demo function I want to test is pretty straight forward.
def is_email_deliverable(email):
try:
return external.verify(email)
except Exception:
logger.error("External failed failed")
return False
This function uses an external service which I want to mock out.
But I can't figure out how to throw an exception from external.verify(email) i.e. how to force the except clause to be executed.
My attempt:
#patch.object(other_module, 'external')
def test_is_email_deliverable(patched_external):
def my_side_effect(email):
raise Exception("Test")
patched_external.verify.side_effects = my_side_effect
# Or,
# patched_external.verify.side_effects = Exception("Test")
# Or,
# patched_external.verify.side_effects = Mock(side_effect=Exception("Test"))
assert is_email_deliverable("some_mail#domain.com") == False
This question claims to have the answer, but didn't work for me.
You have used side_effects instead of side_effect.
Its something like this
#patch.object(Class, "attribute")
def foo(attribute):
attribute.side_effect = Exception()
# Other things can go here
BTW, its not good approach to catch all the Exception and handle according to it.
You can set the side_effect value to None.
I'm new to Python. I need to unit test the except part of a try-except statement in python. I'm using pytest. The problem is that I don't know how to force the try part to raise an exception. Here is my code:
try:
if master_bill_to is False:
master.update(
dbsession,
company_id=master.company_id,
)
except Exception as e:
dbsession.rollback()
raise Conflict(e.message)
The master.update method is called to make an update to the database. But how do I mock this code so that it somehow raises an exception in the try portion?
I'm trying to use monkeypatch with this code. The master object is an instance of the BillTo class so I'm thinking of putting that as the first parameter to monkeypatch.setattr.
def test_create_bill_to_fails_when_master_update_fails(dbsession, invoice_group1, company1,
monkeypatch):
def raise_flush_error():
raise FlushError
context = TestContext()
monkeypatch.setattr(BillTo, 'update', raise_flush_error)
with pytest.raises(FlushError):
create_bill_to(
context,
dbsession=dbsession,
invoice_group_id=invoice_group1.id,
company_id=company1.id,
)
But for some reason, the error is not raised.
Use mock library and side_effect to throw Exception during test case
Mock master and raise an exception in the update method.
Ok, I figured it out. I learned that you must pass parameters to the method called by monkeypatch. These parameters must match the signature of the method being replaced or mocked. Actually I also renamed the method with a prefix fake_ to denote the mocking. Here is what I did:
#staticmethod
def fake_update_flush_error(dbsession, company_id=None, address_id=None, proportion=None,
company_name=None, receiver_name=None, invoice_delivery_method=None,
invoice_delivery_text=None, master_bill_to=False):
raise FlushError
def test_create_bill_to_fails_when_master_update_fails(dbsession, invoice_group1, company1,
bill_to1, monkeypatch):
context = TestContext()
monkeypatch.setattr(BillTo, 'update', fake_update_flush_error)
with pytest.raises(Conflict):
create_bill_to(
context,
dbsession=dbsession,
invoice_group_id=invoice_group1.id,
company_id=company1.id,
address_id=None,
...
)
The BillTo.update method needed all those parameters.
I'm using Python's unittest with pytest for integration testing a library against a third-party API.
Some of the API calls are temporarily returning an error which raises a specific exception in my code. This behaviour is fine in the code.
However, rather than having the tests fail, I'd rather skip these temporary errors.
I have over 150 tests. Rather than rewriting each and every test like this:
class TestMyLibrary(unittest.TestCase):
def test_some_test(self):
try:
// run the test as normal
// assert the normal behaviour
except SomeException:
// skip the test
def test_some_other_test(self):
try:
// run the test as normal
// assert the normal behaviour
except SomeException:
// skip the test
Can I rather wrap them all somehow at the class level, or similar?
If you expect this exception why don't you check its raised when it should?
You can use :
pytest.raises(Exceptiontype, Foo())
This can be done with a decorator. For example:
def handle_lastfm_exceptions(f):
def wrapper(*args, **kw):
try:
return f(*args, **kw)
except pylast.WSError as e:
if (str(e) == "Invalid Method - "
"No method with that name in this package"):
msg = "Ignore broken Last.fm API: " + str(e)
print(msg)
pytest.skip(msg)
else:
raise(e)
return wrapper
And then decorate the problematic functions:
class TestMyLibrary(unittest.TestCase):
#handle_lastfm_exceptions
def test_some_bad_test(self):
// run the test as normal
// assert the normal behaviour
def test_some_good_test(self):
// run the test as normal
// assert the normal behaviour
had the same problem (instable 3rd party library, waiting for fix...). ended up with something like this:
def pytest_runtest_makereport(item, call):
from _pytest.runner import pytest_runtest_makereport as orig_pytest_runtest_makereport
tr = orig_pytest_runtest_makereport(item, call)
if call.excinfo is not None:
if call.excinfo.type == SomeExceptionFromLibrary:
tr.outcome = 'skipped'
tr.wasxfail = "reason: SomeExceptionFromLibrary. shame on them..."
return tr
works like a charm
I'm trying to unit test some code that looks like this:
def main():
parser = optparse.OptionParser(description='This tool is cool', prog='cool-tool')
parser.add_option('--foo', action='store', help='The foo option is self-explanatory')
options, arguments = parser.parse_args()
if not options.foo:
parser.error('--foo option is required')
print "Your foo is %s." % options.foo
return 0
if __name__ == '__main__':
sys.exit(main())
With code that looks like this:
#patch('optparse.OptionParser')
def test_main_with_missing_p4clientsdir_option(self, mock_optionparser):
#
# setup
#
optionparser_mock = Mock()
mock_optionparser.return_value = optionparser_mock
options_stub = Mock()
options_stub.foo = None
optionparser_mock.parse_args.return_value = (options_stub, sentinel.arguments)
def parser_error_mock(message):
self.assertEquals(message, '--foo option is required')
sys.exit(2)
optionparser_mock.error = parser_error_mock
#
# exercise & verify
#
self.assertEquals(sut.main(), 2)
I'm using Michael Foord's Mock, and nose to run the tests.
When I run the test, I get:
File "/Users/dspitzer/Programming/Python/test-optparse-error/tests/sut_tests.py", line 27, in parser_error_mock
sys.exit(2)
SystemExit: 2
----------------------------------------------------------------------
Ran 1 test in 0.012s
FAILED (errors=1)
The problem is that OptionParser.error does a sys.exit(2), and so main() naturally relies on that. But nose or unittest detects the (expected) sys.exit(2) and fails the test.
I can make the test pass by adding "return 2" under the parser.error() call in main() and removing the sys.exit() call from parser_error_mock(), but I find it distasteful to modify the code under test to allow a test to pass. Is there a better solution?
Update: df's answer works, although the correct call is "self.assertRaises(SystemExit, sut.main)".
Which means the test passes whatever the number is in the sys.exit() in parser_error_mock(). Is there any way to test for the exit code?
BTW, the test is more robust if I add:
self.assertEquals(optionparser_mock.method_calls, [('add_option', ('--foo',), {'action': 'store', 'help': 'The foo option is self-explanatory'}), ('parse_args', (), {})])
at the end.
Update 2: I can test for the exit code by replacing "self.assertRaises(SystemExit, sut.main)" with:
try:
sut.main()
except SystemExit, e:
self.assertEquals(type(e), type(SystemExit()))
self.assertEquals(e.code, 2)
except Exception, e:
self.fail('unexpected exception: %s' % e)
else:
self.fail('SystemExit exception expected')
Will this work instead of assertEquals?
self.assertRaises(SystemExit, sut.main, 2)
This should catch the SystemExit exception and prevent the script from terminating.
As noted in my updates to my question, I had to modify dF's answer to:
self.assertRaises(SystemExit, sut.main)
...and I came up with a few longer snippet to test for the exit code.
[Note: I accepted my own answer, but I will delete this answer and accept dF's if he updates his.]
Probably this question contains some new information:
Java: How to test methods that call System.exit()?
Add #raises to your test function. I.e:
from nose.tools import raises
#raises(SystemExit)
#patch('optparse.OptionParser')
def test_main_with_missing_p4clientsdir_option(self, mock_optionparser):
# Setup
...
sut.main()