Develop Reference

How do you show an error message when a test does not throw an expected exception?

I am new to Python. I wanted to test if my code threw an exception. I got the code from How do you test that a Python function throws an exception?
import mymod
import unittest
class MyTestCase(unittest.TestCase):
def test1(self):
self.assertRaises(SomeCoolException, mymod.myfunc, compulsory_argument)
Now, I also want to display a message if the exception is not thrown. How do I do that? The Python documentation does not mention it clearly. I added the message after "compulsory_argument" and it failed.
I tried the first answer with modifications and got an exception. What is my mistake here?
import unittest
def sayHelloTo(name):
print("Hello " + name)
class MyTestCase(unittest.TestCase):
def test1(self):
person = "John"
with self.assertRaises(Exception, "My insightful message"):
sayHelloTo(person)
Error:
Error
Traceback (most recent call last):
File "C:\tests\tester.py", line 9, in test1
with self.assertRaises(Exception, "My insightful message"):
AttributeError: __exit__

As of Python 3.3, assertRaises can be used as a context manager with a message:
import unittest
def sayHelloTo(name):
print("Hello " + name)
class MyTestCase(unittest.TestCase):
def test1(self):
person = "John"
with self.assertRaises(Exception, msg="My insightful message"):
sayHelloTo(person)
if __name__ == "__main__":
unittest.main()
It results in
Hello John
F
======================================================================
FAIL: test1 (__main__.MyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "r.py", line 10, in test1
sayHelloTo(person)
AssertionError: Exception not raised : My insightful message
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)

Now, I also want to display a message if the exception is not thrown. How do I do that ?
The general philosophy of unittest is for the tests to be silent when they succeed and only become verbose when they fail. Accordingly, the API provides a "msg" keyword argument for the unsuccessful case but does not offer an alternative for the successful case.
That said, another part of the philosophy works in your favor. In general, test cases internally raise an exception when a testcase fails. That means that if you want to display a message when there is a success, you just add another statement after the test:
with self.assertRaises(TypeError, msg='Oh no, I did not get a TypeError')
somecode()
logging.info('Yippee, we got a TypeError!') # Runs after a successful test

Can't catch mocked exception because it doesn't inherit BaseException

I'm working on a project that involves connecting to a remote server, waiting for a response, and then performing actions based on that response. We catch a couple of different exceptions, and behave differently depending on which exception is caught. For example:
def myMethod(address, timeout=20):
try:
response = requests.head(address, timeout=timeout)
except requests.exceptions.Timeout:
# do something special
except requests.exceptions.ConnectionError:
# do something special
except requests.exceptions.HTTPError:
# do something special
else:
if response.status_code != requests.codes.ok:
# do something special
return successfulConnection.SUCCESS
To test this, we've written a test like the following
class TestMyMethod(unittest.TestCase):
def test_good_connection(self):
config = {
'head.return_value': type('MockResponse', (), {'status_code': requests.codes.ok}),
'codes.ok': requests.codes.ok
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.SUCCESS
)
def test_bad_connection(self):
config = {
'head.side_effect': requests.exceptions.ConnectionError,
'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
If I run the function directly, everything happens as expected. I even tested by adding raise requests.exceptions.ConnectionError to the try clause of the function. But when I run my unit tests, I get
ERROR: test_bad_connection (test.test_file.TestMyMethod)
----------------------------------------------------------------
Traceback (most recent call last):
File "path/to/sourcefile", line ###, in myMethod
respone = requests.head(address, timeout=timeout)
File "path/to/unittest/mock", line 846, in __call__
return _mock_self.mock_call(*args, **kwargs)
File "path/to/unittest/mock", line 901, in _mock_call
raise effect
my.package.requests.exceptions.ConnectionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "Path/to/my/test", line ##, in test_bad_connection
mypackage.myMethod('some_address',
File "Path/to/package", line ##, in myMethod
except requests.exceptions.ConnectionError:
TypeError: catching classes that do not inherit from BaseException is not allowed
I tried to change the exception I was patching in to BaseException and I got a more or less identical error.
I've read https://stackoverflow.com/a/18163759/3076272 already, so I think it must be a bad __del__ hook somewhere, but I'm not sure where to look for it or what I can even do in the mean time. I'm also relatively new to unittest.mock.patch() so it's very possible that I'm doing something wrong there as well.
This is a Fusion360 add-in so it is using Fusion 360's packaged version of Python 3.3 - as far as I know it's a vanilla version (i.e. they don't roll their own) but I'm not positive of that.

I could reproduce the error with a minimal example:
foo.py:
class MyError(Exception):
pass
class A:
def inner(self):
err = MyError("FOO")
print(type(err))
raise err
def outer(self):
try:
self.inner()
except MyError as err:
print ("catched ", err)
return "OK"
Test without mocking :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
a = foo.A()
self.assertEquals("OK", a.outer())
Ok, all is fine, both test pass
The problem comes with the mocks. As soon as the class MyError is mocked, the expect clause cannot catch anything and I get same error as the example from the question :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
with unittest.mock.patch('foo.MyError'):
a = exc2.A()
self.assertEquals("OK", a.outer())
Immediately gives :
ERROR: test_outer (__main__.FooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "...\foo.py", line 11, in outer
self.inner()
File "...\foo.py", line 8, in inner
raise err
TypeError: exceptions must derive from BaseException
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#78>", line 8, in test_outer
File "...\foo.py", line 12, in outer
except MyError as err:
TypeError: catching classes that do not inherit from BaseException is not allowed
Here I get a first TypeErrorthat you did not have, because I am raising a mock while you forced a true exception with 'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError in config. But the problem remains that the except clause tries to catch a mock.
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this but in your example, you only need to patch requests.head. So I think that this should work :
def test_bad_connection(self):
with mock.patch('path.to.my.package.requests.head',
side_effect=requests.exceptions.ConnectionError):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
That is : only patch the head method with the exception as side effect.

I just ran into the same issue while trying to mock sqlite3 (and found this post while looking for solutions).
What Serge said is correct:
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this
My solution was to mock the entire module, then set the mock attribute for the exception to be equal to the exception in the real class, effectively "un-mocking" the exception. For example, in my case:
#mock.patch(MyClass.sqlite3)
def test_connect_fail(self, mock_sqlite3):
mock_sqlite3.connect.side_effect = sqlite3.OperationalError()
mock_sqlite3.OperationalError = sqlite3.OperationalError
self.assertRaises(sqlite3.OperationalError, MyClass, self.db_filename)
For requests, you could assign exceptions individually like this:
mock_requests.exceptions.ConnectionError = requests.exceptions.ConnectionError
or do it for all of the requests exceptions like this:
mock_requests.exceptions = requests.exceptions
I don't know if this is the "right" way to do it, but so far it seems to work for me without any issue.

For those of us who need to mock an exception and can't do that by simply patching head, here is an easy solution that replaces the target exception with an empty one:
Say we have a generic unit to test with an exception we have to have mocked:
# app/foo_file.py
def test_me():
try:
foo()
return "No foo error happened"
except CustomError: # <-- Mock me!
return "The foo error was caught"
We want to mock CustomError but because it is an exception we run into trouble if we try to patch it like everything else. Normally, a call to patch replaces the target with a MagicMock but that won't work here. Mocks are nifty, but they do not behave like exceptions do. Rather than patching with a mock, let's give it a stub exception instead. We'll do that in our test file.
# app/test_foo_file.py
from mock import patch
# A do-nothing exception we are going to replace CustomError with
class StubException(Exception):
pass
# Now apply it to our test
#patch('app.foo_file.foo')
#patch('app.foo_file.CustomError', new_callable=lambda: StubException)
def test_foo(stub_exception, mock_foo):
mock_foo.side_effect = stub_exception("Stub") # Raise our stub to be caught by CustomError
assert test_me() == "The error was caught"
# Success!
So what's with the lambda? The new_callable param calls whatever we give it and replaces the target with the return of that call. If we pass our StubException class straight, it will call the class's constructor and patch our target object with an exception instance rather than a class which isn't what we want. By wrapping it with lambda, it returns our class as we intend.
Once our patching is done, the stub_exception object (which is literally our StubException class) can be raised and caught as if it were the CustomError. Neat!

I faced a similar issue while trying to mock the sh package. While sh is very useful, the fact that all methods and exceptions are defined dynamically make it more difficult to mock them. So following the recommendation of the documentation:
import unittest
from unittest.mock import Mock, patch
class MockSh(Mock):
# error codes are defined dynamically in sh
class ErrorReturnCode_32(BaseException):
pass
# could be any sh command
def mount(self, *args):
raise self.ErrorReturnCode_32
class MyTestCase(unittest.TestCase):
mock_sh = MockSh()
#patch('core.mount.sh', new=mock_sh)
def test_mount(self):
...

I just ran into the same problem when mocking struct.
I get the error:
TypeError: catching classes that do not inherit from BaseException is not allowed
When trying to catch a struct.error raised from struct.unpack.
I found that the simplest way to get around this in my tests was to simply set the value of the error attribute in my mock to be Exception. For example
The method I want to test has this basic pattern:
def some_meth(self):
try:
struct.unpack(fmt, data)
except struct.error:
return False
return True
The test has this basic pattern.
#mock.patch('my_module.struct')
def test_some_meth(self, struct_mock):
'''Explain how some_func should work.'''
struct_mock.error = Exception
self.my_object.some_meth()
struct_mock.unpack.assert_called()
struct_mock.unpack.side_effect = struct_mock.error
self.assertFalse(self.my_object.some_meth()
This is similar to the approach taken by #BillB, but it is certainly simpler as I don't need to add imports to my tests and still get the same behavior. To me it would seem this is the logical conclusion to the general thread of reasoning in the answers here.

Use patch.object to partially mock a class.
My use case:
import unittest
from unittest import mock
import requests
def test_my_function(self):
response = mock.MagicMock()
response.raise_for_status.side_effect = requests.HTTPError
with mock.patch.object(requests, 'get', return_value=response):
my_function()

Python unittest assertraises error

I had this weird trouble running my unittest in Python:
I used assertRaises, and running the unittest raised the correct exception, but the test still failed. Ok I cannot really explain it, please see the traceback for yourself:
Error
Traceback (most recent call last):
File "/Users/chianti/PycharmProjects/Programming_Project/Part1and4/Part1and4Test.py", line 32, in test_non_alpha_name
self.assertRaises(RestNameContainNonAlphaError, RestaurantName(self.non_alpha_name))
File "/Users/chianti/PycharmProjects/Programming_Project/Part1and4/InputCheck.py", line 29, in __init__
raise RestNameContainNonAlphaError('There are non alphabetic characters that I can not recognize!')
RestNameContainNonAlphaError: There are non alphabetic characters that I can not recognize!
Error
Traceback (most recent call last):
File "/Users/chianti/PycharmProjects/Programming_Project/Part1and4/Part1and4Test.py", line 24, in test_non_string_name
self.assertRaises(InputNotStringError, RestaurantName, self.non_string_name)
File "/Users/chianti/anaconda/lib/python2.7/unittest/case.py", line 473, in assertRaises
callableObj(*args, **kwargs)
File "/Users/chianti/PycharmProjects/Programming_Project/Part1and4/InputCheck.py", line 33, in __init__
raise InputNotStringError('Not String! The input is supposed to be a string type!')
InputNotStringError: Not String! The input is supposed to be a string type!
Why ?????????? Any ideas are appreciated !!! THANK YOU
Here is my unittest:
class RestaurantNameTests(unittest.TestCase):
def setUp(self):
self.non_string_name = 123
self.valid_name = 'Italian rest '
self.non_alpha_name = 'valid ** n'
def tearDown(self):
self.non_string_name = None
self.valid_name = None
self.non_alpha_name = None
def test_non_string_name(self):
with self.assertRaises(InputNotStringError):
RestaurantName(self.non_string_name)
def test_valid_name(self):
self.assertEqual(RestaurantName(self.valid_name).__str__(), 'Italian rest')
def test_non_alpha_name(self):
self.assertRaises(RestNameContainNonAlphaError, RestaurantName(self.non_alpha_name))
If you need to see the definition of RestaurantName, here it is:
class RestaurantName():
def __init__(self, input_contents):
self.name = input_contents
if IsValidString(self.name):
self.no_space_name = self.name.replace(' ', '')
if str.isalpha(self.no_space_name):
pass
else:
raise RestNameContainNonAlphaError('There are non alphabetic characters that I can not recognize!')
else:
raise InputNotStringError('Not String! The input is supposed to be a string type!')
def __repr__(self):
return 'RestaurantName(%s)' % self.name.strip()
def __str__(self):
return self.name.strip()
Thanks again

The traceback doesn't match your description of the problem (nor your code FWIW). The error you get is in test_non_alpha_name() which you didn't post but from your error message looks like:
self.assertRaises(
RestNameContainNonAlphaError,
RestaurantName(self.non_alpha_name)
)
This is not the correct way to use assertRaises(). You must pass ExpectedExceptionClass, callable, *args, **kw to assertRaises, and args and kw will be passed to your callable. IOW you want:
self.assertRaises(
RestNameContainNonAlphaError,
RestaurantName,
self.non_alpha_name
)
The reason is simple: the way you currently call it, the exception is triggered before the call to assertRaises.
As a side note:
your tearDown method is useless
there's already a builtin exception for wrong types, it's named TypeError
there's also a builtin exception for wrong values which is named ValueError

Exception raising does not reflect in test case even though raised as seen in logs

I'm practicing TDD in Python and came across a problem in testing whether an exception is raised.
Here is my test_phonebook.py with test_add_empty_name_raises_exception which fails.
import unittest
import phonebook
class Test(unittest.TestCase):
def test_add_empty_name_raises_exception(self):
self.assertRaises(ValueError, phonebook.add, "", "1111111111")
if __name__ == "__main__":
# import sys;sys.argv = ['', 'Test.testName']
unittest.main()
Below is my phonebook.py with the method add which adds the data into the dictionary.
import re
_phonebook = {}
file_name = "phonebook.txt"
def is_valid_name(name):
return re.search(r"([A-Z][a-z]*)([\\s\\\'-][A-Z][a-z]*)*", name) is not None
def is_valid_number(number):
return re.search(r"\+?[\d ]+$", number) is not None
def add(name, number):
try:
if is_valid_name(name) and is_valid_number(number):
_phonebook[name] = number
else:
raise ValueError("Invalid arguments.", name, number)
except ValueError as err:
print err.args
if __name__ == '__main__':
pass
My problem is that the test fails even though it is seen in the console log that there was a ValueError raised within the add method.
Finding files... done.
Importing test modules ... done.
('Invalid arguments.', '', '1111111111')
======================================================================
FAIL: test_add_empty_name_raises_exception (path.to.phonebook.test_phonebook.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "path\to\phonebook\test_phonebook.py", line 13, in test_add_empty_name_raises_exception
self.assertRaises(ValueError, phonebook.add, "", "1111111111")
AssertionError: ValueError not raised
----------------------------------------------------------------------
Ran 1 test in 0.002s
How do I solve this? I there something I forgot?
I also tried using the new format for handling exceptions in tests in Python 2.7 but it still hasn't caught the ValueError raising.
def test_add_empty_name_raises_exception(self):
with self.assertRaises(ValueError):
self.phonebook.add("", "1111111111)
I also changed the form of the test case into using lambdas but still no changes.
def test_add_empty_name_raises_exception(self):
self.assertRaises(ValueError, lambda: phonebook.add("", "1111111111"))
I also cleaned my directory and restarted Eclipse Luna and problem still persists.
Possible solution
I was reading the 8.Errors and Exceptions documentation and got to the "Raising Exceptions" part which states that:
If you need to determine whether an exception was raised but don’t intend to handle it,
a simpler form of the raise statement allows you to re-raise the exception:
I added this to the existing add method as such:
def add(name, number):
try:
if is_valid_name(name) and is_valid_number(number):
_phonebook[name] = number
print "Added %s:\t%s" % (name, number)
else:
raise ValueError("Invalid arguments.", name, number)
except ValueError as err:
print err.args
raise
Which caused the test case to pass.
Is this the correct way? To call raise again in the except block?

When you catch an exception (in your except ValueError as err: block), you prevent it from continuing back up the call stack to eventually terminate the program. Essentially, you're saying "I know how to handle this, so no need to panic anyone else."
Re-raising an exception is the proper thing to do if you caught the exception but didn't do so to actually fix anything, for instance, to log that it occurred. Typically, though, one catches an exception in order to correct it.
In your case, you're catching the exception almost immediately after you yourself raised it. Why not put your logging statement in the same else block as the raise? No need for a try: ... except: indent at all.
def add(name, number):
if is_valid_name(name) and is_valid_number(number):
_phonebook[name] = number
print "Added %s:\t%s" % (name, number)
else:
print "Invalid arguments.", name, number
raise ValueError("Invalid arguments.", name, number)
return

How do I signal test error (not failure) from python unittest

I have a test case with a helper method assertContains(super, sub). The sub arguments are a hard-coded part of the test cases. In case they're malformed, I would like my test case to abort with an error.
How do I do that? I have tried
def assertContains(super, sub):
if isinstance(super, foo): ...
elif isinstance(super, bar): ...
else: assert False, repr(sub)
However, this turns the test into a failure rather than an error.
I could raise some other exception (e.g. ValueError), but I want to explicitly state that I'm declaring the test case to be in error. I could do things like ErrorInTest = ValueError and then raise ErrorInTest(repr(sub)), but it feels kinda' icky. I feel there should be a batteries-included way of doing this, but reading the friendly manual didn't suggest anything to me.

There is an assertRaises() for aspects in class TestCase in which you want to ensure an error is raised by the to-be-tested code.
If you want to raise an error and abort testing that unit at this point (and continue with the next unit test), just raise an uncaught exception; the unit test module will catch it:
raise NotImplementedError("malformed sub: %r" % (sub,))
I don't think that there is any other API aspect available besides raising errors directly to state that a unit test case results in an error.
class PassingTest(unittest.TestCase):
def runTest(self):
self.assertTrue(True)
class FailingTest(unittest.TestCase):
def runTest(self):
self.assertTrue(False)
class ErrorTest(unittest.TestCase):
def runTest(self):
raise NotImplementedError("error")
class PassingTest2(unittest.TestCase):
def runTest(self):
self.assertTrue(True)
results in:
EF..
======================================================================
ERROR: runTest (__main__.ErrorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./t.py", line 15, in runTest
raise NotImplementedError("error")
NotImplementedError: error
======================================================================
FAIL: runTest (__main__.FailingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./t.py", line 11, in runTest
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 4 tests in 0.002s
FAILED (failures=1, errors=1)