I am customizing exceptions in my python code. I have inherited exception class in to other and now defining some custom errors as classes derived from my custom exception class like this:
class DataCollectorError(Exception): pass
class ParamNullError(DataCollectorError) : pass
class ParamInvalidTypeError(DataCollectorError) : pass
I am raising these exceptions in my python function like:
def READ_METER_DATA (regIndex, numRegisters, slaveUnit):
try:
if not regIndex:
raise ParamNullError, "register index is null"
if not numRegisters:
raise ParamNullError, "number of registers should not be null"
if not slaveUnit:
raise ParamNullError, "Meter Id should not be null"
and logging error like :
except DataCollectorError as d:
lgr.error('DataCollector Error(READ_METER_DATA): '+d.args[0])
print 'DataCollector Error:(READ_METER_DATA)', d.args[0]
except:
lgr.error('Unexpected Error: ', sys.exc_info())
print 'Unexpected Error: ', sys.exc_info()
pass
but this defeats purpose of unit testing script as it doesn't whether exception raised bcz it is being catched by my catch block before my unit test script knows it. so i wanted to log these errors in base class itself -
Class ParamNullError(DataCollectorError):
<----here----------->
pass
can anybody tell me how to fetch that string passed while raising exception?
Simply extend your error-class with an __init__ and an __str__ method.
Example:
class DataCollectorError(Exception):
def __init__(self, msg=''):
self.msg = msg
log(msg) # use your logging things here
def __str__(self):
return self.msg
Use msg='' because then you don't need to always specify a message.
Don't.
Factor out the calls you need to unit test, and move your exception handler out:
try:
testableFunctionCall()
except:
lgr.exception('Unexpected Error')
and test testableFunctionCall().
Alternatively, use the testfixtures library to test the logging itself:
from testfixtures import LogCapture
with LogCapture() as l:
callFunctionUnderTest()
l.check(
('packagename', 'ERROR', 'DataCollector Error(READ_METER_DATA): foobar'),
)
Related
I'd like to keep a solid logging system going, but it's also necessary to raise exceptions. This code accomplishes what I'm going for, but it looks clunky and not very Pythonic. What's a better option?
import logging
if not condition_met:
missing = set_one - set_two
logging.error('Missing keys: {}'.format(missing))
raise ValueError('Missing keys: {}'.format(missing))
you could catch the exception and log the error at this time, so if another exception occurs you can log it as well, and propagate the exception upstream.
try:
# some code
if not condition_met:
missing = set_one - set_two
raise ValueError('Missing keys: {}'.format(missing))
except Exception as e: # or ValueError to narrow it down
logging.error(str(e))
raise # propagate the exception again
note than logging an exception without logging the traceback leaves something unfinished, specially if the exception is caught and handled upstream. It's very likely that you're never going to fix that particular error.
You can use logger's exception() function:
from logger import exception
try:
. . .
except Exception as error:
exception(msg="Your message")
so that all of the stack will be logged.
You can read an interesting article about this here.
Another elegant approach is to define custom exceptions for your application that serve the purpose of clarifying lower-level exceptions such as KeyError as well as centralizing error logic. These custom exceptions can be defined on a separate file to make maintenance and updates easier. custom exceptions are derived from a base Error class to inherit global settings which itself is derived from the built-in Exception class.
exceptions.py
from utils import log
class Error(Exception):
"""base class for errors"""
class EnvironmentAttributeError(Error):
"""
Exception raised when environment variables are empty strings
Attributes:
key_attribute -- environment variable
"""
def __init__(self, environment_variable):
self.environment_variable = environment_variable
self.message = f"Environment variable value for key {environment_variable} was not assigned."
self.log = log.logger.error(f"Environment variable value for key {environment_variable} was not assigned.")
super().__init__(self.message)
class EnvironmentKeyError(Error):
"""
Exception raised when the environment variables dict does not have required keys
Attributes:
key_attribute -- environment variable
"""
def __init__(self, vars):
self.environment_variable = vars
self.message = f"Environment variable {vars} was not declared."
self.log = log.logger.error(f"Environment variable {vars} was not declared.")
super().__init__(self.message)
Notice that the exceptions.py file imports a log utility. That way all you need to do elsewhere in your code is raise the right custom code errors and everything gets logged for you. You can then update these errors in a single place for your entire project.
log.py
import logging
# instantiate a logger object writing to connected-apps.log
logging.basicConfig(
format='%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG,
filename='logs/connected_apps.log'
)
# logger object named after module: https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial
logger = logging.getLogger(__name__)
The logger in the log.py file has been formatted in such a way that logs are both descriptive and readable. You can even define different loggers with different formats and levels.
Here is a simple use of the custom exceptions defined above. Environment variables obtained from .env are sent to this validate() function to verify that the right keys and attributes are available. Notice that we just needed to import exceptions and not logs:
environment.py
from utils import exceptions
def validate(env_dict, env_vars):
# check that each environment variable has been declared and assigned
for vars in env_vars:
try:
# check the local dictionary pulled from os.environ
env_dict[vars]
# check that key value length is non-zero
if len(env_dict[vars]) == 0:
raise exceptions.EnvironmentAttributeError(vars)
except KeyError as error:
# raises error if an environment variable has not been declared
raise exceptions.EnvironmentKeyError(vars)
I'm new to Python. I need to unit test the except part of a try-except statement in python. I'm using pytest. The problem is that I don't know how to force the try part to raise an exception. Here is my code:
try:
if master_bill_to is False:
master.update(
dbsession,
company_id=master.company_id,
)
except Exception as e:
dbsession.rollback()
raise Conflict(e.message)
The master.update method is called to make an update to the database. But how do I mock this code so that it somehow raises an exception in the try portion?
I'm trying to use monkeypatch with this code. The master object is an instance of the BillTo class so I'm thinking of putting that as the first parameter to monkeypatch.setattr.
def test_create_bill_to_fails_when_master_update_fails(dbsession, invoice_group1, company1,
monkeypatch):
def raise_flush_error():
raise FlushError
context = TestContext()
monkeypatch.setattr(BillTo, 'update', raise_flush_error)
with pytest.raises(FlushError):
create_bill_to(
context,
dbsession=dbsession,
invoice_group_id=invoice_group1.id,
company_id=company1.id,
)
But for some reason, the error is not raised.
Use mock library and side_effect to throw Exception during test case
Mock master and raise an exception in the update method.
Ok, I figured it out. I learned that you must pass parameters to the method called by monkeypatch. These parameters must match the signature of the method being replaced or mocked. Actually I also renamed the method with a prefix fake_ to denote the mocking. Here is what I did:
#staticmethod
def fake_update_flush_error(dbsession, company_id=None, address_id=None, proportion=None,
company_name=None, receiver_name=None, invoice_delivery_method=None,
invoice_delivery_text=None, master_bill_to=False):
raise FlushError
def test_create_bill_to_fails_when_master_update_fails(dbsession, invoice_group1, company1,
bill_to1, monkeypatch):
context = TestContext()
monkeypatch.setattr(BillTo, 'update', fake_update_flush_error)
with pytest.raises(Conflict):
create_bill_to(
context,
dbsession=dbsession,
invoice_group_id=invoice_group1.id,
company_id=company1.id,
address_id=None,
...
)
The BillTo.update method needed all those parameters.
I'm working on a project that involves connecting to a remote server, waiting for a response, and then performing actions based on that response. We catch a couple of different exceptions, and behave differently depending on which exception is caught. For example:
def myMethod(address, timeout=20):
try:
response = requests.head(address, timeout=timeout)
except requests.exceptions.Timeout:
# do something special
except requests.exceptions.ConnectionError:
# do something special
except requests.exceptions.HTTPError:
# do something special
else:
if response.status_code != requests.codes.ok:
# do something special
return successfulConnection.SUCCESS
To test this, we've written a test like the following
class TestMyMethod(unittest.TestCase):
def test_good_connection(self):
config = {
'head.return_value': type('MockResponse', (), {'status_code': requests.codes.ok}),
'codes.ok': requests.codes.ok
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.SUCCESS
)
def test_bad_connection(self):
config = {
'head.side_effect': requests.exceptions.ConnectionError,
'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
If I run the function directly, everything happens as expected. I even tested by adding raise requests.exceptions.ConnectionError to the try clause of the function. But when I run my unit tests, I get
ERROR: test_bad_connection (test.test_file.TestMyMethod)
----------------------------------------------------------------
Traceback (most recent call last):
File "path/to/sourcefile", line ###, in myMethod
respone = requests.head(address, timeout=timeout)
File "path/to/unittest/mock", line 846, in __call__
return _mock_self.mock_call(*args, **kwargs)
File "path/to/unittest/mock", line 901, in _mock_call
raise effect
my.package.requests.exceptions.ConnectionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "Path/to/my/test", line ##, in test_bad_connection
mypackage.myMethod('some_address',
File "Path/to/package", line ##, in myMethod
except requests.exceptions.ConnectionError:
TypeError: catching classes that do not inherit from BaseException is not allowed
I tried to change the exception I was patching in to BaseException and I got a more or less identical error.
I've read https://stackoverflow.com/a/18163759/3076272 already, so I think it must be a bad __del__ hook somewhere, but I'm not sure where to look for it or what I can even do in the mean time. I'm also relatively new to unittest.mock.patch() so it's very possible that I'm doing something wrong there as well.
This is a Fusion360 add-in so it is using Fusion 360's packaged version of Python 3.3 - as far as I know it's a vanilla version (i.e. they don't roll their own) but I'm not positive of that.
I could reproduce the error with a minimal example:
foo.py:
class MyError(Exception):
pass
class A:
def inner(self):
err = MyError("FOO")
print(type(err))
raise err
def outer(self):
try:
self.inner()
except MyError as err:
print ("catched ", err)
return "OK"
Test without mocking :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
a = foo.A()
self.assertEquals("OK", a.outer())
Ok, all is fine, both test pass
The problem comes with the mocks. As soon as the class MyError is mocked, the expect clause cannot catch anything and I get same error as the example from the question :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
with unittest.mock.patch('foo.MyError'):
a = exc2.A()
self.assertEquals("OK", a.outer())
Immediately gives :
ERROR: test_outer (__main__.FooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "...\foo.py", line 11, in outer
self.inner()
File "...\foo.py", line 8, in inner
raise err
TypeError: exceptions must derive from BaseException
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#78>", line 8, in test_outer
File "...\foo.py", line 12, in outer
except MyError as err:
TypeError: catching classes that do not inherit from BaseException is not allowed
Here I get a first TypeErrorthat you did not have, because I am raising a mock while you forced a true exception with 'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError in config. But the problem remains that the except clause tries to catch a mock.
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this but in your example, you only need to patch requests.head. So I think that this should work :
def test_bad_connection(self):
with mock.patch('path.to.my.package.requests.head',
side_effect=requests.exceptions.ConnectionError):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
That is : only patch the head method with the exception as side effect.
I just ran into the same issue while trying to mock sqlite3 (and found this post while looking for solutions).
What Serge said is correct:
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this
My solution was to mock the entire module, then set the mock attribute for the exception to be equal to the exception in the real class, effectively "un-mocking" the exception. For example, in my case:
#mock.patch(MyClass.sqlite3)
def test_connect_fail(self, mock_sqlite3):
mock_sqlite3.connect.side_effect = sqlite3.OperationalError()
mock_sqlite3.OperationalError = sqlite3.OperationalError
self.assertRaises(sqlite3.OperationalError, MyClass, self.db_filename)
For requests, you could assign exceptions individually like this:
mock_requests.exceptions.ConnectionError = requests.exceptions.ConnectionError
or do it for all of the requests exceptions like this:
mock_requests.exceptions = requests.exceptions
I don't know if this is the "right" way to do it, but so far it seems to work for me without any issue.
For those of us who need to mock an exception and can't do that by simply patching head, here is an easy solution that replaces the target exception with an empty one:
Say we have a generic unit to test with an exception we have to have mocked:
# app/foo_file.py
def test_me():
try:
foo()
return "No foo error happened"
except CustomError: # <-- Mock me!
return "The foo error was caught"
We want to mock CustomError but because it is an exception we run into trouble if we try to patch it like everything else. Normally, a call to patch replaces the target with a MagicMock but that won't work here. Mocks are nifty, but they do not behave like exceptions do. Rather than patching with a mock, let's give it a stub exception instead. We'll do that in our test file.
# app/test_foo_file.py
from mock import patch
# A do-nothing exception we are going to replace CustomError with
class StubException(Exception):
pass
# Now apply it to our test
#patch('app.foo_file.foo')
#patch('app.foo_file.CustomError', new_callable=lambda: StubException)
def test_foo(stub_exception, mock_foo):
mock_foo.side_effect = stub_exception("Stub") # Raise our stub to be caught by CustomError
assert test_me() == "The error was caught"
# Success!
So what's with the lambda? The new_callable param calls whatever we give it and replaces the target with the return of that call. If we pass our StubException class straight, it will call the class's constructor and patch our target object with an exception instance rather than a class which isn't what we want. By wrapping it with lambda, it returns our class as we intend.
Once our patching is done, the stub_exception object (which is literally our StubException class) can be raised and caught as if it were the CustomError. Neat!
I faced a similar issue while trying to mock the sh package. While sh is very useful, the fact that all methods and exceptions are defined dynamically make it more difficult to mock them. So following the recommendation of the documentation:
import unittest
from unittest.mock import Mock, patch
class MockSh(Mock):
# error codes are defined dynamically in sh
class ErrorReturnCode_32(BaseException):
pass
# could be any sh command
def mount(self, *args):
raise self.ErrorReturnCode_32
class MyTestCase(unittest.TestCase):
mock_sh = MockSh()
#patch('core.mount.sh', new=mock_sh)
def test_mount(self):
...
I just ran into the same problem when mocking struct.
I get the error:
TypeError: catching classes that do not inherit from BaseException is not allowed
When trying to catch a struct.error raised from struct.unpack.
I found that the simplest way to get around this in my tests was to simply set the value of the error attribute in my mock to be Exception. For example
The method I want to test has this basic pattern:
def some_meth(self):
try:
struct.unpack(fmt, data)
except struct.error:
return False
return True
The test has this basic pattern.
#mock.patch('my_module.struct')
def test_some_meth(self, struct_mock):
'''Explain how some_func should work.'''
struct_mock.error = Exception
self.my_object.some_meth()
struct_mock.unpack.assert_called()
struct_mock.unpack.side_effect = struct_mock.error
self.assertFalse(self.my_object.some_meth()
This is similar to the approach taken by #BillB, but it is certainly simpler as I don't need to add imports to my tests and still get the same behavior. To me it would seem this is the logical conclusion to the general thread of reasoning in the answers here.
Use patch.object to partially mock a class.
My use case:
import unittest
from unittest import mock
import requests
def test_my_function(self):
response = mock.MagicMock()
response.raise_for_status.side_effect = requests.HTTPError
with mock.patch.object(requests, 'get', return_value=response):
my_function()
Is there a way I can catch exceptions in the __enter__ method of a context manager without wrapping the whole with block inside a try?
class TstContx(object):
def __enter__(self):
raise Exception("I'd like to catch this exception")
def __exit__(self, e_typ, e_val, trcbak):
pass
with TstContx():
raise Exception("I don't want to catch this exception")
pass
I know that I can catch the exception within __enter__() itself, but can I access that error from the function that contains the with statement?
On the surface the question Catching exception in context manager __enter__() seems to be the same thing but that question is actually about making sure that __exit__ gets called, not with treating the __enter__ code differently from the block that the with statement encloses.
...evidently the motivation should be clearer. The with statement is setting up some logging for a fully automated process. If the program fails before the logging is set up, then I can't rely on the logging to notify me, so I have to do something special. And I'd rather achieve the effect without having to add more indentation, like this:
try:
with TstContx():
try:
print "Do something"
except Exception:
print "Here's where I would handle exception generated within the body of the with statement"
except Exception:
print "Here's where I'd handle an exception that occurs in __enter__ (and I suppose also __exit__)"
Another downside to using two try blocks is that the code that handles the exception in __enter__ comes after the code that handles exception in the subsequent body of the with block.
You can catch the exception using try/except inside of __enter__, then save the exception instance as an instance variable of the TstContx class, allowing you to access it inside of the with block:
class TstContx(object):
def __enter__(self):
self.exc = None
try:
raise Exception("I'd like to catch this exception")
except Exception as e:
self.exc = e
return self
def __exit__(self, e_typ, e_val, trcbak):
pass
with TstContx() as tst:
if tst.exc:
print("We caught an exception: '%s'" % tst.exc)
raise Exception("I don't want to catch this exception")
Output:
We caught an exception: 'I'd like to catch this exception'.
Traceback (most recent call last):
File "./torn.py", line 20, in <module>
raise Exception("I don't want to catch this exception")
Exception: I don't want to catch this exception
Not sure why you'd want to do this, though....
You can use contextlib.ExitStack as outlined in this doc example in order to check for __enter__ errors separately:
from contextlib import ExitStack
stack = ExitStack()
try:
stack.enter_context(TstContx())
except Exception: # `__enter__` produced an exception.
pass
else:
with stack:
... # Here goes the body of the `with`.
I'm having some problems implementing an exception system in my program.
I found somewhere the following piece of code that I am trying to use for my program:
class InvalidProgramStateException(Exception):
def __init__(self, expr, msg):
self.expr = expr
self.msg = msg
I think msg must be a string message to be shown, but how do I fill the "expr" when I want to raise this exception? Do I have to write it by hand?
raise InvalidProgramStateException(what_here?, "there was an error")
Your custom exceptions don't actually need to take parameters at all. If you haven't got any particular error message or state to encapsulate in the Exception, this will work just fine:
class MyException(Exception):
pass
This would allow your program to catch cases of this exception by type:
try:
raise MyException()
except MyException:
print "Doing something with MyException"
except:
print "Some other error occurred... handling it differently"
If you want the Exception to have some meaningful string representation, or have properties that would provide your application greater details on what went wrong, that's when you pass additional arguments to the constructor. The number, name and type of these arguments is not really pre-defined by Python... they can be anything. Just be sure to provide a custom __str__ or __unicode__ method so you can provide a meaningful text depiction:
class MyException(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return "MyException with %s" % self.msg
In the case of the example you're quoting, the expr and msg parameters are specific to the fictional case of the example. A contrived scenario for how these might be used is:
def do_something(expr):
if 'foo' in expr:
raise InvalidProgramStateException(expr, "We don't allow foos here")
return 5
user_input = 'foo bar'
try:
do_something(user_input)
except InvalidProgramStateException, e:
print "%s (using expression %s)" % (e.msg, e.expr)
Since it doesn't appear that your application requires it, just drop the parameters you don't require.