Develop Reference

Simple Python Logging Exception from Future

This should be a really simple question, but after googling, reading docs, and several other SO threads, I don't see the answer: How do I log an exception with Python standard logging? One small wrinkle is that I'm getting the exception from a Future. I'm not writing the except exception handler myself. Ideally, I would get the exception message, a stack trace, the extra message sent, and maybe the type of exception. Here's a simple program that shows my issue:
import logging
from concurrent.futures import ThreadPoolExecutor
logger = logging.getLogger(__name__)
def test_f(a, b=-99, c=50):
logger.info("test_f a={} b={} c={}".format(a, b, c))
def future_callback_error_logger(future):
e = future.exception()
if e is not None:
# This log statement does not seem to do what I want.
# It logs "Executor Exception" with no information about the exception.
# I would like to see the exception type, message, and stack trace.
logger.error("Executor Exception", exc_info=e)
def submit_with_log_on_error(executor, func, *args, **kwargs):
future = executor.submit(func, *args, **kwargs)
future.add_done_callback(future_callback_error_logger)
if __name__ == "__main__":
logging.basicConfig(level="DEBUG")
logger.info("start")
executor = ThreadPoolExecutor(max_workers=5)
# This will work.
submit_with_log_on_error(executor, test_f, 10, c=20)
# This will intentionally trigger an error due to too many arguments.
# I would like that error to be properly logged.
submit_with_log_on_error(executor, test_f, 10, 20, 30, 40)
# This will work.
submit_with_log_on_error(executor, test_f, 50, c=60)
executor.shutdown(True)
logger.info("shutdown")

To use logger.exception and get the traceback etc, you need to be inside an except block. Instead of checking the future.exception(), which returns the exception (if any), use the future.result() which raises the exception (if any).
So, try this instead (if you'll pardon the pun):
def future_callback_error_logger(future):
try:
future.result()
except Exception:
logger.exception("Executor Exception")

Catching exception in decorator that wraps generator

I have a decorator that wraps a generator that yields from inside a nose test case. For every iteration, I'm looking to catch and run a specific teardown if an exception occurs, however it does not seem to behave as expected.
def print_log(test_case):
#wraps(test_case)
def run_test(self):
try:
for _ in test_case(self): pass
except:
Test_Loop.failure_teardown(self)
raise
return run_test
Is there something I am doing wrong?

I'm not sure exactly what the unexpected behavior is, but maybe it is happening because you are not trying each loop iteration individually.
Maybe this will work?
def print_log(test_case):
#wraps(test_case)
def run_test(self):
from six.moves import next
test_iter = iter(test_case(self))
while True:
try:
next(test_iter)
except StopIteration:
break
except Exception:
Test_Loop.failure_teardown(self)
raise
return run_test

python custom exception handler for a class

I want to build my application with a redis cache. but maybe redis is not available all the time in our case,
so I hope, it redis works well, we use it. if it can't work, just logging and ignore it this time.
for example:
try:
conn.sadd('s', *array)
except :
...
since there are many place I will run some conn.{rediscommand}, I don't like to use try/except every place.
so the solution maybe :
class softcache(redis.StrictRedis):
def sadd(key, *p):
try:
super(redis.StrictRedis, self).sadd(key, p)
except:
..
but since redis have many commands, I have to warp them one by one.
is it possible to custom a exception handler for a class to handle all the exceptions which come from this class ?

Silencing per default all exceptions is probably the worst thing you can do.
Anyway, for your problem you can write a generic wrapper that just redirects to the connection object.
class ReddisWrapper(object):
conn = conn # Here your reddis object
def __getattr__(self, attr):
def wrapper(*args, **kwargs):
# Get the real reddis function
fn = getattr(self.conn, attr)
# Execute the function catching exceptions
try:
return fn(*args, **kwargs)
# Specify here the exceptions you expect
except:
log(...)
return wrapper
And then you would call like this:
reddis = ReddisWrapper()
reddis.do_something(4)
This has not been tested, and will only work with methods. For properties you should catch the non callable exception and react properly.

Is it always the same Exception?
If so you could write a custom, Exception catching and logging decorator.
Something like the following:
def exception_catcher(fn):
try:
fn()
except Exception as e:
log(e)
Then just use it around your code:
#exception_catcher
sadd('s', *array)
The comment and link to Exceptions for the whole class suggested by #idanshmu will offer more detailed handling of different Exceptions per method.

Exceptions for the whole class

I'm writing a program in Python, and nearly every method im my class is written like this:
def someMethod(self):
try:
#...
except someException:
#in case of exception, do something here
#e.g display a dialog box to inform the user
#that he has done something wrong
As the class grows, it is a little bit annoying to write the same try-except block over and over. Is it possible to create some sort of 'global' exception for the whole class? What's the recommended way in Python to deal with this?

Write one or more exception handler functions that, given a function and the exception raised in it, does what you want to do (e.g. displays an alert). If you need more than one, write them.
def message(func, e):
print "Exception", type(e).__name__, "in", func.__name__
print str(e)
Now write a decorator that applies a given handler to a called function:
import functools
def handle_with(handler, *exceptions):
try:
handler, cleanup = handler
except TypeError:
cleanup = lambda f, e: None
def decorator(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except exceptions or Exception as e:
return handler(func, e)
else:
e = None
finally:
cleanup(func, e)
return wrapper
return decorator
This only captures the exceptions you specify. If you don't specify any, Exception is caught. Additionally, the first argument can be a tuple (or other sequence) of two handler functions; the second handler, if given, is called in a finally clause. The value returned from the primary handler is returned as the value of the function call.
Now, given the above, you can write:
#handle_with(message, TypeError, ValueError)
def add(x, y):
return x + y
You could also do this with a context manager:
from contextlib import contextmanager
#contextmanager
def handler(handler, *exceptions):
try:
handler, cleanup = handler
except TypeError:
cleanup = lambda e: None
try:
yield
except exceptions or Exception as e:
handler(e)
else:
e = None
finally:
cleanup(e)
Now you can write:
def message(e):
print "Exception", type(e).__name__
print str(e)
def add(x, y):
with handler(message, TypeError, ValueError):
return x + y
Note that the context manager doesn't know what function it's in (you can find this out, sorta, using inspect, though this is "magic" so I didn't do it) so it gives you a little less useful information. Also, the context manager doesn't give you the opportunity to return anything in your handler.

I can think of two options:
Write a decorator that can wrap each method in the try block.
Write a "dispatcher" method that calls the appropriate method inside a try block, then call that method instead of the individual ones. That is, instead of calling obj.someMethod(), obj.otherMethod, you call obj.dispatch('someMethod') or obj.dispatch('otherMethod'), where dispatch is a wrapper that contains the try block.
Your approach seems like a bit of a strange design, though. It might make more sense to have the dialog-box stuff in some other part of the code, some higher-level event loop that catches errors and displays messages about them.

Make Python unittest fail on exception from any thread

I am using the unittest framework to automate integration tests of multi-threaded python code, external hardware and embedded C. Despite my blatant abuse of a unittesting framework for integration testing, it works really well. Except for one problem: I need the test to fail if an exception is raised from any of the spawned threads. Is this possible with the unittest framework?
A simple but non-workable solution would be to either a) refactor the code to avoid multi-threading or b) test each thread separately. I cannot do that because the code interacts asynchronously with the external hardware. I have also considered implementing some kind of message passing to forward the exceptions to the main unittest thread. This would require significant testing-related changes to the code being tested, and I want to avoid that.
Time for an example. Can I modify the test script below to fail on the exception raised in my_thread without modifying the x.ExceptionRaiser class?
import unittest
import x
class Test(unittest.TestCase):
def test_x(self):
my_thread = x.ExceptionRaiser()
# Test case should fail when thread is started and raises
# an exception.
my_thread.start()
my_thread.join()
if __name__ == '__main__':
unittest.main()

At first, sys.excepthook looked like a solution. It is a global hook which is called every time an uncaught exception is thrown.
Unfortunately, this does not work. Why? well threading wraps your run function in code which prints the lovely tracebacks you see on screen (noticed how it always tells you Exception in thread {Name of your thread here}? this is how it's done).
Starting with Python 3.8, there is a function which you can override to make this work: threading.excepthook
... threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled
So what do we do? Replace this function with our logic, and voilà:
For python >= 3.8
import traceback
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self, args):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
self.__org_hook(args)
formated_exc = traceback.format_exception(args.exc_type, args.exc_value, args.exc_traceback)
self._exceptions.append('\n'.join(formated_exc))
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading.excepthook
threading.excepthook = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading.excepthook = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception(f'Exceptions in other threads: {tracebacks}')
For older versions of Python, this is a bit more complicated.
Long story short, it appears that the threading nodule has an undocumented import which does something along the lines of:
threading._format_exc = traceback.format_exc
Not very surprisingly, this function is only called when an exception is thrown from a thread's run function.
So for python <= 3.7
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
formated_exc = self.__org_hook()
self._exceptions.append(formated_exc)
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading._format_exc
threading._format_exc = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading._format_exc = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception('Exceptions in other threads: %s' % tracebacks)
Usage:
my_thread = x.ExceptionRaiser()
# will fail when thread is started and raises an exception.
with GlobalExceptionWatcher():
my_thread.start()
my_thread.join()
You still need to join yourself, but upon exit, the with-statement's context manager will check for any exception thrown in other threads, and will raise an exception appropriately.
THE CODE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED
This is an undocumented, sort-of-horrible hack. I tested it on linux and windows, and it seems to work. Use it at your own risk.

I've come across this problem myself, and the only solution I've been able to come up with is subclassing Thread to include an attribute for whether or not it terminates without an uncaught exception:
from threading import Thread
class ErrThread(Thread):
"""
A subclass of Thread that will log store exceptions if the thread does
not exit normally
"""
def run(self):
try:
Thread.run(self)
except Exception as self.err:
pass
else:
self.err = None
class TaskQueue(object):
"""
A utility class to run ErrThread objects in parallel and raises and exception
in the event that *any* of them fail.
"""
def __init__(self, *tasks):
self.threads = []
for t in tasks:
try:
self.threads.append(ErrThread(**t)) ## passing in a dict of target and args
except TypeError:
self.threads.append(ErrThread(target=t))
def run(self):
for t in self.threads:
t.start()
for t in self.threads:
t.join()
if t.err:
raise Exception('Thread %s failed with error: %s' % (t.name, t.err))

I've been using the accepted answer above for a while now, but since Python 3.8 the solution doesn't work anymore because the threading module doesn't have this _format_exc import anymore.
On the other hand the threading module now has a nice way to register custom except hooks in Python 3.8 so here is a simple solution to run unit tests which assert that some exceptions are raised inside threads:
def test_in_thread():
import threading
exceptions_caught_in_threads = {}
def custom_excepthook(args):
thread_name = args.thread.name
exceptions_caught_in_threads[thread_name] = {
'thread': args.thread,
'exception': {
'type': args.exc_type,
'value': args.exc_value,
'traceback': args.exc_traceback
}
}
# Registering our custom excepthook to catch the exception in the threads
threading.excepthook = custom_excepthook
# dummy function that raises an exception
def my_function():
raise Exception('My Exception')
# running the funciton in a thread
thread_1 = threading.Thread(name='thread_1', target=my_function, args=())
thread_1.start()
thread_1.join()
assert 'thread_1' in exceptions_caught_in_threads # there was an exception in thread 1
assert exceptions_caught_in_threads['thread_1']['exception']['type'] == Exception
assert str(exceptions_caught_in_threads['thread_1']['exception']['value']) == 'My Exception'