Develop Reference

patch object spawned in sub-processs

I am using multiprocessing package for crating sub-processes. I need to handle exceptions from sub-process. catch, report, terminate and re-spawn sub-process.
I struggle to create test for it.
I would like to patch object which represent my sub-process and raise exception to see if handling is correct.
But it looks like that object is patched only in main process and in spawned process is unchanged version. Any ideas how to accomplish requested functionality?
Example:
import multiprocessing
import time
class SubprocessClass(multiprocessing.Process):
def __init__(self) -> None:
super().__init__()
def simple_method(self):
return 42
def run(self):
try:
self.simple_method()
except Exception:
# ok, exception handled
pass
else:
# I wanted exception ! <- code goes here
assert False
#mock.patch.object(SubprocessClass, "simple_method")
def test_patch_subprocess(mock_simple_method):
mock_simple_method.side_effect = Exception("exception from mock")
subprocess = SubprocessClass()
subprocess.run()
subprocess.start()
time.sleep(0.1)
subprocess.join()

you can monkey-patch the object before it is started
(it is a bit iffy but you will get actual process running that code)
def _this_always_raises(*args, **kwargs):
raise RuntimeError("I am overridden")
def test_patch_subprocess():
subprocess = SubprocessClass()
subprocess.simple_method = _this_always_raises
subprocess.start()
time.sleep(0.1)
subprocess.join()
assert subprocess.exitcode == 0
you could also mock multiprocessing to behave like threading but that is a bit unpredictable
if you want to do it genericly for all objects you can mock the class for another one derived from the original one with only one method overriden
class SubprocessClassThatRaisesInSimpleMethod(SubprocessClass):
def simple_method(self):
raise RuntimeError("I am overridden")
# then mock with unittest mock the process spawner to use this class instead of SubprocessClass

Using context managers for recovering from celery's SoftTimeLimitExceeded

I am trying to set a maximum run time for my celery jobs.
I am currently recovering from exceptions with a context manager. I ended up with code very similar to this snippet:
from celery.exceptions import SoftTimeLimitExceeded
class Manager:
def __enter__(self):
return self
def __exit__(self, error_type, error, tb):
if error_type == SoftTimeLimitExceeded:
logger.info('job killed.')
# swallow the exception
return True
#task
def do_foo():
with Manager():
run_task1()
run_task2()
run_task3()
What I expected:
If do_foo times out in run_task1, the logger logs, the SoftTimeLimitExceeded exception is swallowed, the body of the manager is skipped, the job ends without running run_task2 and run_task3.
What I observe:
do_foo times out in run_task1, SoftTimeLimitExceeded is raised, the logger logs, the SoftTimeLimitExceeded exception is swallowed but run_task2 and run_task3 are running nevertheless.
I am looking for an answer to following two questions:
Why is run_task2 still executed when SoftTimeLimitExceeded is raised in run_task1 in this setting?
Is there an easy way to transform my code so that it can performs as expected?

Cleaning up the code
This code is pretty good; there's not much cleaning up to do.
You shouldn't return self from __enter__ if the context manager isn't designed to be used with the as keyword.
is should be used when checking classes, since they are singletons...
but you should prefer issubclass to properly emulate exception handling.
Implementing these changes gives:
from celery.exceptions import SoftTimeLimitExceeded
class Manager:
def __enter__(self):
pass
def __exit__(self, error_type, error, tb):
if issubclass(error_type, SoftTimeLimitExceeded):
logger.info('job killed.')
# swallow the exception
return True
#task
def do_foo():
with Manager():
run_task1()
run_task2()
run_task3()
Debugging
I created a mock environment for debugging:
class SoftTimeLimitExceeded(Exception):
pass
class Logger:
info = print
logger = Logger()
del Logger
def task(f):
return f
def run_task1():
print("running task 1")
raise SoftTimeLimitExceeded
def run_task2():
print("running task 2")
def run_task_3():
print("running task 3")
Executing this and then your program gives:
>>> do_foo()
running task 1
job killed.
This is the expected behaviour.
Hypotheses
I can think of two possibilities:
Something in the chain, probably run_task1, is asynchronous.
celery is doing something weird.
I'll run with the second hypothesis because I can't test the former.
I've been bitten by the obscure behaviour of a combination between context managers, exceptions and coroutines before, so I know what sorts of problems it causes. This seems like one of them, but I'll have to look at celery's code before I can go any further.
Edit: I can't make head nor tail of celery's code, and searching hasn't turned up the code that raises SoftTimeLimitExceeded to allow me to trace it backwards. I'll pass it on to somebody more experienced with celery to see if they can work out how it works.

Exception handled surprisingly in Pyside slots

Problem: When exceptions are raised in slots, invoked by signals, they do not seem to propagate as usual through Pythons call stack. In the example code below invoking:
on_raise_without_signal(): Will handle the exception as expected.
on_raise_with_signal(): Will print the exception and then unexpectedly print the success message from the else block.
Question: What is the reason behind the exception being handled surprisingly when raised in a slot? Is it some implementation detail/limitation of the PySide Qt wrapping of signals/slots? Is there something to read about in the docs?
PS: I initially came across that topic when I got surprising results upon using try/except/else/finally when implementing a QAbstractTableModels virtual methods insertRows() and removeRows().
# -*- coding: utf-8 -*-
"""Testing exception handling in PySide slots."""
from __future__ import unicode_literals, print_function, division
import logging
import sys
from PySide import QtCore
from PySide import QtGui
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class ExceptionTestWidget(QtGui.QWidget):
raise_exception = QtCore.Signal()
def __init__(self, *args, **kwargs):
super(ExceptionTestWidget, self).__init__(*args, **kwargs)
self.raise_exception.connect(self.slot_raise_exception)
layout = QtGui.QVBoxLayout()
self.setLayout(layout)
# button to invoke handler that handles raised exception as expected
btn_raise_without_signal = QtGui.QPushButton("Raise without signal")
btn_raise_without_signal.clicked.connect(self.on_raise_without_signal)
layout.addWidget(btn_raise_without_signal)
# button to invoke handler that handles raised exception via signal unexpectedly
btn_raise_with_signal = QtGui.QPushButton("Raise with signal")
btn_raise_with_signal.clicked.connect(self.on_raise_with_signal)
layout.addWidget(btn_raise_with_signal)
def slot_raise_exception(self):
raise ValueError("ValueError on purpose")
def on_raise_without_signal(self):
"""Call function that raises exception directly."""
try:
self.slot_raise_exception()
except ValueError as exception_instance:
logger.error("{}".format(exception_instance))
else:
logger.info("on_raise_without_signal() executed successfully")
def on_raise_with_signal(self):
"""Call slot that raises exception via signal."""
try:
self.raise_exception.emit()
except ValueError as exception_instance:
logger.error("{}".format(exception_instance))
else:
logger.info("on_raise_with_signal() executed successfully")
if (__name__ == "__main__"):
application = QtGui.QApplication(sys.argv)
widget = ExceptionTestWidget()
widget.show()
sys.exit(application.exec_())

As you've already noted in your question, the real issue here is the treatment of unhandled exceptions raised in python code executed from C++. So this is not only about signals: it also affects reimplemented virtual methods as well.
In PySide, PyQt4, and all PyQt5 versions up to 5.5, the default behaviour is to automatically catch the error on the C++ side and dump a traceback to stderr. Normally, a python script would also automatically terminate after this. But that is not what happens here. Instead, the PySide/PyQt script just carries on regardless, and many people quite rightly regard this as a bug (or at least a misfeature). In PyQt-5.5, this behaviour has now been changed so that qFatal() is also called on the C++ side, and the program will abort like a normal python script would. (I don't know what the current situation is with PySide2, though).
So - what should be done about all this? The best solution for all versions of PySide and PyQt is to install an exception hook - because it will always take precedence over the default behaviour (whatever that may be). Any unhandled exception raised by a signal, virtual method or other python code will firstly invoke sys.excepthook, allowing you to fully customise the behaviour in whatever way you like.
In your example script, this could simply mean adding something like this:
def excepthook(cls, exception, traceback):
print('calling excepthook...')
logger.error("{}".format(exception))
sys.excepthook = excepthook
and now the exception raised by on_raise_with_signal can be handled in the same way as all other unhandled exceptions.
Of course, this does imply that best practice for most PySide/PyQt applications is to use largely centralised exception handling. This often includes showing some kind of crash-dialog where the user can report unexpected errors.

According to the Qt5 docs you need to handle exceptions within the slot being invoked.
Throwing an exception from a slot invoked by Qt's signal-slot connection mechanism is considered undefined behaviour, unless it is handled within the slot
State state;
StateListener stateListener;
// OK; the exception is handled before it leaves the slot.
QObject::connect(&state, SIGNAL(stateChanged()), &stateListener, SLOT(throwHandledException()));
// Undefined behaviour; upon invocation of the slot, the exception will be propagated to the
// point of emission, unwinding the stack of the Qt code (which is not guaranteed to be exception safe).
QObject::connect(&state, SIGNAL(stateChanged()), &stateListener, SLOT(throwUnhandledException()));
If the slot was invoked directly, like a regular function call,
exceptions may be used. This is because the connection mechanism is
bypassed when invoking slots directly
In the first case you call slot_raise_exception() directly, so this is fine.
In the second case you are invoking it via the raise_exception signal, so the exception will only propagate up to the point where slot_raise_exception() is called. You need to place the try/except/else inside slot_raise_exception() for the exception to be handled correctly.

Thanks for answering guys. I found ekhumoros answer particularly useful to understand where the exceptions are handled and because of the idea to utilize sys.excepthook.
I mocked up a quick solution via context manager to temporarily extend the current sys.excepthook to record any exception in the realm of "C++ calling Python" (as it seems to happen when slots are invoked by signals or virtual methods) and possibly re-raise upon exiting the context to achieve expected control flow in try/except/else/finally blocks.
The context manager allows on_raise_with_signal to maintain the same control flow as on_raise_without_signal with the surrounding try/except/else/finally block.
# -*- coding: utf-8 -*-
"""Testing exception handling in PySide slots."""
from __future__ import unicode_literals, print_function, division
import logging
import sys
from functools import wraps
from PySide import QtCore
from PySide import QtGui
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class ExceptionHook(object):
def extend_exception_hook(self, exception_hook):
"""Decorate sys.excepthook to store a record on the context manager
instance that might be used upon leaving the context.
"""
#wraps(exception_hook)
def wrapped_exception_hook(exc_type, exc_val, exc_tb):
self.exc_val = exc_val
return exception_hook(exc_type, exc_val, exc_tb)
return wrapped_exception_hook
def __enter__(self):
"""Temporary extend current exception hook."""
self.current_exception_hook = sys.excepthook
sys.excepthook = self.extend_exception_hook(sys.excepthook)
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Reset current exception hook and re-raise in Python call stack after
we have left the realm of `C++ calling Python`.
"""
sys.excepthook = self.current_exception_hook
try:
exception_type = type(self.exc_val)
except AttributeError:
pass
else:
msg = "{}".format(self.exc_val)
raise exception_type(msg)
class ExceptionTestWidget(QtGui.QWidget):
raise_exception = QtCore.Signal()
def __init__(self, *args, **kwargs):
super(ExceptionTestWidget, self).__init__(*args, **kwargs)
self.raise_exception.connect(self.slot_raise_exception)
layout = QtGui.QVBoxLayout()
self.setLayout(layout)
# button to invoke handler that handles raised exception as expected
btn_raise_without_signal = QtGui.QPushButton("Raise without signal")
btn_raise_without_signal.clicked.connect(self.on_raise_without_signal)
layout.addWidget(btn_raise_without_signal)
# button to invoke handler that handles raised exception via signal unexpectedly
btn_raise_with_signal = QtGui.QPushButton("Raise with signal")
btn_raise_with_signal.clicked.connect(self.on_raise_with_signal)
layout.addWidget(btn_raise_with_signal)
def slot_raise_exception(self):
raise ValueError("ValueError on purpose")
def on_raise_without_signal(self):
"""Call function that raises exception directly."""
try:
self.slot_raise_exception()
except ValueError as exception_instance:
logger.error("{}".format(exception_instance))
else:
logger.info("on_raise_without_signal() executed successfully")
def on_raise_with_signal(self):
"""Call slot that raises exception via signal."""
try:
with ExceptionHook() as exception_hook:
self.raise_exception.emit()
except ValueError as exception_instance:
logger.error("{}".format(exception_instance))
else:
logger.info("on_raise_with_signal() executed successfully")
if (__name__ == "__main__"):
application = QtGui.QApplication(sys.argv)
widget = ExceptionTestWidget()
widget.show()
sys.exit(application.exec_())

This way of handling expcetions is not a surprise taking into consideration that the Signal/Slot architecture proposes a loosely coupled intercation between the signals and the slots. This means that the signal should not be expecting anything to happen inside the slots.
Although timmwagener's solution was pretty clever, it should be used with caution. Probably the problem is not with how Exceptions are handled between Qt Connections, but that the signal/slot architecture is not ideal for your application. Also, that solution would not work if a slot from a different thread is connected, or a Qt.QueuedConnection is used.
A good way of tackling the problem of errors raised in slots is to determine that at the connection and not the emitting. Then the erros can be handled in a loosely coupled way.
class ExceptionTestWidget(QtGui.QWidget):
error = QtCore.Signal(object)
def abort_execution():
pass
def error_handler(self, err):
self.error.emit(error)
self.abort_execution()
(...)
def connect_with_async_error_handler(sig, slot, error_handler, *args,
conn_type=None, **kwargs):
#functools.wraps(slot)
def slot_with_error_handler(*args):
try:
slot(*args)
except Exception as err:
error_handler(err)
if conn_type is not None:
sig.connect(slot_with_error_handler, conn_type)
else:
sig.connect(slot_with_error_handler)
This way, we would be complying to the requirements in the Qt5 docs, stating that you need to handle exceptions within the slot being invoked.
Throwing an exception from a slot invoked by Qt's signal-slot
connection mechanism is considered undefined behaviour, unless it is
handled within the slot
PS: This is just a sugestion based on a very small overview of your use case. There is no right/wrong way of solving this, I just wanted to bring out a different point of view : )

Python magic method to alter the way raising an object is handled

I was wondering, is there a simple magic method in python that allows customization of the behaviour of an exception-derived object when it is raised? I'm looking for something like __raise__ if that exists. If no such magic methods exist, is there any way I could do something like the following (it's just an example to prove my point):
class SpecialException(Exception):
def __raise__(self):
print('Error!')
raise SpecialException() #this is the part of the code that must stay
Is it possible?

I don't know about such magic method but even if it existed it is just some piece of code that gets executed before actually raising the exception object. Assuming that its a good practice to raise exception objects that are instantiated in-place you can put such code into the __init__ of the exception. Another workaround: instead of raising your exception directly you call an error handling method/function that executes special code and then finally raises an exception.

import time
from functools import wraps
def capture_exception(callback=None, *c_args, **c_kwargs):
"""捕获到异常后执行回调函数"""
assert callable(callback), "callback 必须是可执行对象"
def _out(func):
#wraps(func)
def _inner(*args, **kwargs):
try:
res = func(*args, **kwargs)
return res
except Exception as e:
callback(*c_args, **c_kwargs)
raise e
return _inner
return _out
def send_warning():
print("warning message..............")
class A(object):
#capture_exception(callback=send_warning)
def run(self):
print('run')
raise SystemError("测试异常捕获回调功能")
time.sleep(0.2)
if __name__ == '__main__':
a = A()
a.run()

Make Python unittest fail on exception from any thread

I am using the unittest framework to automate integration tests of multi-threaded python code, external hardware and embedded C. Despite my blatant abuse of a unittesting framework for integration testing, it works really well. Except for one problem: I need the test to fail if an exception is raised from any of the spawned threads. Is this possible with the unittest framework?
A simple but non-workable solution would be to either a) refactor the code to avoid multi-threading or b) test each thread separately. I cannot do that because the code interacts asynchronously with the external hardware. I have also considered implementing some kind of message passing to forward the exceptions to the main unittest thread. This would require significant testing-related changes to the code being tested, and I want to avoid that.
Time for an example. Can I modify the test script below to fail on the exception raised in my_thread without modifying the x.ExceptionRaiser class?
import unittest
import x
class Test(unittest.TestCase):
def test_x(self):
my_thread = x.ExceptionRaiser()
# Test case should fail when thread is started and raises
# an exception.
my_thread.start()
my_thread.join()
if __name__ == '__main__':
unittest.main()

At first, sys.excepthook looked like a solution. It is a global hook which is called every time an uncaught exception is thrown.
Unfortunately, this does not work. Why? well threading wraps your run function in code which prints the lovely tracebacks you see on screen (noticed how it always tells you Exception in thread {Name of your thread here}? this is how it's done).
Starting with Python 3.8, there is a function which you can override to make this work: threading.excepthook
... threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled
So what do we do? Replace this function with our logic, and voilà:
For python >= 3.8
import traceback
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self, args):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
self.__org_hook(args)
formated_exc = traceback.format_exception(args.exc_type, args.exc_value, args.exc_traceback)
self._exceptions.append('\n'.join(formated_exc))
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading.excepthook
threading.excepthook = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading.excepthook = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception(f'Exceptions in other threads: {tracebacks}')
For older versions of Python, this is a bit more complicated.
Long story short, it appears that the threading nodule has an undocumented import which does something along the lines of:
threading._format_exc = traceback.format_exc
Not very surprisingly, this function is only called when an exception is thrown from a thread's run function.
So for python <= 3.7
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
formated_exc = self.__org_hook()
self._exceptions.append(formated_exc)
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading._format_exc
threading._format_exc = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading._format_exc = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception('Exceptions in other threads: %s' % tracebacks)
Usage:
my_thread = x.ExceptionRaiser()
# will fail when thread is started and raises an exception.
with GlobalExceptionWatcher():
my_thread.start()
my_thread.join()
You still need to join yourself, but upon exit, the with-statement's context manager will check for any exception thrown in other threads, and will raise an exception appropriately.
THE CODE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED
This is an undocumented, sort-of-horrible hack. I tested it on linux and windows, and it seems to work. Use it at your own risk.

I've come across this problem myself, and the only solution I've been able to come up with is subclassing Thread to include an attribute for whether or not it terminates without an uncaught exception:
from threading import Thread
class ErrThread(Thread):
"""
A subclass of Thread that will log store exceptions if the thread does
not exit normally
"""
def run(self):
try:
Thread.run(self)
except Exception as self.err:
pass
else:
self.err = None
class TaskQueue(object):
"""
A utility class to run ErrThread objects in parallel and raises and exception
in the event that *any* of them fail.
"""
def __init__(self, *tasks):
self.threads = []
for t in tasks:
try:
self.threads.append(ErrThread(**t)) ## passing in a dict of target and args
except TypeError:
self.threads.append(ErrThread(target=t))
def run(self):
for t in self.threads:
t.start()
for t in self.threads:
t.join()
if t.err:
raise Exception('Thread %s failed with error: %s' % (t.name, t.err))

I've been using the accepted answer above for a while now, but since Python 3.8 the solution doesn't work anymore because the threading module doesn't have this _format_exc import anymore.
On the other hand the threading module now has a nice way to register custom except hooks in Python 3.8 so here is a simple solution to run unit tests which assert that some exceptions are raised inside threads:
def test_in_thread():
import threading
exceptions_caught_in_threads = {}
def custom_excepthook(args):
thread_name = args.thread.name
exceptions_caught_in_threads[thread_name] = {
'thread': args.thread,
'exception': {
'type': args.exc_type,
'value': args.exc_value,
'traceback': args.exc_traceback
}
}
# Registering our custom excepthook to catch the exception in the threads
threading.excepthook = custom_excepthook
# dummy function that raises an exception
def my_function():
raise Exception('My Exception')
# running the funciton in a thread
thread_1 = threading.Thread(name='thread_1', target=my_function, args=())
thread_1.start()
thread_1.join()
assert 'thread_1' in exceptions_caught_in_threads # there was an exception in thread 1
assert exceptions_caught_in_threads['thread_1']['exception']['type'] == Exception
assert str(exceptions_caught_in_threads['thread_1']['exception']['value']) == 'My Exception'