Develop Reference

Python 3.5 multiprocessing pool and queue don't work

I encounter a multiprocessing problem. The code is included below. The code can execute as expected, but when uncommenting self.queue = multiprocessing.Queue(), this program will exit immediately and it seems that the subprocess can't be started successfully.
I don't know what happened. Could someone help me out? Many Thanks!
import multiprocessing
import time
class Test:
def __init__(self):
self.pool = multiprocessing.Pool(1)
#self.queue = multiprocessing.Queue()
def subprocess(self):
for i in range(10):
print("Running")
time.sleep(1)
print("Subprocess Completed")
def start(self):
self.pool.apply_async(func=self.subprocess)
print("Subprocess has been started")
self.pool.close()
self.pool.join()
def __getstate__(self):
self_dict = self.__dict__.copy()
del self_dict['pool']
return self_dict
def __setstate__(self, state):
self.__dict__.update(state)
if __name__ == '__main__':
test = Test()
test.start()

I can reproduce your Issue and also no Traceback raised up.
This should raise the following error, don't know why not:
RuntimeError: Queue objects should only be shared between processes through inheritance
Replace your line of code with:
m = multiprocessing.Manager()
self.queue = m.Queue()
Why does this happens?
multiprocessing.Queue is for use with one Process, you are using multiprocessing.Pool, which uses multiple Process, you have to use
multiprocessing.Manager().Queue().
Tested with Python: 3.4.2

you use apply_async, which returns immediately. so you should wait for the result somewhere
under the hood, python will pickle the function to be executed to the child process. but self.process as a method is not pickle-able here (because of the self.pool attribute, see comment by ShadowRanger below).
import multiprocessing
import time
def subprocess(): # is a plain old (pickle-able) function
for i in range(10):
print("Running")
time.sleep(1)
print("Subprocess Completed")
return True
class Test:
def __init__(self):
self.pool = multiprocessing.Pool(1)
def start(self):
result = self.pool.apply_async(subprocess)
print("Subprocess has been started")
result.get() # wait for the end of subprocess
self.pool.close()
self.pool.join()
if __name__ == '__main__':
test = Test()
test.start()

Catch an exception raised by a child process in a parent process

I'm trying to catch an Exception raised by a child process, and have been running into some issues. The gist of my code is:
class CustomException(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return self.msg
def update(partition):
if os.getpid() % 2 == 0:
raise CustomException('PID was divisible by 2!')
else:
# Do something fancy
if __name__ == '__main__':
try:
some_response = get_response_from_another_method()
partition_size = 100
p = Pool(config.NUMBER_OF_PROCESSES)
for i in range(0, NUMBER_OF_PROCESSES):
partition = get_partition(some_response, partition_size)
x = p.apply_async(update, args=(partition,))
x.get()
p.close()
p.join()
except CustomException as e:
log.error('There was an error')
if email_notifier.send_notification(e.msg):
log.debug('Email notification sent')
else:
log.error('An error occurred while sending an email.')
When I run this, I am seeing:
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
task = get()
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class 'CustomException'>, ())
Is there some facility to do this? Thanks!!

In short, this is something of a quirk in Python 2, and a related issue is referenced in this bug report. It has to do with how exceptions are pickled. The simplest solution is perhaps to alter CustomException so that it calls its parent class initializer. Alternatively, if you're able, I'd suggest moving to Python 3.
For example, this code works fine in both Python 2 and Python 3:
from multiprocessing import Pool
class CustomException(Exception):
pass
def foo():
raise CustomException('PID was divisible by 2!')
pool = Pool()
result = pool.apply_async(foo, [])
But if we alter CustomException so that it has a required argument:
class CustomException(Exception):
def __init__(self, required):
self.required = required
The above example results in a TypeError being raised under Python 2. It works under Python 3.
The problem is that CustomException inherits Exception's __reduce__ method, which tells Python how to pickle an instance. The inherited __reduce__ knows nothing about CustomException's call signature, so unpickling isn't done correctly.
A quick fix is to simply call the parent class's __init__:
class CustomException(Exception):
def __init__(self, msg):
super(Exception, self).__init__()
self.msg = msg
But since you really aren't doing anything special with the message, why not just define:
class CustomException(Exception):
pass

Why no errors from multiprocessing is reported in Python and how to switch on reporting errors?

I setup some simple code to test some problem handling with multiprocessing and I can not track the bug inside this code because not feedback from processes. How can I receive exception from subprocesses since now I am blind to it. How to debug this code.
# coding=utf-8
import multiprocessing
import multiprocessing.managers
import logging
def callback(result):
print multiprocessing.current_process().name, 'callback', result
def worker(io_lock, value):
# error
raise RuntimeError()
result = value + 1
with io_lock:
print multiprocessing.current_process().name, value, result
return result
def main():
manager = multiprocessing.Manager()
io_lock = manager.Lock()
pool = multiprocessing.Pool(multiprocessing.cpu_count())
for i in range(10):
print pool.apply_async(worker, args=(io_lock, i), callback = callback)
pool.close()
pool.join()
if __name__ == '__main__':
logging.basicConfig(level = logging.DEBUG)
main()

You can try, catch, and log exceptions that occur inside your worker processes. Something like this
def worker(io_lock, value):
try:
raise RuntimeError('URGH') # Or do your actual work here
except:
logging.exception('Exception occurred: ')
raise # Re-raise the exception so that the process exits
The exception log handler will automatically include the stacktrace.

Post mortem debugging of the multi-thread scripts

I am trying to debugging multi-thread script. Once the exception is
raised I want to:
report it to monitoring system (just print in following example)
stop whole script (including all other threads)
call post mortem debugger prompt in a perspective raised exception
I prepare pretty complicated example to show how I tried to solve it:
#!/usr/bin/env python
import threading
import inspect
import traceback
import sys
import os
import time
def POST_PORTEM_DEBUGGER(type, value, tb):
traceback.print_exception(type, value, tb)
print
if hasattr(sys, 'ps1') or not sys.stderr.isatty():
import rpdb
rpdb.pdb.pm()
else:
import pdb
pdb.pm()
sys.excepthook = POST_PORTEM_DEBUGGER
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.exception = None
self.info = None
self.the_calling_script_name = os.path.abspath(inspect.currentframe().f_back.f_code.co_filename)
def main(self):
"Virtual method to be implemented by inherited worker"
return self
def run(self):
try:
self.main()
except Exception as exception:
self.exception = exception
self.info = traceback.extract_tb(sys.exc_info()[2])[-1]
# because of bug http://bugs.python.org/issue1230540
# I cannot use just "raise" under threading.Thread
sys.excepthook(*sys.exc_info())
def __del__(self):
print 'MyThread via {} catch "{}: {}" in {}() from {}:{}: {}'.format(self.the_calling_script_name, type(self.exception).__name__, str(self.exception), self.info[2], os.path.basename(self.info[0]), self.info[1], self.info[3])
class Worker(MyThread):
def __init__(self):
super(Worker, self).__init__()
def main(self):
""" worker job """
counter = 0
while True:
counter += 1
print self
time.sleep(1.0)
if counter == 3:
pass # print 1/0
def main():
Worker().start()
counter = 1
while True:
counter += 1
time.sleep(1.0)
if counter == 3:
pass # print 1/0
if __name__ == '__main__':
main()
The trick with
sys.excepthook = POST_PORTEM_DEBUGGER
works perfectly if no threads are involved. I found that in case of
multi-thread script I can use rpdb for debuggig by calling:
import rpdb; rpdb.set_trace()
It works perfectly for defined breakpoint but I want to debug
multi-thread script post mortem (after the uncatched exception is
raised). When I try to use rpdb in the POST_PORTEM_DEBUGGER function
with multi-thread application I get following:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "./demo.py", line 49, in run
sys.excepthook(*sys.exc_info())
File "./demo.py", line 22, in POST_PORTEM_DEBUGGER
pdb.pm()
File "/usr/lib/python2.7/pdb.py", line 1270, in pm
post_mortem(sys.last_traceback)
AttributeError: 'module' object has no attribute 'last_traceback'
I looks like the
sys.excepthook(*sys.exc_info())
did not set up all what the raise command does.
I want the same behavior if the exception is raised in main() even
under started thread.

(I haven't tested my answer, but it seems to me that...)
The call to pdb.pm (pm="post mortem") fails simply because there had been no "mortem" prior to it. I.e. the program is still running.
Looking at the pdb source code, you find the implementation of pdb.pm:
def pm():
post_mortem(sys.last_traceback)
which makes me guess that what you actually want to do is call pdb.post_mortem() with no args. Looks like the default behavior does exactly what you need.
Some more source code (notice the t = sys.exc_info()[2] line):
def post_mortem(t=None):
# handling the default
if t is None:
# sys.exc_info() returns (type, value, traceback) if an exception is
# being handled, otherwise it returns None
t = sys.exc_info()[2]
if t is None:
raise ValueError("A valid traceback must be passed if no "
"exception is being handled")
p = Pdb()
p.reset()
p.interaction(None, t)

Building on #shx2's above, I now use the following pattern in the context of multithreading.
import sys, pdb
try:
... # logic that may fail
except exception as exc:
pdb.post_mortem(exc.__traceback__)
Here is a more verbose alternative:
import sys, pdb
try:
... # logic that may fail
except exception as exc:
if hasattr(sys, "last_traceback"):
pdb.pm()
else:
pdb.post_mortem(exc.__traceback__)

This can help:
import sys
from IPython.core import ultratb
sys.excepthook = ultratb.FormattedTB(mode='Verbose', color_scheme='Linux',
call_pdb=True, ostream=sys.__stdout__)

Make Python unittest fail on exception from any thread

I am using the unittest framework to automate integration tests of multi-threaded python code, external hardware and embedded C. Despite my blatant abuse of a unittesting framework for integration testing, it works really well. Except for one problem: I need the test to fail if an exception is raised from any of the spawned threads. Is this possible with the unittest framework?
A simple but non-workable solution would be to either a) refactor the code to avoid multi-threading or b) test each thread separately. I cannot do that because the code interacts asynchronously with the external hardware. I have also considered implementing some kind of message passing to forward the exceptions to the main unittest thread. This would require significant testing-related changes to the code being tested, and I want to avoid that.
Time for an example. Can I modify the test script below to fail on the exception raised in my_thread without modifying the x.ExceptionRaiser class?
import unittest
import x
class Test(unittest.TestCase):
def test_x(self):
my_thread = x.ExceptionRaiser()
# Test case should fail when thread is started and raises
# an exception.
my_thread.start()
my_thread.join()
if __name__ == '__main__':
unittest.main()

At first, sys.excepthook looked like a solution. It is a global hook which is called every time an uncaught exception is thrown.
Unfortunately, this does not work. Why? well threading wraps your run function in code which prints the lovely tracebacks you see on screen (noticed how it always tells you Exception in thread {Name of your thread here}? this is how it's done).
Starting with Python 3.8, there is a function which you can override to make this work: threading.excepthook
... threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled
So what do we do? Replace this function with our logic, and voilà:
For python >= 3.8
import traceback
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self, args):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
self.__org_hook(args)
formated_exc = traceback.format_exception(args.exc_type, args.exc_value, args.exc_traceback)
self._exceptions.append('\n'.join(formated_exc))
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading.excepthook
threading.excepthook = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading.excepthook = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception(f'Exceptions in other threads: {tracebacks}')
For older versions of Python, this is a bit more complicated.
Long story short, it appears that the threading nodule has an undocumented import which does something along the lines of:
threading._format_exc = traceback.format_exc
Not very surprisingly, this function is only called when an exception is thrown from a thread's run function.
So for python <= 3.7
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
formated_exc = self.__org_hook()
self._exceptions.append(formated_exc)
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading._format_exc
threading._format_exc = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading._format_exc = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception('Exceptions in other threads: %s' % tracebacks)
Usage:
my_thread = x.ExceptionRaiser()
# will fail when thread is started and raises an exception.
with GlobalExceptionWatcher():
my_thread.start()
my_thread.join()
You still need to join yourself, but upon exit, the with-statement's context manager will check for any exception thrown in other threads, and will raise an exception appropriately.
THE CODE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED
This is an undocumented, sort-of-horrible hack. I tested it on linux and windows, and it seems to work. Use it at your own risk.

I've come across this problem myself, and the only solution I've been able to come up with is subclassing Thread to include an attribute for whether or not it terminates without an uncaught exception:
from threading import Thread
class ErrThread(Thread):
"""
A subclass of Thread that will log store exceptions if the thread does
not exit normally
"""
def run(self):
try:
Thread.run(self)
except Exception as self.err:
pass
else:
self.err = None
class TaskQueue(object):
"""
A utility class to run ErrThread objects in parallel and raises and exception
in the event that *any* of them fail.
"""
def __init__(self, *tasks):
self.threads = []
for t in tasks:
try:
self.threads.append(ErrThread(**t)) ## passing in a dict of target and args
except TypeError:
self.threads.append(ErrThread(target=t))
def run(self):
for t in self.threads:
t.start()
for t in self.threads:
t.join()
if t.err:
raise Exception('Thread %s failed with error: %s' % (t.name, t.err))

I've been using the accepted answer above for a while now, but since Python 3.8 the solution doesn't work anymore because the threading module doesn't have this _format_exc import anymore.
On the other hand the threading module now has a nice way to register custom except hooks in Python 3.8 so here is a simple solution to run unit tests which assert that some exceptions are raised inside threads:
def test_in_thread():
import threading
exceptions_caught_in_threads = {}
def custom_excepthook(args):
thread_name = args.thread.name
exceptions_caught_in_threads[thread_name] = {
'thread': args.thread,
'exception': {
'type': args.exc_type,
'value': args.exc_value,
'traceback': args.exc_traceback
}
}
# Registering our custom excepthook to catch the exception in the threads
threading.excepthook = custom_excepthook
# dummy function that raises an exception
def my_function():
raise Exception('My Exception')
# running the funciton in a thread
thread_1 = threading.Thread(name='thread_1', target=my_function, args=())
thread_1.start()
thread_1.join()
assert 'thread_1' in exceptions_caught_in_threads # there was an exception in thread 1
assert exceptions_caught_in_threads['thread_1']['exception']['type'] == Exception
assert str(exceptions_caught_in_threads['thread_1']['exception']['value']) == 'My Exception'