Develop Reference

Script stuck on exit when using atexit to terminate threads

I'm playing around with threads on python 3.7.4, and I want to use atexit to register a cleanup function that will (cleanly) terminate the threads.
For example:
# example.py
import threading
import queue
import atexit
import sys
Terminate = object()
class Worker(threading.Thread):
def __init__(self):
super().__init__()
self.queue = queue.Queue()
def send_message(self, m):
self.queue.put_nowait(m)
def run(self):
while True:
m = self.queue.get()
if m is Terminate:
break
else:
print("Received message: ", m)
def shutdown_threads(threads):
for t in threads:
print(f"Terminating thread {t}")
t.send_message(Terminate)
for t in threads:
print(f"Joining on thread {t}")
t.join()
else:
print("All threads terminated")
if __name__ == "__main__":
threads = [
Worker()
for _ in range(5)
]
atexit.register(shutdown_threads, threads)
for t in threads:
t.start()
for t in threads:
t.send_message("Hello")
#t.send_message(Terminate)
sys.exit(0)
However, it seems interacting with the threads and queues in the atexit callback creates a deadlock with some internal shutdown routine:
$ python example.py
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
^CException ignored in: <module 'threading' from '/usr/lib64/python3.7/threading.py'>
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 1308, in _shutdown
lock.acquire()
KeyboardInterrupt
Terminating thread <Worker(Thread-1, started 140612492904192)>
Terminating thread <Worker(Thread-2, started 140612484511488)>
Terminating thread <Worker(Thread-3, started 140612476118784)>
Terminating thread <Worker(Thread-4, started 140612263212800)>
Terminating thread <Worker(Thread-5, started 140612254820096)>
Joining on thread <Worker(Thread-1, stopped 140612492904192)>
Joining on thread <Worker(Thread-2, stopped 140612484511488)>
Joining on thread <Worker(Thread-3, stopped 140612476118784)>
Joining on thread <Worker(Thread-4, stopped 140612263212800)>
Joining on thread <Worker(Thread-5, stopped 140612254820096)>
All threads terminated
(the KeyboardInterrupt is me using ctrl-c since the process seems to be hanging indefinitely).
However, if I send the Terminate message before exit(uncomment the line after t.send_message("Hello")), the program doesn't hang and terminates gracefully:
$ python example.py
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Terminating thread <Worker(Thread-1, stopped 140516051592960)>
Terminating thread <Worker(Thread-2, stopped 140516043200256)>
Terminating thread <Worker(Thread-3, stopped 140515961992960)>
Terminating thread <Worker(Thread-4, stopped 140515953600256)>
Terminating thread <Worker(Thread-5, stopped 140515945207552)>
Joining on thread <Worker(Thread-1, stopped 140516051592960)>
Joining on thread <Worker(Thread-2, stopped 140516043200256)>
Joining on thread <Worker(Thread-3, stopped 140515961992960)>
Joining on thread <Worker(Thread-4, stopped 140515953600256)>
Joining on thread <Worker(Thread-5, stopped 140515945207552)>
All threads terminated
This begs the question, when does this threading._shutdown routine gets executed, relative to atexit handlers?
Does it make sense to interact with threads in atexit handlers?

You can use one daemon thread to ask your non-daemon threads to clean up gracefully. For an example where this is necessary, if you are using a third-party library that starts a non-daemon thread, you'd either have to change that library or do something like:
import threading
def monitor_thread():
main_thread = threading.main_thread()
main_thread.join()
send_signal_to_non_daemon_thread_to_gracefully_shutdown()
monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()
start_non_daemon_thread()
To put this in the context of the original poster's code (note we don't need the atexit function, since that won't get called until all the non-daemon threads are stopped):
if __name__ == "__main__":
threads = [
Worker()
for _ in range(5)
]
for t in threads:
t.start()
for t in threads:
t.send_message("Hello")
#t.send_message(Terminate)
def monitor_thread():
main_thread = threading.main_thread()
main_thread.join()
shutdown_threads(threads)
monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()

atexit.register(func) registers func as a function to be executed at termination.
After execute the last line of code (it is sys.exit(0) in above example) in main thread, threading._shutdown was invoked (by interpreter) to wait for all non-daemon threads (Workers created in above example) exit
The entire Python program exits when no alive non-daemon threads are left.
So after typing CTRL+C, the main thread was terminated by SIGINT signal, and then atexit registered functions are called by interpreter.
By the way, if you pass daemon=True to Thread.__init__, the program would run straightforward without any human interactive.

Can't catch SIGINT in multithreaded program

I've seen many topics about this particular problem but i still can't figure why i'm not catching a SIGINT in my main Thread.
Here is my code:
def connect(self, retry=100):
tries=retry
logging.info('connecting to %s' % self.path)
while True:
try:
self.sp = serial.Serial(self.path, 115200)
self.pileMessage = pilemessage.Pilemessage()
self.pileData = pilemessage.Pilemessage()
self.reception = reception.Reception(self.sp,self.pileMessage,self.pileData)
self.reception.start()
self.collisionlistener = collisionListener.CollisionListener(self)
self.message = messageThread.Message(self.pileMessage,self.collisionlistener)
self.datastreaminglistener = dataStreamingListener.DataStreamingListener(self)
self.datastreaming = dataStreaming.Data(self.pileData,self.datastreaminglistener)
return
except serial.serialutil.SerialException:
logging.info('retrying')
if not retry:
raise SpheroError('failed to connect after %d tries' % (tries-retry))
retry -= 1
def disconnect(self):
self.reception.stop()
self.message.stop()
self.datastreaming.stop()
while not self.pileData.isEmpty():
self.pileData.pop()
self.datastreaminglistener.remove()
while not self.pileMessage.isEmpty():
self.pileMessage.pop()
self.collisionlistener.remove()
self.sp.close()
if __name__ == '__main__':
import time
try:
logging.getLogger().setLevel(logging.DEBUG)
s = Sphero("/dev/rfcomm0")
s.connect()
s.set_motion_timeout(65525)
s.set_rgb(0,255,0)
s.set_back_led_output(255)
s.configure_locator(0,0)
except KeyboardInterrupt:
s.disconnect()
In the main function I call Connect() which is launching Threads over which i don't have direct controll.
When I launch this script I would like to be able to stop it when hitting Control+C by calling the "disconnect()" function which stops all the other threads.
In the code i provided it doesn't work because there is no thread in the main function. But I already tryied putting all the instuctions from Main() in a Thread with a While loop without success.
Is there a simple way to solve my problem ?
Thanx

Your indentation is messed up, but there's enough to go on.
Your main thread isn't catching SIGINT because it's not alive. There is nothing that stops your main thread from continuing past the try block, seeing no more code, and closing up shop.
I am not familiar with Sphero. I just attempted to google its docs and was linked to a bunch of 404 pages, so I'll tell you what you would normally do in a threaded environment - join your threads to the main thread so that the main thread can't finish execution before the worker threads.
for t in my_thread_list:
t.join() #main thread can't get past here until all the threads finish
If your Sphero object doesn't provide join-like functionality, you could hack something in that blocks, i.e.
raw_input('Press Enter to disconnect')
s.disconnect()

How to create global error handler in a multi-threaded python applcation

I am developing a multi-threaded application in python. I have following scenario.
There are 2-3 producer threads which communicate with DB and get some data in large chunks and fill them up in a queue
There is an intermediate worker which breaks large chunks fetched by producer threads into smaller ones and fill them up in another queue.
There are 5 consumer threads which consume queue created by intermediate worker thread.
objects of data sources are accessed by producer threads through their API. these data sources are completely separate. So these producer understands only presence or absence of data which is supposed to be given out by data source object.
I create threads of these three types and i make main thread wait for completion of these threads by calling join() on them.
Now for such a setup I want a common error handler which senses failure of any thread, any exception and decides what to do. For e.g if I press ctrl+c after I start my application, main thread dies but producer, consumer threads continue to run. I would like that once ctrl+c is pressed entire application should shut down. Similarly if some DB error occurs in data source module, then producer thread should get notified of that.
This is what I have done so far:
I have created a class ThreadManager, it's object is passed to all threads. I have written an error handler method and passed it to sys.excepthook. This handler should catch exceptions, error and then it should call methods of ThreadManager class to control the running threads. Here is snippet:
class Producer(threading.Thread):
....
def produce():
data = dataSource.getData()
class DataSource:
....
def getData():
raise Exception("critical")
def customHandler(exceptionType, value, stackTrace):
print "In custom handler"
sys.excepthook = customHandler
Now when a thread of producer class calls getData() of DataSource class, exception is thrown. But this exception is never caught by my customHandler method.
What am I missing? Also in such scenario what other strategy can I apply? Please help. Thank you for having enough patience to read all this :)

What you need is a decorator. In essence you are modifying your original function and putting in inside a try-except:
def exception_decorator(func):
def _function(*args):
try:
result = func(*args)
except:
print('*** ESC default handler ***')
os._exit(1)
return result
return _function
If your thread function is called myfunc, then you add the following line above your function definition
#exception_decorator
def myfunc():
pass;

Can't you just catch "KeyboardInterrupt" when pressing Ctrl+C and do:
for thread in threading.enumerate():
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
and have a flag in each threaded class which is self.alive
you could theoretically call thread.alive = False and have it stop gracefully?
for thread in threading.enumerate():
thread.alive = False
time.sleep(5) # Grace period
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
example:
import os
from threading import *
from time import sleep
class worker(Thread):
def __init__(self):
self.alive = True
Thread.__init__(self)
self.start()
def run(self):
while self.alive:
sleep(0.1)
runner = worker()
try:
raw_input('Press ctrl+c!')
except:
pass
for thread in enumerate():
thread.alive = False
sleep(1)
try:
thread._Thread__stop()
thread._Thread__delete()
except:
pass
# There will always be 1 thread alive and that's the __main__ thread.
while len(enumerate()) > 1:
sleep(1)
os._exit(0)
Try going about it by changing the internal system exception handler?
import sys
origExcepthook = sys.excepthook
def uberexcept(exctype, value, traceback):
if exctype == KeyboardInterrupt:
print "Gracefully shutting down all the threads"
# enumerate() thingie here.
else:
origExcepthook(exctype, value, traceback)
sys.excepthook = uberexcept

How to properly handle and retain system shutdown (and SIGTERM) in order to finish its job in Python?

Basic need : I've a Python daemon that's calling another program through os.system. My wish is to be able to properly to handle system shutdown or SIGTERM in order to let the called program return and then exiting.
What I've already tried: I've tried an approach using signal :
import signal, time
def handler(signum = None, frame = None):
print 'Signal handler called with signal', signum
time.sleep(3)
#here check if process is done
print 'Wait done'
signal.signal(signal.SIGTERM , handler)
while True:
time.sleep(6)
The usage of time.sleep doesn't seems to work and the second print is never called.
I've read few words about atexit.register(handler) instead of signal.signal(signal.SIGTERM, handler) but nothing is called on kill.

Your code does almost work, except you forgot to exit after cleaning up.
We often need to catch various other signals such as INT, HUP and QUIT, but not so much with daemons.
import sys, signal, time
def handler(signum = None, frame = None):
print 'Signal handler called with signal', signum
time.sleep(1) #here check if process is done
print 'Wait done'
sys.exit(0)
for sig in [signal.SIGTERM, signal.SIGINT, signal.SIGHUP, signal.SIGQUIT]:
signal.signal(sig, handler)
while True:
time.sleep(6)
On many systems, ordinary processes don't have much time to clean up during shutdown. To be safe, you could write an init.d script to stop your daemon and wait for it.

How to kill a child thread with Ctrl+C?

I would like to stop the execution of a process with Ctrl+C in Python. But I have read somewhere that KeyboardInterrupt exceptions are only raised in the main thread. I have also read that the main thread is blocked while the child thread executes. So how can I kill the child thread?
For instance, Ctrl+C has no effect with the following code:
def main():
try:
thread = threading.Thread(target=f)
thread.start() # thread is totally blocking (e.g. while True)
thread.join()
except KeyboardInterrupt:
print "Ctrl+C pressed..."
sys.exit(1)
def f():
while True:
pass # do the actual work

If you want to have main thread to receive the CTRL+C signal while joining, it can be done by adding timeout to join() call.
The following seems to be working (don't forget to add daemon=True if you want main to actually end):
thread1.start()
while True:
thread1.join(600)
if not thread1.isAlive():
break

The problem there is that you are using thread1.join(), which will cause your program to wait until that thread finishes to continue.
The signals will always be caught by the main process, because it's the one that receives the signals, it's the process that has threads.
Doing it as you show, you are basically running a 'normal' application, without thread features, as you start 1 thread and wait until it finishes to continue.

KeyboardInterrupt exceptions are raised only in the main thread of each process. But the method Thread.join blocks the calling thread, including KeyboardInterrupt exceptions. That is why Ctrl+C seems to have no effect.
A simple solution to your problem is to make the method Thread.join time out to unblock KeyboardInterrupt exceptions, and make the child thread daemonic to let the parent thread kill it at exit (non-daemonic child threads are not killed but joined by their parent at exit):
def main():
try:
thread = threading.Thread(target=f)
thread.daemon = True # let the parent kill the child thread at exit
thread.start()
while thread.is_alive():
thread.join(1) # time out not to block KeyboardInterrupt
except KeyboardInterrupt:
print "Ctrl+C pressed..."
sys.exit(1)
def f():
while True:
pass # do the actual work
A better solution if you control the code of the child thread is to notify the child thread to exit gracefully (instead of abruptly like with the simple solution), for instance using a threading.Event:
def main():
try:
event = threading.Event()
thread = threading.Thread(target=f, args=(event,))
thread.start()
event.wait() # wait without blocking KeyboardInterrupt
except KeyboardInterrupt:
print "Ctrl+C pressed..."
event.set() # notify the child thread to exit
sys.exit(1)
def f(event):
while not event.is_set():
pass # do the actual work