I have a script which uses threads, but it is unable to catch Ctrl + C.
Here it is the sample code to reproduce this error:
import threading
import time
import signal
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
def stop(self, signum=None, frame=None):
self._running = False
def run(self):
while self._running:
time.sleep(1)
print("Running")
if __name__ == "__main__":
try:
t = DummyThread()
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
print("This never gets printed")
t.stop()
finally:
print("Exit")
When I run python3 script.py it starts running, but it does not catch ctrl+c. I have googled it but I have not found a solution. I have to kill the script with SIGTERM, but I want DummyThread to stop gracefully.
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
The program actually does not work as expected because of those last two lines and would work without them.
The reason is that, if you press Ctrl-C, the SIGINT signal is handled by the signal handler that is set up by signal.signal and self.stop is called. So the thread should actually stop.
But in the main thread, the while True loop is still running. Since the signal has already been handled, there will be no KeyboardInterrupt exception raised by the Python runtime. Therefore you never get to the except part.
if __name__ == "__main__":
try:
t = DummyThread()
t.start()
while True: # you are stuck in this loop
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt: # this never happens
print("This never gets printed")
t.stop()
Only one signal handler should be set up to call the stop method. So there are two options to solve the problem:
Handle the signal implicitly by catching the KeyboardInterrupt exception. This is achieved by simply removing the two signal.signal(...) lines.
Set up an explicit signal handler (as you did by using signal.signal in DummyThread.__init__), but remove the while True: loop from the main thread and do not try to handle KeyboardInterrupt. Instead, just wait for the DummyThread to finish on its own by using its join method:
if __name__ == "__main__":
t = DummyThread()
t.start()
t.join()
print("Exit")
The main point is that you can't work with signals in any other Thread except the Main Thread. The Main Thread is the only one which can receive signals and handle them. I can offer the following solution, it is based on Event sync primitive.
According to Python documantation:
Signals and threads
Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.
Besides, only the main thread is allowed to set a new signal handler.
from threading import Thread, Event
import time
class DummyThread(Thread):
def __init__(self, event: Event):
Thread.__init__(self)
self.stop_event = event
def run(self):
# we are monitoring the event in the Main Thread
while not self.stop_event.is_set():
time.sleep(1)
print("Running")
# only Main Thread can make the point reachable
print("I am done !")
if __name__ == "__main__":
try:
e = Event()
t = DummyThread(e)
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
e.set()
finally:
print("Exit")
Another possible choice is to use daemon Thread for such tasks like in your code example (when you just printing smth in the screen every second, rather than e.g. close database connection or some similar task). If main thread is stoped the daemon Thread will stop too.
As shown in your code, you used KeyboardInterrupt to call stop() function. See how Listener does the same task and stops the execution which was unable to catch from Ctrl + C. You dont have to kill the script with SIGTERM anymore
import threading
import time
import signal
import os
from pynput.keyboard import Key, Listener
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
def stop(self, signum=None, frame=None):
self._running = False
print ("Bye Bye . .")
os._exit(1)
def run(self):
while self._running:
time.sleep(1)
print("Running")
if __name__ == "__main__":
t = DummyThread()
def func2():
try:
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
print("No need for this")
t.stop()
finally:
print("Exit")
def func1():
with Listener(on_press = t.stop) as listener :
listener.join()
threading.Thread(target=func1).start()
threading.Thread(target=func2).start()
Related
In many cases I have a worker thread which pops data from a Queue and acts on it. At some kind of event I want my worker thread to stop. The simple solution is to add a timeout to the get call and check the Event/flag every time the get times out. This however as two problems:
Causes an unnecessary context switch
Delays the shutdown until a timeout occurs
Is there any better way to listen both to a stop event and new data in the Queue? Is it possible to listen to two Queue's at the same time and block until there's data in the first one? (In this case one can use a second Queue just to trigger the shutdown.)
The solution I'm currently using:
from queue import Queue, Empty
from threading import Event, Thread
from time import sleep
def worker(exit_event, queue):
print("Worker started.")
while not exit_event.isSet():
try:
data = queue.get(timeout=10)
print("got {}".format(data))
except Empty:
pass
print("Worker quit.")
if __name__ == "__main__":
exit_event = Event()
queue = Queue()
th = Thread(target=worker, args=(exit_event, queue))
th.start()
queue.put("Testing")
queue.put("Hello!")
sleep(2)
print("Asking worker to quit")
exit_event.set()
th.join()
print("All done..")
I guess you may easily reduce timeout to 0.1...0.01 sec. Slightly different solution is to use the queue to send both data and control commands to the thread:
import queue
import threading
import time
THREADSTOP = 0
class ThreadControl:
def __init__(self, command):
self.command = command
def worker(q):
print("Worker started.")
while True:
data = q.get()
if isinstance(data, ThreadControl):
if data.command == THREADSTOP:
break
print("got {}".format(data))
print("Worker quit.")
if __name__ == '__main__':
q = queue.Queue()
th = threading.Thread(target=worker, args=(q,))
th.start()
q.put("Testing")
q.put("Hello!")
time.sleep(2)
print("Asking worker to quit")
q.put(ThreadControl(command=THREADSTOP)) # sending command
th.join()
print("All done..")
Another option is to use sockets instead of queues.
How to terminate the program from a thread? I want to terminate the whole program if it can not connect to the server and the user clicks on cancel.
class Client(tk.Frame):
def __init__(self, master=None):
super().__init__(master)
t = threading.Thread(target = self.connect)
t.setDaemon(True)
t.start()
def connect(self):
try:
r.connect( "localhost", 28015)
self.refresh_objects()
except r.ReqlDriverError as e:
self.db_exception_handler()
def db_exception_handler(self):
if(tk.messagebox.askretrycancel('ERROR','ERROR: Unable to connect to the database.')):
try:
r.connect( "localhost", 28015)
except r.ReqlDriverError as e:
self.db_exception_handler()
else:
root.destroy() #I want to terminate here the program
#I tried: threading.main_thread().destroy() but not working
def main():
root = tk.Tk()
root.title("IOT_LPS_Client")
cln = Client(master=root)
cln.mainloop()
try:
root.destroy()
except:
pass
if __name__ == '__main__':
main()
I strongly suggest coding a disciplined shutdown wherein the thread notifies the main thread of the need to exit, and the main thread calling sys.exit. [note that only in the main thread sys.exit will kill the process].
This blog post discusses some of the issues and solutions around this. In brief, you can use something like threading.Event() to convey the stop signal from any thread to the main thread.
import threading
import time
def worker(i):
while True:
try:
print i
time.sleep(10)
break
except Exception, msg:
print msg
threads = []
for i in range(10):
t1 = threading.Thread(target=worker, args=(i,))
threads.append(t1)
for t in threads:
t.start()
print "started all threads... waiting to be finished"
for t in threads:
t.join()
if i press ^C while the threads are running, does the thread gets the SIGINT?
if this is true, what can i do from the caller thread to stop it from propagating SIGINT to running threads?
signal handler in caller thread would prevent it?
or do i need signal handler for each thread?
if i press ^C while the threads are running, does the thread gets the SIGINT?
No. As it says in the documentation:
Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread.
You can see that this is true with a simple test:
import threading
import time
def worker():
while True:
print('Worker working')
time.sleep(0.5)
pass
worker_thread = threading.Thread(target=worker)
worker_thread.start()
while True:
print('Parent parenting')
time.sleep(0.5)
After you send SIGINT with ^C, you will see that the main thread is killed (no more 'Parent parenting' logs) and the child thread continues to run.
In your example, your child threads exit because you break out of their while loops after 10 seconds.
As referred in Python's docs, you should use the attribute daemon:
daemon: A boolean value indicating whether this thread is a daemon
thread (True) or not (False). This must be set before start() is
called, otherwise RuntimeError is raised. Its initial value is
inherited from the creating thread; the main thread is not a daemon
thread and therefore all threads created in the main thread default to
daemon = False.
The entire Python program exits when no alive non-daemon threads are
left.
New in version 2.6.
To control the CTRL+C signal, you should capture it changing the handler with the signal.signal(signal_number, handler) function. The child process inherits the signal handler for SIGINT.
import threading
import time
import signal
def worker(i):
while True:
try:
print(i)
time.sleep(10)
break
except Exception as msg:
print(msg)
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
print("I will wait for all threads... waiting to be finished")
for t in threads:
t.join()
signal.signal(signal.SIGINT, signal_handler)
threads = []
for i in range(10):
t1 = threading.Thread(target=worker, args=(i,))
threads.append(t1)
for t in threads:
t.start()
print("started all threads... waiting to be finished")
for t in threads:
t.join()
I came across a very strange issue when using threading.Event() and couldn't understand what is going on? I must have missed something, please can you point it out?
I have a Listener class which shares the same event object with signal handler, here is my simplified code:
import threading, time
class Listener(object):
def __init__(self, event):
super(Listener, self).__init__()
self.event = event
def start(self):
while not self.event.is_set():
print("Listener started, waiting for messages ...")
self.event.wait()
print("Listener is terminated ...")
self.event.clear()
event = threading.Event()
def handler(signum, frame):
global event
event.set()
print('Signal handler called with signal [%s]' % signum)
if __name__ == "__main__":
signal.signal(signal.SIGINT, handler)
listener = Listener(event)
listener.start()
After I run the code, I press ctrl+c to interrupt it, nothing actually happens. I have to use kill -9 to kill the process if I want to quit. However if I supply an argument to event.wait(), it worked. but it keeps printing out:
Listener started, waiting for messages ..."
every timedout seconds. But it will print out:
Listener is terminated ...
upon Ctrl+c which is what I want.
while not self.event.is_set():
print("Listener started, waiting for messages ...")
self.event.wait(1)
Why do I have to give a timeout argument in event.wait() to make it responding to ctrl+c event please? According to the document http://docs.python.org/2/library/threading.html#event-objects, the event.wait() Threads that call wait() once the flag is true will not block at all. I am using python 2.7.3 by the way.
There are several threads discussing related problems with python's threads, interrupts, locks, events.
For example, see here and here, but there are more.
The situation is much better in python3, where the implementation of wait() was improved, to make it intterruptible.
Will this work for you? Basically, start another thread for Listener and wait there while main thread waits for signal.
#!/usr/bin/python
import threading, signal
class Listener(threading.Thread):
def __init__(self, event):
super(Listener, self).__init__()
self.event = event
def run(self):
while not self.event.is_set():
print("Listener started, waiting for messages ...")
self.event.wait()
print("Listener is terminated ...")
self.event.clear()
event = threading.Event()
def handler(signum, frame):
global event
event.set()
print('Signal handler called with signal [%s]' % signum)
if __name__ == "__main__":
signal.signal(signal.SIGINT, handler)
listener = Listener(event)
listener.start()
while listener.is_alive():
pass
The following code is similar to the original.
Differences:
subclass the Thread class, use run (vs start)
use a simple wait() with no timeout, which is more predictable
signal handler doesn't trigger the Event directly. Instead, it just implicitly wakes up the main process, which is sitting on signal.pause().
main proc triggers Event.set() after it wakes up from pause() -- main proc will do this on any signal, not just SIGINT (control-C). For testing purposes, there's an alarm call after two seconds.
Hope this helps!
source
import signal, threading, time
class Listener(threading.Thread):
def __init__(self, event):
super(Listener, self).__init__()
self.event = event
def run(self):
print("Listener started, waiting for messages ...")
while not self.event.wait():
print('(timeout)')
print("Listener is terminated ...")
self.event.clear()
event = threading.Event()
def handler(signum, _frame):
# global event
# event.set()
print('Signal handler called with signal [%s]' % signum)
if __name__ == "__main__":
signal.signal(signal.SIGINT, handler)
signal.signal(signal.SIGALRM, handler)
signal.alarm(2)
listener = Listener(event)
listener.start()
print '* PAUSE'
signal.pause() # wait for a signal
print '* SIGNALLING'
event.set()
listener.join()
print('* DONE')
output
Listener started, waiting for messages ...
* PAUSE
Signal handler called with signal [14]
* SIGNALLING
Listener is terminated ...
* DONE
I am developing a multi-threaded application in python. I have following scenario.
There are 2-3 producer threads which communicate with DB and get some data in large chunks and fill them up in a queue
There is an intermediate worker which breaks large chunks fetched by producer threads into smaller ones and fill them up in another queue.
There are 5 consumer threads which consume queue created by intermediate worker thread.
objects of data sources are accessed by producer threads through their API. these data sources are completely separate. So these producer understands only presence or absence of data which is supposed to be given out by data source object.
I create threads of these three types and i make main thread wait for completion of these threads by calling join() on them.
Now for such a setup I want a common error handler which senses failure of any thread, any exception and decides what to do. For e.g if I press ctrl+c after I start my application, main thread dies but producer, consumer threads continue to run. I would like that once ctrl+c is pressed entire application should shut down. Similarly if some DB error occurs in data source module, then producer thread should get notified of that.
This is what I have done so far:
I have created a class ThreadManager, it's object is passed to all threads. I have written an error handler method and passed it to sys.excepthook. This handler should catch exceptions, error and then it should call methods of ThreadManager class to control the running threads. Here is snippet:
class Producer(threading.Thread):
....
def produce():
data = dataSource.getData()
class DataSource:
....
def getData():
raise Exception("critical")
def customHandler(exceptionType, value, stackTrace):
print "In custom handler"
sys.excepthook = customHandler
Now when a thread of producer class calls getData() of DataSource class, exception is thrown. But this exception is never caught by my customHandler method.
What am I missing? Also in such scenario what other strategy can I apply? Please help. Thank you for having enough patience to read all this :)
What you need is a decorator. In essence you are modifying your original function and putting in inside a try-except:
def exception_decorator(func):
def _function(*args):
try:
result = func(*args)
except:
print('*** ESC default handler ***')
os._exit(1)
return result
return _function
If your thread function is called myfunc, then you add the following line above your function definition
#exception_decorator
def myfunc():
pass;
Can't you just catch "KeyboardInterrupt" when pressing Ctrl+C and do:
for thread in threading.enumerate():
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
and have a flag in each threaded class which is self.alive
you could theoretically call thread.alive = False and have it stop gracefully?
for thread in threading.enumerate():
thread.alive = False
time.sleep(5) # Grace period
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
example:
import os
from threading import *
from time import sleep
class worker(Thread):
def __init__(self):
self.alive = True
Thread.__init__(self)
self.start()
def run(self):
while self.alive:
sleep(0.1)
runner = worker()
try:
raw_input('Press ctrl+c!')
except:
pass
for thread in enumerate():
thread.alive = False
sleep(1)
try:
thread._Thread__stop()
thread._Thread__delete()
except:
pass
# There will always be 1 thread alive and that's the __main__ thread.
while len(enumerate()) > 1:
sleep(1)
os._exit(0)
Try going about it by changing the internal system exception handler?
import sys
origExcepthook = sys.excepthook
def uberexcept(exctype, value, traceback):
if exctype == KeyboardInterrupt:
print "Gracefully shutting down all the threads"
# enumerate() thingie here.
else:
origExcepthook(exctype, value, traceback)
sys.excepthook = uberexcept