import threading
import time
def worker(i):
while True:
try:
print i
time.sleep(10)
break
except Exception, msg:
print msg
threads = []
for i in range(10):
t1 = threading.Thread(target=worker, args=(i,))
threads.append(t1)
for t in threads:
t.start()
print "started all threads... waiting to be finished"
for t in threads:
t.join()
if i press ^C while the threads are running, does the thread gets the SIGINT?
if this is true, what can i do from the caller thread to stop it from propagating SIGINT to running threads?
signal handler in caller thread would prevent it?
or do i need signal handler for each thread?
if i press ^C while the threads are running, does the thread gets the SIGINT?
No. As it says in the documentation:
Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread.
You can see that this is true with a simple test:
import threading
import time
def worker():
while True:
print('Worker working')
time.sleep(0.5)
pass
worker_thread = threading.Thread(target=worker)
worker_thread.start()
while True:
print('Parent parenting')
time.sleep(0.5)
After you send SIGINT with ^C, you will see that the main thread is killed (no more 'Parent parenting' logs) and the child thread continues to run.
In your example, your child threads exit because you break out of their while loops after 10 seconds.
As referred in Python's docs, you should use the attribute daemon:
daemon: A boolean value indicating whether this thread is a daemon
thread (True) or not (False). This must be set before start() is
called, otherwise RuntimeError is raised. Its initial value is
inherited from the creating thread; the main thread is not a daemon
thread and therefore all threads created in the main thread default to
daemon = False.
The entire Python program exits when no alive non-daemon threads are
left.
New in version 2.6.
To control the CTRL+C signal, you should capture it changing the handler with the signal.signal(signal_number, handler) function. The child process inherits the signal handler for SIGINT.
import threading
import time
import signal
def worker(i):
while True:
try:
print(i)
time.sleep(10)
break
except Exception as msg:
print(msg)
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
print("I will wait for all threads... waiting to be finished")
for t in threads:
t.join()
signal.signal(signal.SIGINT, signal_handler)
threads = []
for i in range(10):
t1 = threading.Thread(target=worker, args=(i,))
threads.append(t1)
for t in threads:
t.start()
print("started all threads... waiting to be finished")
for t in threads:
t.join()
Related
I have a script which uses threads, but it is unable to catch Ctrl + C.
Here it is the sample code to reproduce this error:
import threading
import time
import signal
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
def stop(self, signum=None, frame=None):
self._running = False
def run(self):
while self._running:
time.sleep(1)
print("Running")
if __name__ == "__main__":
try:
t = DummyThread()
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
print("This never gets printed")
t.stop()
finally:
print("Exit")
When I run python3 script.py it starts running, but it does not catch ctrl+c. I have googled it but I have not found a solution. I have to kill the script with SIGTERM, but I want DummyThread to stop gracefully.
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
The program actually does not work as expected because of those last two lines and would work without them.
The reason is that, if you press Ctrl-C, the SIGINT signal is handled by the signal handler that is set up by signal.signal and self.stop is called. So the thread should actually stop.
But in the main thread, the while True loop is still running. Since the signal has already been handled, there will be no KeyboardInterrupt exception raised by the Python runtime. Therefore you never get to the except part.
if __name__ == "__main__":
try:
t = DummyThread()
t.start()
while True: # you are stuck in this loop
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt: # this never happens
print("This never gets printed")
t.stop()
Only one signal handler should be set up to call the stop method. So there are two options to solve the problem:
Handle the signal implicitly by catching the KeyboardInterrupt exception. This is achieved by simply removing the two signal.signal(...) lines.
Set up an explicit signal handler (as you did by using signal.signal in DummyThread.__init__), but remove the while True: loop from the main thread and do not try to handle KeyboardInterrupt. Instead, just wait for the DummyThread to finish on its own by using its join method:
if __name__ == "__main__":
t = DummyThread()
t.start()
t.join()
print("Exit")
The main point is that you can't work with signals in any other Thread except the Main Thread. The Main Thread is the only one which can receive signals and handle them. I can offer the following solution, it is based on Event sync primitive.
According to Python documantation:
Signals and threads
Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.
Besides, only the main thread is allowed to set a new signal handler.
from threading import Thread, Event
import time
class DummyThread(Thread):
def __init__(self, event: Event):
Thread.__init__(self)
self.stop_event = event
def run(self):
# we are monitoring the event in the Main Thread
while not self.stop_event.is_set():
time.sleep(1)
print("Running")
# only Main Thread can make the point reachable
print("I am done !")
if __name__ == "__main__":
try:
e = Event()
t = DummyThread(e)
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
e.set()
finally:
print("Exit")
Another possible choice is to use daemon Thread for such tasks like in your code example (when you just printing smth in the screen every second, rather than e.g. close database connection or some similar task). If main thread is stoped the daemon Thread will stop too.
As shown in your code, you used KeyboardInterrupt to call stop() function. See how Listener does the same task and stops the execution which was unable to catch from Ctrl + C. You dont have to kill the script with SIGTERM anymore
import threading
import time
import signal
import os
from pynput.keyboard import Key, Listener
class DummyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._running = True
signal.signal(signal.SIGINT, self.stop)
signal.signal(signal.SIGTERM, self.stop)
def stop(self, signum=None, frame=None):
self._running = False
print ("Bye Bye . .")
os._exit(1)
def run(self):
while self._running:
time.sleep(1)
print("Running")
if __name__ == "__main__":
t = DummyThread()
def func2():
try:
t.start()
while True:
print("Main thread running")
time.sleep(0.5)
except KeyboardInterrupt:
print("No need for this")
t.stop()
finally:
print("Exit")
def func1():
with Listener(on_press = t.stop) as listener :
listener.join()
threading.Thread(target=func1).start()
threading.Thread(target=func2).start()
I'm playing around with threads on python 3.7.4, and I want to use atexit to register a cleanup function that will (cleanly) terminate the threads.
For example:
# example.py
import threading
import queue
import atexit
import sys
Terminate = object()
class Worker(threading.Thread):
def __init__(self):
super().__init__()
self.queue = queue.Queue()
def send_message(self, m):
self.queue.put_nowait(m)
def run(self):
while True:
m = self.queue.get()
if m is Terminate:
break
else:
print("Received message: ", m)
def shutdown_threads(threads):
for t in threads:
print(f"Terminating thread {t}")
t.send_message(Terminate)
for t in threads:
print(f"Joining on thread {t}")
t.join()
else:
print("All threads terminated")
if __name__ == "__main__":
threads = [
Worker()
for _ in range(5)
]
atexit.register(shutdown_threads, threads)
for t in threads:
t.start()
for t in threads:
t.send_message("Hello")
#t.send_message(Terminate)
sys.exit(0)
However, it seems interacting with the threads and queues in the atexit callback creates a deadlock with some internal shutdown routine:
$ python example.py
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
^CException ignored in: <module 'threading' from '/usr/lib64/python3.7/threading.py'>
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 1308, in _shutdown
lock.acquire()
KeyboardInterrupt
Terminating thread <Worker(Thread-1, started 140612492904192)>
Terminating thread <Worker(Thread-2, started 140612484511488)>
Terminating thread <Worker(Thread-3, started 140612476118784)>
Terminating thread <Worker(Thread-4, started 140612263212800)>
Terminating thread <Worker(Thread-5, started 140612254820096)>
Joining on thread <Worker(Thread-1, stopped 140612492904192)>
Joining on thread <Worker(Thread-2, stopped 140612484511488)>
Joining on thread <Worker(Thread-3, stopped 140612476118784)>
Joining on thread <Worker(Thread-4, stopped 140612263212800)>
Joining on thread <Worker(Thread-5, stopped 140612254820096)>
All threads terminated
(the KeyboardInterrupt is me using ctrl-c since the process seems to be hanging indefinitely).
However, if I send the Terminate message before exit(uncomment the line after t.send_message("Hello")), the program doesn't hang and terminates gracefully:
$ python example.py
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Received message: Hello
Terminating thread <Worker(Thread-1, stopped 140516051592960)>
Terminating thread <Worker(Thread-2, stopped 140516043200256)>
Terminating thread <Worker(Thread-3, stopped 140515961992960)>
Terminating thread <Worker(Thread-4, stopped 140515953600256)>
Terminating thread <Worker(Thread-5, stopped 140515945207552)>
Joining on thread <Worker(Thread-1, stopped 140516051592960)>
Joining on thread <Worker(Thread-2, stopped 140516043200256)>
Joining on thread <Worker(Thread-3, stopped 140515961992960)>
Joining on thread <Worker(Thread-4, stopped 140515953600256)>
Joining on thread <Worker(Thread-5, stopped 140515945207552)>
All threads terminated
This begs the question, when does this threading._shutdown routine gets executed, relative to atexit handlers?
Does it make sense to interact with threads in atexit handlers?
You can use one daemon thread to ask your non-daemon threads to clean up gracefully. For an example where this is necessary, if you are using a third-party library that starts a non-daemon thread, you'd either have to change that library or do something like:
import threading
def monitor_thread():
main_thread = threading.main_thread()
main_thread.join()
send_signal_to_non_daemon_thread_to_gracefully_shutdown()
monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()
start_non_daemon_thread()
To put this in the context of the original poster's code (note we don't need the atexit function, since that won't get called until all the non-daemon threads are stopped):
if __name__ == "__main__":
threads = [
Worker()
for _ in range(5)
]
for t in threads:
t.start()
for t in threads:
t.send_message("Hello")
#t.send_message(Terminate)
def monitor_thread():
main_thread = threading.main_thread()
main_thread.join()
shutdown_threads(threads)
monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()
atexit.register(func) registers func as a function to be executed at termination.
After execute the last line of code (it is sys.exit(0) in above example) in main thread, threading._shutdown was invoked (by interpreter) to wait for all non-daemon threads (Workers created in above example) exit
The entire Python program exits when no alive non-daemon threads are left.
So after typing CTRL+C, the main thread was terminated by SIGINT signal, and then atexit registered functions are called by interpreter.
By the way, if you pass daemon=True to Thread.__init__, the program would run straightforward without any human interactive.
I have a program that has two threads, the main thread and one additional that works on handling jobs from a FIFO queue.
Something like this:
import queue
import threading
q = queue.Queue()
def _worker():
while True:
msg = q.get(block=True)
print(msg)
q.task_done()
t = threading.Thread(target=_worker)
#t.daemon = True
t.start()
q.put('asdf-1')
q.put('asdf-2')
q.put('asdf-4')
q.put('asdf-4')
What I want to accomplish is basically to make sure the queue is emptied before the main thread exits.
If I set t.daemon to be True the program will exit before the queue is emptied, however if it's set to False the program will never exit. Is there some way to make sure the thread running the _worker() method clears the queue on main thread exit?
The comments touch on using .join(), but depending on your use case, using a join may make threading pointless.
I assume that your main thread will be doing things other than adding items to the queue - and may be shut down at any point, you just want to ensure that your queue is empty before shutting down is complete.
At the end of your main thread, you could add a simple empty check in a loop.
while not q.empty():
sleep(1)
If you don't set t.daemon = True then the thread will never finish. Setting the thread as a daemon thread will mean that the thread does not cause your program to stay running when the main thread finishes.
Put a special item (e.g. None) in the queue, that signals the worker thread to stop.
import queue
import threading
q = queue.Queue()
def _worker():
while True:
msg = q.get(block=True)
if msg is None:
return
print(msg) # do your stuff here
t = threading.Thread(target=_worker)
#t.daemon = True
t.start()
q.put('asdf-1')
q.put('asdf-2')
q.put('asdf-4')
q.put('asdf-4')
q.put(None)
t.join()
I want to use threads to do some blocking work. What should I do to:
Spawn a thread safely
Do useful work
Wait until the thread finishes
Continue with the function
Here is my code:
import threading
def my_thread(self):
# Wait for the server to respond..
def main():
a = threading.thread(target=my_thread)
a.start()
# Do other stuff here
You can use Thread.join. Few lines from docs.
Wait until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception – or until the optional timeout occurs.
For your example it will be like.
def main():
a = threading.thread(target = my_thread)
a.start()
a.join()
I would like to stop the execution of a process with Ctrl+C in Python. But I have read somewhere that KeyboardInterrupt exceptions are only raised in the main thread. I have also read that the main thread is blocked while the child thread executes. So how can I kill the child thread?
For instance, Ctrl+C has no effect with the following code:
def main():
try:
thread = threading.Thread(target=f)
thread.start() # thread is totally blocking (e.g. while True)
thread.join()
except KeyboardInterrupt:
print "Ctrl+C pressed..."
sys.exit(1)
def f():
while True:
pass # do the actual work
If you want to have main thread to receive the CTRL+C signal while joining, it can be done by adding timeout to join() call.
The following seems to be working (don't forget to add daemon=True if you want main to actually end):
thread1.start()
while True:
thread1.join(600)
if not thread1.isAlive():
break
The problem there is that you are using thread1.join(), which will cause your program to wait until that thread finishes to continue.
The signals will always be caught by the main process, because it's the one that receives the signals, it's the process that has threads.
Doing it as you show, you are basically running a 'normal' application, without thread features, as you start 1 thread and wait until it finishes to continue.
KeyboardInterrupt exceptions are raised only in the main thread of each process. But the method Thread.join blocks the calling thread, including KeyboardInterrupt exceptions. That is why Ctrl+C seems to have no effect.
A simple solution to your problem is to make the method Thread.join time out to unblock KeyboardInterrupt exceptions, and make the child thread daemonic to let the parent thread kill it at exit (non-daemonic child threads are not killed but joined by their parent at exit):
def main():
try:
thread = threading.Thread(target=f)
thread.daemon = True # let the parent kill the child thread at exit
thread.start()
while thread.is_alive():
thread.join(1) # time out not to block KeyboardInterrupt
except KeyboardInterrupt:
print "Ctrl+C pressed..."
sys.exit(1)
def f():
while True:
pass # do the actual work
A better solution if you control the code of the child thread is to notify the child thread to exit gracefully (instead of abruptly like with the simple solution), for instance using a threading.Event:
def main():
try:
event = threading.Event()
thread = threading.Thread(target=f, args=(event,))
thread.start()
event.wait() # wait without blocking KeyboardInterrupt
except KeyboardInterrupt:
print "Ctrl+C pressed..."
event.set() # notify the child thread to exit
sys.exit(1)
def f(event):
while not event.is_set():
pass # do the actual work