In a multithreaded Python program, one thread sometimes asks for console input using the built-in raw_input(). I'd like to be able to be able to close the program while at a raw_input prompt by typing ^C at the shell (i.e., with a SIGINT signal). However, when the child thread is executing raw_input, typing ^C does nothing -- the KeyboardInterrupt is not raised until I hit return (leaving raw_input).
For example, in the following program:
import threading
class T(threading.Thread):
def run(self):
x = raw_input()
print x
if __name__ == '__main__':
t = T()
t.start()
t.join()
Typing ^C does nothing until after the input is finished. However, if we just call T().run() (i.e., the single-threaded case: just run raw_input in the main thread), ^C closes the program immediately.
Presumably, this is because SIGINT is sent to the main thread, which is suspended (waiting for the GIL) while the forked thread blocks on the console read. The main thread does not get to execute its signal handler until it grabs the GIL after raw_input returns. (Please correct me if I'm wrong about this -- I'm not an expert on Python's threading implementation.)
Is there a way to read from stdin in a raw_input-like way while allowing the SIGINT to be handled by the main thread and thus bring down the whole process?
[I've observed the behavior above on Mac OS X and a few different Linuxes.]
Edit: I've mischaracterized the underlying problem above. On further investigation, it's the main thread's call to join() that's preventing signal handling: Guido van Rossum himself has explained that the underlying lock acquire in join is uninterruptible. This means that the signal is actually being deferred until the entire thread finishes -- so this really has nothing to do with raw_input at all (just the fact that the background thread is blocking so that the join does not complete).
When join is called with no timeout, it is uninterruptable, but when it is called with a timeout, it is interruptable. Try adding an arbitrary timeout and putting it in a while loop:
while my_thread.isAlive():
my_thread.join(5.0)
There is really no easy way around this, period.
One approach is to reorganize and break up your code in a way that parts of functions which need Ctrl-C interruptibility are executed on the main thread. You use queues to send execution requests and likewise for the result values. You need one input queue for the main thread, and one output queue per non-main thread; and a coordinated main thread exit. Obviously, only one blocking function is executed at any given time this way, which may not be what you want.
Here's a working example of this idea with slightly perverse use of semaphores for the coordinated main thread exit.
Related
Python has async kill signal handling:
import signal
import sys
def signal_handler(*_):
print("\nExiting...")
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
From what I understand, when Python receives the signal registered here, it stops doing whatever it was doing, and executes the signal handler. If the signal handler doesn't stop the program, when it returns, Python continues doing whatever it was doing.
Is this some sort of special case? It seems strange that something can just interrupt the main thread, make it jump to a complete different function (the signal handler), and then return to what it was doing when the handler returns. I've been programming in Python for a few years, and I'm not aware of anything else in Python that allows you to register a function that at some point just forces the main thread to temporarily do something else no matter what it was doing, before returning to it. Sure, Python has Async support, but you need to await things there before Python will switch what it is executing. The main thread here is not doing any kind of awaiting before focus is taken away from whatever it was doing.
I'm currently studying Rust. As far as I can tell, Rust has a few options for letting you handle kill signals, including:
Running a background thread that watches for an incoming kill signal (like SIGINT), and sets a boolean for your main thread to check, and act on
Running a background thread that watches for an incoming kill signal, and does something about it
Neither interrupt the main thread, and make it execute the signal handler before returning the main thread to what it was doing. This sounds like a Python-specific way to handle kill signals that is built into the interpreter itself, and only possible in a higher level language like Python, where you can have the interpreter run code in-between each line of user code it runs. Or am I wrong?
this is done in python 2.7.12
serialHelper is a class module arround python serial and this code does work nicely
#!/usr/bin/env python
import threading
from time import sleep
import serialHelper
sh = serialHelper.SerialHelper()
def serialGetter():
h = 0
while True:
h = h + 1
s_resp = sh.getResponse()
print ('response ' + s_resp)
sleep(3)
if __name__ == '__main__':
try:
t = threading.Thread(target=sh.serialReader)
t.setDaemon(True)
t.start()
serialGetter()
#tSR = threading.Thread(target=serialGetter)
#tSR.setDaemon(True)
#tSR.start()
except Exception as e:
print (e)
however the attemp to run serialGetter as thread as remarked it just dies.
Any reason why that function can not run as thread ?
Quoting from the Python documentation:
The entire Python program exits when no alive non-daemon threads are left.
So if you setDaemon(True) every new thread and then exit the main thread (by falling off the end of the script), the whole program will exit immediately. This kills all of the threads. Either don't use setDaemon(True), or don't exit the main thread without first calling join() on all of the threads you want to wait for.
Stepping back for a moment, it may help to think about the intended use case of a daemon thread. In Unix, a daemon is a process that runs in the background and (typically) serves requests or performs operations, either on behalf of remote clients over the network or local processes. The same basic idea applies to daemon threads:
You launch the daemon thread with some kind of work queue.
When you need some work done on the thread, you hand it a work object.
When you want the result of that work, you use an event or a future to wait for it to complete.
After requesting some work, you always eventually wait for it to complete, or perhaps cancel it (if your worker protocol supports cancellation).
You don't have to clean up the daemon thread at program termination. It just quietly goes away when there are no other threads left.
The problem is step (4). If you forget about some work object, and exit the app without waiting for it to complete, the work may get interrupted. Daemon threads don't gracefully shut down, so you could leave the outside world in an inconsistent state (e.g. an incomplete database transaction, a file that never got closed, etc.). It's often better to use a regular thread, and replace step (5) with an explicit "Finish up your work and shut down" work object that the main thread hands to the worker thread before exiting. The worker thread then recognizes this object, stops waiting on the work queue, and terminates itself once it's no longer doing anything else. This is slightly more up-front work, but is much safer in the event that a work object is inadvertently abandoned.
Because of all of the above, I recommend not using daemon threads unless you have a strong reason for them.
I was really confused by sys.exit() in python.
In python documentation, it says "Exit from Python"; does that mean when sys.exit() is called in a python program, the process will exit? If so, the code below shows a different result:
import sys
import time
import threading
def threadrun():
while(True):
time.sleep(1)
if __name__=="__main__":
t=threading.Thread(target=threadrun)
t.start()
sys.exit()
Launching this program in linux, result was not the expected one as python documentation says but still run in the system, so what does sys.exit() really do?
(Paraphrasing what's in the Python 2 documentation for Thread Objects)
Normally a Python program exits only when there's nothing but daemon✶
threads (ignoring itself) left running. The “main thread” object which corresponds to the initial thread of control in the program isn't a daemon thread. Threads created using threading.Thread inherit their daemonic status from the creating thread, so if that's the main thread, they will also be non-daemonic.
This means that by default any threads created and started by your main program will prevent it from exiting if they are still running when the main thread is terminated (by sys.exit() or simply by just hitting the end of its code). In other words, the program exits only when no alive non‑daemon threads (i.e. only daemon threads) are left.
You can override this default behavior by explicitly setting✶✶ the
daemon property of any
created thread objects to True before starting it.
if __name__=="__main__":
t = threading.Thread(target=threadrun)
t.daemon = True # Explicitly set property.
t.start()
sys.exit()
Which will allow the program to actually end when sys.exit() is called (although calling it explicitly like that would not be necessary since presumably the code above would be at the end of the script anyway).
✶ A daemon thread is one that runs in the background and does not prevent the interpreter from exiting. See Daemon Threads Explanation.
✶✶ In Python 3.3, a daemon keyword argument with a default value of None was added to the Thread
class constructor
which means that, starting from that version onwards, you can simply use:
# Sets whether the thread is daemonic via "daemon" keyword argument.
t = threading.Thread(target=threadrun, daemon=True)
However, doing it separately via an explicit attribute assignment statement
still works, and would therefore be the more version-portable way of
doing it.
As per the documentation sys.exit() raises SystemExit:
Exit the interpreter by raising SystemExit(status).
If SystemExit reaches the default exception handler,
it calls handle_system_exit(), which more or less pushes through to Py_Finalize(), which in turn calls wait_for_thread_shutdown() in Python 2, so sys.exit() is the same as the normal falling off the bottom of the main module in waiting for all non-daemon threads to terminate.
In your case, the end of the program is when the last thread will be terminated. Maybe kind of join() method(like in Java) in python will wait for other threads.
Please, read this article(: there is a good explanation how to play with threads in your case
Use of threading.Thread.join()
and
documentation
https://docs.python.org/2/library/threading.html (but relax, it is only for additional knowledge.
and read this article about daemon property(if you do not want to wait for others threads become terminated
Meaning of daemon property on Python Threads
I am trying to write a unix client program that is listening to a socket, stdin, and reading from file descriptors. I assign each of these tasks to an individual thread and have them successfully communicating with the "main" application using synchronized queues and a semaphore. The problem is that when I want to shutdown these child threads they are all blocking on input. Also, the threads cannot register signal handlers in the threads because in Python only the main thread of execution is allowed to do so.
Any suggestions?
There is no good way to work around this, especially when the thread is blocking.
I had a similar issue ( Python: How to terminate a blocking thread) and the only way I was able to stop my threads was to close the underlying connection. Which resulted in the thread that was blocking to raise and exception and then allowed me to check the stop flag and close.
Example code:
class Example(object):
def __init__(self):
self.stop = threading.Event()
self.connection = Connection()
self.mythread = Thread(target=self.dowork)
self.mythread.start()
def dowork(self):
while(not self.stop.is_set()):
try:
blockingcall()
except CommunicationException:
pass
def terminate():
self.stop.set()
self.connection.close()
self.mythread.join()
Another thing to note is commonly blocking operations generally offer up a timeout. If you have that option I would consider using it. My last comment is that you could always set the thread to deamonic,
From the pydoc :
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.
Also, the threads cannot register signal handlers
Signals to kill threads is potentially horrible, especially in C, especially if you allocate memory as part of the thread, since it won't be freed when that particular thread dies (as it belongs to the heap of the process). There is no garbage collection in C, so if that pointer goes out of scope, it's gone out of scope, the memory remains allocated. So just be careful with that one - only do it that way in C if you're going to actually kill all the threads and end the process so that the memory is handed back to the OS - adding and removing threads from a threadpool for example will give you a memory leak.
The problem is that when I want to shutdown these child threads they are all blocking on input.
Funnily enough I've been fighting with the same thing recently. The solution is literally don't make blocking calls without a timeout. So, for example, what you want ideally is:
def threadfunc(running):
while running:
blockingcall(timeout=1)
where running is passed from the controlling thread - I've never used threading but I have used multiprocessing and with this you actually need to pass an Event() object and check is_set(). But you asked for design patterns, that's the basic idea.
Then, when you want this thread to end, you run:
running.clear()
mythread.join()
and your main thread should then allow your client thread to handle its last call, and return, and the whole program folds up nicely.
What do you do if you have a blocking call without a timeout? Use the asynchronous option, and sleep (as in call whatever method you have to suspend the thread for a period of time so you're not spinning) if you need to. There's no other way around it.
See these answers:
Python SocketServer
How to exit a multithreaded program?
Basically, don't block on recv() by using select() with a timeout to check for readability of the socket, and poll a quit flag when select() times out.
If I have a thread in an infinite loop, is there a way to terminate it when the main program ends (for example, when I press Ctrl+C)?
If you make your worker threads daemon threads, they will die when all your non-daemon threads (e.g. the main thread) have exited.
http://docs.python.org/library/threading.html#threading.Thread.daemon
Check this question. The correct answer has great explanation on how to terminate threads the right way:
Is there any way to kill a Thread in Python?
To make the thread stop on Keyboard Interrupt signal (ctrl+c) you can catch the exception "KeyboardInterrupt" and cleanup before exiting. Like this:
try:
start_thread()
except (KeyboardInterrupt, SystemExit):
cleanup_stop_thread()
sys.exit()
This way you can control what to do whenever the program is abruptly terminated.
You can also use the built-in signal module that lets you setup signal handlers (in your specific case the SIGINT signal): http://docs.python.org/library/signal.html
Try to enable the sub-thread as daemon-thread.
Recommended:
from threading import Thread
t = Thread(target=desired_method)
t.daemon = True # Dies when main thread (only non-daemon thread) exits.
t.start()
Inline:
t = Thread(target=desired_method, daemon=True).start()
Old API:
t.setDaemon(True)
t.start()
When your main thread terminates (e.g. Ctrl+C keystrokes), other threads will also be killed by the instructions above.
Use the atexit module of Python's standard library to register "termination" functions that get called (on the main thread) on any reasonably "clean" termination of the main thread, including an uncaught exception such as KeyboardInterrupt. Such termination functions may (though inevitably in the main thread!) call any stop function you require; together with the possibility of setting a thread as daemon, that gives you the tools to properly design the system functionality you need.
If you spawn a Thread like so - myThread = Thread(target = function) - and then do myThread.start(); myThread.join(). When CTRL-C is initiated, the main thread doesn't exit because it is waiting on that blocking myThread.join() call. To fix this, simply put in a timeout on the .join() call. The timeout can be as long as you wish. If you want it to wait indefinitely, just put in a really long timeout, like 99999. It's also good practice to do myThread.daemon = True so all the threads exit when the main thread(non-daemon) exits.
Daemon threads are killed ungracefully so any finalizer instructions are not executed.
A possible solution is to check is main thread is alive instead of infinite loop.
E.g. for Python 3:
while threading.main_thread().isAlive():
do.you.subthread.thing()
gracefully.close.the.thread()
See Check if the Main Thread is still alive from another thread.