Develop Reference

Python thread or process model where child thread or process can survive parent?

This is a design question in reference to python scripting in using threads versus multi-processes. As I understand it, spawning a thread using the threading module cannot survive termination of the the parent thread, i.e. process. The parent thread must either do a join (i.e. wait timeout not withstanding) or exit, if no join, on parent exit the child threads are terminated. This is due to the shared resources model of threads, right?
Whereas the multiprocessing module when a process is spawned it can survive, i.e. continue to completion, regardless if the parent process which created it exits or terminates. This assumes of course that the parent process never called a join for the child process to complete.

Both, threading and multiprocessing are designed to achieve parallelism within a program. Their goal is not to launch independent processes. Hence both packages implicitly terminate their parallel execution paths during preparation for interpreter shutdown.
Threads are subsets of processes, they cannot outlive the process that created them.
Active non daemonic threads are implicitly joined upon interpreter shutdown using the function _shutdown() in the threading module. This function is called during the finalization routine in the Python interpreter lifecycle.Daemonic threads simply end with the interpreter process.
If processes, created via multiprocessing, are still alive when the interpreter prepares to shut down, they are terminated by the _exit_function(), that has been registered as exit handler via atexit. Similar to threading, multiprocessing joins non daemonic child processes; on daemonic childs, terminate() is called.
If you want to launch processes from a Python program and have that program exit afterwards, use subprocess.Popen. If you are on a POSIX platform, you might also want to take a look at python-daemon.

Is the Python non-daemon thread a non-detached thread? When is its resource freed?

Will the resources of non-daemon thread get released back to OS once the threads completes? ie If the main thread is not calling join() on these non-daemon threads, will the python GC call join on them and release the resources once held by the thread?

If you spawn a thread that runs a function, and then that function completes before the end of the program, then yes, the thread will get garbage collected once it is (a) no longer running, and (b) no longer referenced by anything else.

"
A thread can be flagged as a “daemon thread”. The significance of this
flag is that the entire Python program exits when only daemon threads
are left. The initial value is inherited from the creating thread. The
flag can be set through the daemon property.
Note Daemon threads are abruptly stopped at shutdown. Their resources
(such as open files, database transactions, etc.) may not be released
properly. If you want your threads to stop gracefully, make them
non-daemonic and use a suitable signalling mechanism such as an Event.
" -- Python Thread Docs
Daemons are cleaned up by Python, Non-daemonic threads are not - you have to signal them to stop. This is useful in executing some complicated parallel code though :)
This does mean though you can have random dummy / useless threads sitting around for you to manually cleanup if you use non-daemonic threads.
tl;dr Non-daemonic threads never 'finish' you have to signal them to finish via your own mechanism or one of the SIGS e.g. SIGTERM.

How can I kill a thread in python [duplicate]

This question already has answers here:
Is there any way to kill a Thread?
(31 answers)
Closed 6 years ago.
I start a thread using the following code.
t = thread.start_new_thread(myfunction)
How can I kill the thread t from another thread. So basically speaking in terms of code, I want to be able to do something like this.
t.kill()
Note that I'm using Python 2.4.

In Python, you simply cannot kill a Thread.
If you do NOT really need to have a Thread (!), what you can do, instead of using the threading package (http://docs.python.org/2/library/threading.html), is to use the multiprocessing package (http://docs.python.org/2/library/multiprocessing.html). Here, to kill a process, you can simply call the method:
yourProcess.terminate() # kill the process!
Python will kill your process (on Unix through the SIGTERM signal, while on Windows through the TerminateProcess() call). Pay attention to use it while using a Queue or a Pipe! (it may corrupt the data in the Queue/Pipe)
Note that the multiprocessing.Event and the multiprocessing.Semaphore work exactly in the same way of the threading.Event and the threading.Semaphore respectively. In fact, the first ones are clones of the latters.
If you REALLY need to use a Thread, there is no way to kill your threads directly. What you can do, however, is to use a "daemon thread". In fact, in Python, a Thread can be flagged as daemon:
yourThread.daemon = True # set the Thread as a "daemon thread"
The main program will exit when no alive non-daemon threads are left. In other words, when your main thread (which is, of course, a non-daemon thread) will finish its operations, the program will exit even if there are still some daemon threads working.
Note that it is necessary to set a Thread as daemon before the start() method is called!
Of course you can, and should, use daemon even with multiprocessing. Here, when the main process exits, it attempts to terminate all of its daemonic child processes.
Finally, please, note that sys.exit() and os.kill() are not choices.

If your thread is busy executing Python code, you have a bigger problem than the inability to kill it. The GIL will prevent any other thread from even running whatever instructions you would use to do the killing. (After a bit of research, I've learned that the interpreter periodically releases the GIL, so the preceding statement is bogus. The remaining comment stands, however.)
Your thread must be written in a cooperative manner. That is, it must periodically check in with a signalling object such as a semaphore, which the main thread can use to instruct the worker thread to voluntarily exit.
while not sema.acquire(False):
# Do a small portion of work…
or:
for item in work:
# Keep working…
# Somewhere deep in the bowels…
if sema.acquire(False):
thread.exit()

You can't kill a thread from another thread. You need to signal to the other thread that it should end. And by "signal" I don't mean use the signal function, I mean that you have to arrange for some communication between the threads.

Gracefully Terminating Python Threads

I am trying to write a unix client program that is listening to a socket, stdin, and reading from file descriptors. I assign each of these tasks to an individual thread and have them successfully communicating with the "main" application using synchronized queues and a semaphore. The problem is that when I want to shutdown these child threads they are all blocking on input. Also, the threads cannot register signal handlers in the threads because in Python only the main thread of execution is allowed to do so.
Any suggestions?

There is no good way to work around this, especially when the thread is blocking.
I had a similar issue ( Python: How to terminate a blocking thread) and the only way I was able to stop my threads was to close the underlying connection. Which resulted in the thread that was blocking to raise and exception and then allowed me to check the stop flag and close.
Example code:
class Example(object):
def __init__(self):
self.stop = threading.Event()
self.connection = Connection()
self.mythread = Thread(target=self.dowork)
self.mythread.start()
def dowork(self):
while(not self.stop.is_set()):
try:
blockingcall()
except CommunicationException:
pass
def terminate():
self.stop.set()
self.connection.close()
self.mythread.join()
Another thing to note is commonly blocking operations generally offer up a timeout. If you have that option I would consider using it. My last comment is that you could always set the thread to deamonic,
From the pydoc :
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.

Also, the threads cannot register signal handlers
Signals to kill threads is potentially horrible, especially in C, especially if you allocate memory as part of the thread, since it won't be freed when that particular thread dies (as it belongs to the heap of the process). There is no garbage collection in C, so if that pointer goes out of scope, it's gone out of scope, the memory remains allocated. So just be careful with that one - only do it that way in C if you're going to actually kill all the threads and end the process so that the memory is handed back to the OS - adding and removing threads from a threadpool for example will give you a memory leak.
The problem is that when I want to shutdown these child threads they are all blocking on input.
Funnily enough I've been fighting with the same thing recently. The solution is literally don't make blocking calls without a timeout. So, for example, what you want ideally is:
def threadfunc(running):
while running:
blockingcall(timeout=1)
where running is passed from the controlling thread - I've never used threading but I have used multiprocessing and with this you actually need to pass an Event() object and check is_set(). But you asked for design patterns, that's the basic idea.
Then, when you want this thread to end, you run:
running.clear()
mythread.join()
and your main thread should then allow your client thread to handle its last call, and return, and the whole program folds up nicely.
What do you do if you have a blocking call without a timeout? Use the asynchronous option, and sleep (as in call whatever method you have to suspend the thread for a period of time so you're not spinning) if you need to. There's no other way around it.

See these answers:
Python SocketServer
How to exit a multithreaded program?
Basically, don't block on recv() by using select() with a timeout to check for readability of the socket, and poll a quit flag when select() times out.

What difference it makes when I set python thread as a Daemon

What difference makes it when I set a python thread as a daemon, using thread.setDaemon(True)?

A daemon thread will not prevent the application from exiting. The program ends when all non-daemon threads (main thread included) are complete.
So generally, if you're doing something in the background, you might want to set the thread as daemon so you don't have to explicitly have that thread's function return before the app can exit.
For example, if you are writing a GUI application and the user closes the main window, the program should quit. But if you have non-daemon threads hanging around, it won't.
From the docs: http://docs.python.org/library/threading.html#threading.Thread.daemon
Its initial value is inherited from
the creating thread; the main thread
is not a daemon thread and therefore
all threads created in the main thread
default to daemon = False.
The entire Python program exits when
no alive non-daemon threads are left.