this thread discusses at great length why it is not a good idea to kill threads. And I agree when we are talking about an actual program.
I am writing unit tests for several components as well as some integration tests between them. Some require threading. When tests fail, some threads stay open, locked in a .get() call of a queue. This causes the whole test suite to get stuck and not complete. I am running either ptw (pytest watch) or a custom loop of pytest with inotifywait to watch for changes in my files to rerun the suite.
When all tests have completed, I want the suite to kill any remaining threads to complete the loop and not be stuck on a thread somewhere that is just open because of a test failure. Is this possible?
There isn't an elegant way to stop a thread, but you can set their daemon to True.
code snippet:
import threading
all_child_threads = [thread for thread in threading.enumerate() if thread != threading.main_thread()]
for thread in all_child_threads:
thread.daemon = True
Then they will terminate when main thread terminates.
Based on Sraw's response, I was able to set the thread to terminate when the main thread terminates, passing daemon=True when starting the thread:
threading.Thread(target=method, daemon=True).start()
So when I run my unit tests, for example, it would end the execution instead of keep running forever since there's still a thread running.
Related
this is done in python 2.7.12
serialHelper is a class module arround python serial and this code does work nicely
#!/usr/bin/env python
import threading
from time import sleep
import serialHelper
sh = serialHelper.SerialHelper()
def serialGetter():
h = 0
while True:
h = h + 1
s_resp = sh.getResponse()
print ('response ' + s_resp)
sleep(3)
if __name__ == '__main__':
try:
t = threading.Thread(target=sh.serialReader)
t.setDaemon(True)
t.start()
serialGetter()
#tSR = threading.Thread(target=serialGetter)
#tSR.setDaemon(True)
#tSR.start()
except Exception as e:
print (e)
however the attemp to run serialGetter as thread as remarked it just dies.
Any reason why that function can not run as thread ?
Quoting from the Python documentation:
The entire Python program exits when no alive non-daemon threads are left.
So if you setDaemon(True) every new thread and then exit the main thread (by falling off the end of the script), the whole program will exit immediately. This kills all of the threads. Either don't use setDaemon(True), or don't exit the main thread without first calling join() on all of the threads you want to wait for.
Stepping back for a moment, it may help to think about the intended use case of a daemon thread. In Unix, a daemon is a process that runs in the background and (typically) serves requests or performs operations, either on behalf of remote clients over the network or local processes. The same basic idea applies to daemon threads:
You launch the daemon thread with some kind of work queue.
When you need some work done on the thread, you hand it a work object.
When you want the result of that work, you use an event or a future to wait for it to complete.
After requesting some work, you always eventually wait for it to complete, or perhaps cancel it (if your worker protocol supports cancellation).
You don't have to clean up the daemon thread at program termination. It just quietly goes away when there are no other threads left.
The problem is step (4). If you forget about some work object, and exit the app without waiting for it to complete, the work may get interrupted. Daemon threads don't gracefully shut down, so you could leave the outside world in an inconsistent state (e.g. an incomplete database transaction, a file that never got closed, etc.). It's often better to use a regular thread, and replace step (5) with an explicit "Finish up your work and shut down" work object that the main thread hands to the worker thread before exiting. The worker thread then recognizes this object, stops waiting on the work queue, and terminates itself once it's no longer doing anything else. This is slightly more up-front work, but is much safer in the event that a work object is inadvertently abandoned.
Because of all of the above, I recommend not using daemon threads unless you have a strong reason for them.
A very simple script.
test.py
import temp
temp.start()
temp.py
import threading, time
f=open("output.txt","w")
def temp():
for i in range(5):
f.write(str(i))
time.sleep(5)
f.close()
def start():
t=threading.Thread(target=temp)
t.setDaemon(True)
t.start()
I expected Daemon thread to complete as main process test.py exits immediately.But the daemon thread exits with the main and does not act like a daemon.Am i missing something basic here?
This is described in some detail in the python documentation at
https://docs.python.org/3/library/threading.html
The most relevant bits are:
A thread can be flagged as a “daemon thread”. The significance of this
flag is that the entire Python program exits when only daemon threads
are left.
and
Daemon threads are abruptly stopped at shutdown. Their resources (such
as open files, database transactions, etc.) may not be released
properly.
The overloading of the term 'daemon' and the negation contortions can make this a bit confusing but it what it boils down to is: A python program exits only after all of its threads complete, except for daemon threads which are simply terminated if no other non-daemon threads are left. In your case, that means the program exits killing your daemon thread before it has a chance to do anything (or, conversely, does not exit until your thread completes, if you setDaemon(false)).
complementing pvg's nice answer, a possible solution for your question is to use join(), in your case:
t.join()
More about join in "what is the use of join() in python threading"
A nice guide explaining in a practical way can be found at: https://realpython.com/intro-to-python-threading/
I have a fairly simple program that each task added into the taskq is executing and computing something, say for 30 seconds. This task is 'not' running in some kind of while or for loop.
def run(self):
while not self.stopper.is_set():
DO_MY_30_SECONDS_WORK(self)
self.task_done()
Now, assuming i have a thread.event and this can check before/after the task is done. But is there a way to tell the already running thread to stop or exit it's execution.
There's no way to stop your running thread if DO_MY_30_SECONDS_WORK(self) is blocking. Well arguably you could set it as daemon thread and it'll be abruptly killed when your main program execution finishes, this would cause problems if the thread is actually holding resources (e.g. writing to a file) and is generally not a good idea to finish a thread.
What you could do is re-design DO_MY_30_SECONDS_WORK(self) and make it non-blocking, which means cutting the work into small pieces and make it check for the stop sign in a reasonable interval, so that your thread will be responsive enough to finish itself when you tell it to do so.
This is a two part question,
After I cancel my script it still continues run, what I'm doing is queering an exchange api and saving the data for various assets.
My parent script can be seen here you can see i'm testing it out with just 3 assets, a sample of one of the child scripts can be seen here.
After I cancel the script the script for BTC seems to still be running and new .json files are still being generated in it's respective folder. The only way to stop it is to delete the folder and create it again.
This is really a bonus, my code was working with two assets but now with the addition of another it seems to only take in data for BTC and not the other 2.
Your first problem is that you are not really creating worker threads.
t1 = Thread(target=BTC.main()) executes BTC.main() and uses its return code to try to start a thread. Since main loops forever, you don't start any other threads.
Once you fix that, you'll still have a problem.
In python, only the root thread sees signals such as ctrl-c. Other threads will continue executing no matter how hard you press the key. When python exits, it tries to join non-daemon threads and that can cause the program to hang. The main thread is waiting for a thread to terminate, but the thread is happily continuing with its execution.
You seem to be depending on this in your code. Your parent starts a bunch of threads (or will, when you fix the first bug) and then exits. Really, its waiting for the threads to exit. If you solve the problem with daemon threads (below), you'll also need to add code for your thread to wait and not exit.
Back to the thread problem...
One solution is to mark threads as "daemon" (do mythread.daemon = True before starting the thread). Python won't wait for those threads and the threads will be killed when the main thread exits. This is great if you don't care about what state the thread is in while terminating. But it can do bad things like leave partially written files laying around.
Another solution is to figure out some way for the main thread to interrupt the thread. Suppose the threads waits of socket traffic. You could close the socket and the thread would be woken by that event.
Another solution is to only run threads for short-lived tasks that you want to complete. Your ctrl-c gets delayed a bit but you eventually exit. You could even set them up to run off of a queue and send a special "kill" message to them when done. In fact, python thread pools are a good way to go.
Another solution is to have the thread check a Event to see if its time to exit.
I'm a little confused about what setting a thread to be a daemon means.
The documentation says this:
A thread can be flagged as a “daemon
thread”. The significance of this flag
is that the entire Python program
exits when only daemon threads are
left. The initial value is inherited
from the creating thread. The flag can
be set through the daemon property.
I'm not sure what makes this different from a normal thread.
Is this saying that this program won't ever finish?
def threadfunc():
while True:
time.sleep(1)
threading.Thread(target=threadfunc).start()
Even though the main thread finishes it's execution. While will finish immediately?
def threadfunc():
while True:
time.sleep(1)
th = threading.Thread(target=threadfunc)
th.daemon = True
th.start()
I ask because I have a situation where in my main thread I'm calling sys.exit(), and the process just hangs and my other threads are running as I can see the log.
Does this have anything to do with sys.exit() being called with threads alive?
Is this saying that this program won't ever finish?
Yes, that program won't finish, just try it out.
I ask because I have a situation where
in my main thread I'm calling
sys.exit(), and the process just hangs
and my other threads are running as I
can see the log. Does this have
anything to do with sys.exit() being
called with threads alive?
Yes, even exit won't stop other threads, it simply raises SystemExit in the main thread. So while the main thread will stop (just like it does on any other unhandled Exception), all other non-daemonic threads will continue to work.
Setting thread.daemon = True will allow the main program to exit. Apps normally wait till all child threads are finished before completing.
th.daemon = True #set this thread as a Daemon Thread
You can think in a Daemon thread as a service this means that it will be running in the background of your computer doing differents task, like indexing files, parsing xml, retrieving news etc, anything that is a long running process.
Your Main thread will finish and your daemon will still be running in the background, that is the reason why your program aka Main thread finish, if you want just put an infinite loop and you will see your thread still running.
An example for a daemon thread is the garbage collection.