Is there a way to "pause" the main python thread of an application perminantly?
I have some code that fires off two threads
class start():
def __init__(self):
Thread1= functions.threads.Thread1()
Thread1.setDaemon(True)
Thread1.start()
Thread2= functions.threads.Thread2()
Thread2.setDaemon(True)
Thread2.start()
#Stop thread here
At the moment, when the program gets to the end of that function it exits (There is nothing else for the main thread to do after that), killing the threads which run infinately (Looping). How do I stop the main process from exiting? I can do it with a while True: None loop but that uses a lot of CPU and there's probably a better way.
If you don't do setDaemon(True) on the threads, the process will keep running as long as the threads run for.
The daemon flag indicates that the interpreter needn't wait for a thread. It will exit when only daemon threads are left.
Use join:
Thread1.join()
Thread2.join()
Also note that setDaemon is the old API.
Thread1.daemon = True
is the preferred way now.
The whole point of daemon threads is to not prevent the application from being terminated if any of them is still running. You obviously want your threads to keep the application process alive, so don't make them daemons.
Related
A very simple script.
test.py
import temp
temp.start()
temp.py
import threading, time
f=open("output.txt","w")
def temp():
for i in range(5):
f.write(str(i))
time.sleep(5)
f.close()
def start():
t=threading.Thread(target=temp)
t.setDaemon(True)
t.start()
I expected Daemon thread to complete as main process test.py exits immediately.But the daemon thread exits with the main and does not act like a daemon.Am i missing something basic here?
This is described in some detail in the python documentation at
https://docs.python.org/3/library/threading.html
The most relevant bits are:
A thread can be flagged as a “daemon thread”. The significance of this
flag is that the entire Python program exits when only daemon threads
are left.
and
Daemon threads are abruptly stopped at shutdown. Their resources (such
as open files, database transactions, etc.) may not be released
properly.
The overloading of the term 'daemon' and the negation contortions can make this a bit confusing but it what it boils down to is: A python program exits only after all of its threads complete, except for daemon threads which are simply terminated if no other non-daemon threads are left. In your case, that means the program exits killing your daemon thread before it has a chance to do anything (or, conversely, does not exit until your thread completes, if you setDaemon(false)).
complementing pvg's nice answer, a possible solution for your question is to use join(), in your case:
t.join()
More about join in "what is the use of join() in python threading"
A nice guide explaining in a practical way can be found at: https://realpython.com/intro-to-python-threading/
just as the sample code shows:
for temp in range(0, 10):
thread = threading.Thread(target = current_post.post)
thread.start()
threads.append(thread)
for current in range(10):
threads[current].join()
the code is just a part of a python file, but it stands for most circumstances: i should execute join() after start() in multithreading. i have been confusing by it for a few days. as we all know, when we execute thread.start(), a new thread then starts, python runs through different threads automatically, thats all we need. if so, why should i add thread.join() after start()? join() means waiting until the current thread finished IMO. but does it means a kind of single-thread? i have to wait each thread to finish their tasks, its not multithreading! join() only means executing the specified function one by one IMO. cannot start() finish the multi-threading perfectly? why should i add join() function to let them finish one by one? thx for any help :)
You do it in order to be sure that your threads have actually finished (and do not become, for example, zombie processes, after your main thread exits).
However, you don't have to do it right after starting the threads. You can do it at the very end of your process.
Join will block the current thread until the thread upon which join is called has finished.
Essentially your code is starting a load of threads and then waiting for them all to complete.
If you didn't then the chances are the process would exit and none of your threads would do anything.
My python script creates alot of threads, they are all daemon threads, I find that I get an error saying "out of memory".
How do I kill a daemon thread whilst my script/application is running?
I understand the concept of daemon threads, that they destroy themselves when my process(script or application) closes/finishes. But I want to kill some of my daemon threads whilst my script is still running to avoid the "out of memory" error.
Will my thread below kill itself when there are no more tasks in the queue?
class ParsePageThread(threading.Thread):
THREAD_NUM = 0
def __init__(self, _queue):
threading.Thread.__init__(self)
self.queue = _queue
def run(self):
while(True):
try:
url = self.queue.get()
except Queue.Empty,e:
return # WILL this kill the thread?
finally:
self.queue.task_done()
I'll answer your second question first because it is easier. Yes, returning from the run method will indeed stop the thread. A detailed explanation is threading: Thread Objects doc.
To stop a thread that is running before it's natural completion you have to get a little more creative. There is no direct kill method on a thread object. What you need to do is use a shared variable to define the state of the thread.
alive = True
class MyThread(threading.Thread):
def run():
while(alive):
#do work here
In some other piece of code, when you detect a condition for stopping that thread, the other thread simply sets alive to False:
alive = False
This is a simple example, I'll leave it to you to scale to multiple threads.
DANGER
This example works because reading and setting a boolean variable are atomic actions in python because of the Global Interpreter Lock. Here is an excellent tutorial for lower level python threading. You should stick to using the Queue object because that's exactly what it's for.
If you do anything more than reading and setting simple variables from multiple threads you should use Locks or alternatively Reentrant Locks depending on your design and needs. Even something as simple as a compare and swap without a lock can cause problems in your program that are very difficult to debug.
Another piece of advice for python multithreading is to never do any significant work in the interpreter thread. It should setup and start all the other threads and then sleep or wait on a condition object until the program exits. The reason for this is no other python thread can receive operating system signals. This means that no other thread can deal with Ctrl+C aka KeyboardInterrupt exceptions. It can be a good practice to have the main thread handle the KeyboardInterrupt exception and then set all the alive variables to False so you can exit your program quickly. This is especially helpful while developing so you don't have to constantly kill things when you make a mistake.
I'm a little confused about what setting a thread to be a daemon means.
The documentation says this:
A thread can be flagged as a “daemon
thread”. The significance of this flag
is that the entire Python program
exits when only daemon threads are
left. The initial value is inherited
from the creating thread. The flag can
be set through the daemon property.
I'm not sure what makes this different from a normal thread.
Is this saying that this program won't ever finish?
def threadfunc():
while True:
time.sleep(1)
threading.Thread(target=threadfunc).start()
Even though the main thread finishes it's execution. While will finish immediately?
def threadfunc():
while True:
time.sleep(1)
th = threading.Thread(target=threadfunc)
th.daemon = True
th.start()
I ask because I have a situation where in my main thread I'm calling sys.exit(), and the process just hangs and my other threads are running as I can see the log.
Does this have anything to do with sys.exit() being called with threads alive?
Is this saying that this program won't ever finish?
Yes, that program won't finish, just try it out.
I ask because I have a situation where
in my main thread I'm calling
sys.exit(), and the process just hangs
and my other threads are running as I
can see the log. Does this have
anything to do with sys.exit() being
called with threads alive?
Yes, even exit won't stop other threads, it simply raises SystemExit in the main thread. So while the main thread will stop (just like it does on any other unhandled Exception), all other non-daemonic threads will continue to work.
Setting thread.daemon = True will allow the main program to exit. Apps normally wait till all child threads are finished before completing.
th.daemon = True #set this thread as a Daemon Thread
You can think in a Daemon thread as a service this means that it will be running in the background of your computer doing differents task, like indexing files, parsing xml, retrieving news etc, anything that is a long running process.
Your Main thread will finish and your daemon will still be running in the background, that is the reason why your program aka Main thread finish, if you want just put an infinite loop and you will see your thread still running.
An example for a daemon thread is the garbage collection.
If I have a thread in an infinite loop, is there a way to terminate it when the main program ends (for example, when I press Ctrl+C)?
If you make your worker threads daemon threads, they will die when all your non-daemon threads (e.g. the main thread) have exited.
http://docs.python.org/library/threading.html#threading.Thread.daemon
Check this question. The correct answer has great explanation on how to terminate threads the right way:
Is there any way to kill a Thread in Python?
To make the thread stop on Keyboard Interrupt signal (ctrl+c) you can catch the exception "KeyboardInterrupt" and cleanup before exiting. Like this:
try:
start_thread()
except (KeyboardInterrupt, SystemExit):
cleanup_stop_thread()
sys.exit()
This way you can control what to do whenever the program is abruptly terminated.
You can also use the built-in signal module that lets you setup signal handlers (in your specific case the SIGINT signal): http://docs.python.org/library/signal.html
Try to enable the sub-thread as daemon-thread.
Recommended:
from threading import Thread
t = Thread(target=desired_method)
t.daemon = True # Dies when main thread (only non-daemon thread) exits.
t.start()
Inline:
t = Thread(target=desired_method, daemon=True).start()
Old API:
t.setDaemon(True)
t.start()
When your main thread terminates (e.g. Ctrl+C keystrokes), other threads will also be killed by the instructions above.
Use the atexit module of Python's standard library to register "termination" functions that get called (on the main thread) on any reasonably "clean" termination of the main thread, including an uncaught exception such as KeyboardInterrupt. Such termination functions may (though inevitably in the main thread!) call any stop function you require; together with the possibility of setting a thread as daemon, that gives you the tools to properly design the system functionality you need.
If you spawn a Thread like so - myThread = Thread(target = function) - and then do myThread.start(); myThread.join(). When CTRL-C is initiated, the main thread doesn't exit because it is waiting on that blocking myThread.join() call. To fix this, simply put in a timeout on the .join() call. The timeout can be as long as you wish. If you want it to wait indefinitely, just put in a really long timeout, like 99999. It's also good practice to do myThread.daemon = True so all the threads exit when the main thread(non-daemon) exits.
Daemon threads are killed ungracefully so any finalizer instructions are not executed.
A possible solution is to check is main thread is alive instead of infinite loop.
E.g. for Python 3:
while threading.main_thread().isAlive():
do.you.subthread.thing()
gracefully.close.the.thread()
See Check if the Main Thread is still alive from another thread.