I'm a little confused about what setting a thread to be a daemon means.
The documentation says this:
A thread can be flagged as a “daemon
thread”. The significance of this flag
is that the entire Python program
exits when only daemon threads are
left. The initial value is inherited
from the creating thread. The flag can
be set through the daemon property.
I'm not sure what makes this different from a normal thread.
Is this saying that this program won't ever finish?
def threadfunc():
while True:
time.sleep(1)
threading.Thread(target=threadfunc).start()
Even though the main thread finishes it's execution. While will finish immediately?
def threadfunc():
while True:
time.sleep(1)
th = threading.Thread(target=threadfunc)
th.daemon = True
th.start()
I ask because I have a situation where in my main thread I'm calling sys.exit(), and the process just hangs and my other threads are running as I can see the log.
Does this have anything to do with sys.exit() being called with threads alive?
Is this saying that this program won't ever finish?
Yes, that program won't finish, just try it out.
I ask because I have a situation where
in my main thread I'm calling
sys.exit(), and the process just hangs
and my other threads are running as I
can see the log. Does this have
anything to do with sys.exit() being
called with threads alive?
Yes, even exit won't stop other threads, it simply raises SystemExit in the main thread. So while the main thread will stop (just like it does on any other unhandled Exception), all other non-daemonic threads will continue to work.
Setting thread.daemon = True will allow the main program to exit. Apps normally wait till all child threads are finished before completing.
th.daemon = True #set this thread as a Daemon Thread
You can think in a Daemon thread as a service this means that it will be running in the background of your computer doing differents task, like indexing files, parsing xml, retrieving news etc, anything that is a long running process.
Your Main thread will finish and your daemon will still be running in the background, that is the reason why your program aka Main thread finish, if you want just put an infinite loop and you will see your thread still running.
An example for a daemon thread is the garbage collection.
Related
this is done in python 2.7.12
serialHelper is a class module arround python serial and this code does work nicely
#!/usr/bin/env python
import threading
from time import sleep
import serialHelper
sh = serialHelper.SerialHelper()
def serialGetter():
h = 0
while True:
h = h + 1
s_resp = sh.getResponse()
print ('response ' + s_resp)
sleep(3)
if __name__ == '__main__':
try:
t = threading.Thread(target=sh.serialReader)
t.setDaemon(True)
t.start()
serialGetter()
#tSR = threading.Thread(target=serialGetter)
#tSR.setDaemon(True)
#tSR.start()
except Exception as e:
print (e)
however the attemp to run serialGetter as thread as remarked it just dies.
Any reason why that function can not run as thread ?
Quoting from the Python documentation:
The entire Python program exits when no alive non-daemon threads are left.
So if you setDaemon(True) every new thread and then exit the main thread (by falling off the end of the script), the whole program will exit immediately. This kills all of the threads. Either don't use setDaemon(True), or don't exit the main thread without first calling join() on all of the threads you want to wait for.
Stepping back for a moment, it may help to think about the intended use case of a daemon thread. In Unix, a daemon is a process that runs in the background and (typically) serves requests or performs operations, either on behalf of remote clients over the network or local processes. The same basic idea applies to daemon threads:
You launch the daemon thread with some kind of work queue.
When you need some work done on the thread, you hand it a work object.
When you want the result of that work, you use an event or a future to wait for it to complete.
After requesting some work, you always eventually wait for it to complete, or perhaps cancel it (if your worker protocol supports cancellation).
You don't have to clean up the daemon thread at program termination. It just quietly goes away when there are no other threads left.
The problem is step (4). If you forget about some work object, and exit the app without waiting for it to complete, the work may get interrupted. Daemon threads don't gracefully shut down, so you could leave the outside world in an inconsistent state (e.g. an incomplete database transaction, a file that never got closed, etc.). It's often better to use a regular thread, and replace step (5) with an explicit "Finish up your work and shut down" work object that the main thread hands to the worker thread before exiting. The worker thread then recognizes this object, stops waiting on the work queue, and terminates itself once it's no longer doing anything else. This is slightly more up-front work, but is much safer in the event that a work object is inadvertently abandoned.
Because of all of the above, I recommend not using daemon threads unless you have a strong reason for them.
My python script creates alot of threads, they are all daemon threads, I find that I get an error saying "out of memory".
How do I kill a daemon thread whilst my script/application is running?
I understand the concept of daemon threads, that they destroy themselves when my process(script or application) closes/finishes. But I want to kill some of my daemon threads whilst my script is still running to avoid the "out of memory" error.
Will my thread below kill itself when there are no more tasks in the queue?
class ParsePageThread(threading.Thread):
THREAD_NUM = 0
def __init__(self, _queue):
threading.Thread.__init__(self)
self.queue = _queue
def run(self):
while(True):
try:
url = self.queue.get()
except Queue.Empty,e:
return # WILL this kill the thread?
finally:
self.queue.task_done()
I'll answer your second question first because it is easier. Yes, returning from the run method will indeed stop the thread. A detailed explanation is threading: Thread Objects doc.
To stop a thread that is running before it's natural completion you have to get a little more creative. There is no direct kill method on a thread object. What you need to do is use a shared variable to define the state of the thread.
alive = True
class MyThread(threading.Thread):
def run():
while(alive):
#do work here
In some other piece of code, when you detect a condition for stopping that thread, the other thread simply sets alive to False:
alive = False
This is a simple example, I'll leave it to you to scale to multiple threads.
DANGER
This example works because reading and setting a boolean variable are atomic actions in python because of the Global Interpreter Lock. Here is an excellent tutorial for lower level python threading. You should stick to using the Queue object because that's exactly what it's for.
If you do anything more than reading and setting simple variables from multiple threads you should use Locks or alternatively Reentrant Locks depending on your design and needs. Even something as simple as a compare and swap without a lock can cause problems in your program that are very difficult to debug.
Another piece of advice for python multithreading is to never do any significant work in the interpreter thread. It should setup and start all the other threads and then sleep or wait on a condition object until the program exits. The reason for this is no other python thread can receive operating system signals. This means that no other thread can deal with Ctrl+C aka KeyboardInterrupt exceptions. It can be a good practice to have the main thread handle the KeyboardInterrupt exception and then set all the alive variables to False so you can exit your program quickly. This is especially helpful while developing so you don't have to constantly kill things when you make a mistake.
Is there a way to "pause" the main python thread of an application perminantly?
I have some code that fires off two threads
class start():
def __init__(self):
Thread1= functions.threads.Thread1()
Thread1.setDaemon(True)
Thread1.start()
Thread2= functions.threads.Thread2()
Thread2.setDaemon(True)
Thread2.start()
#Stop thread here
At the moment, when the program gets to the end of that function it exits (There is nothing else for the main thread to do after that), killing the threads which run infinately (Looping). How do I stop the main process from exiting? I can do it with a while True: None loop but that uses a lot of CPU and there's probably a better way.
If you don't do setDaemon(True) on the threads, the process will keep running as long as the threads run for.
The daemon flag indicates that the interpreter needn't wait for a thread. It will exit when only daemon threads are left.
Use join:
Thread1.join()
Thread2.join()
Also note that setDaemon is the old API.
Thread1.daemon = True
is the preferred way now.
The whole point of daemon threads is to not prevent the application from being terminated if any of them is still running. You obviously want your threads to keep the application process alive, so don't make them daemons.
If I have a thread in an infinite loop, is there a way to terminate it when the main program ends (for example, when I press Ctrl+C)?
If you make your worker threads daemon threads, they will die when all your non-daemon threads (e.g. the main thread) have exited.
http://docs.python.org/library/threading.html#threading.Thread.daemon
Check this question. The correct answer has great explanation on how to terminate threads the right way:
Is there any way to kill a Thread in Python?
To make the thread stop on Keyboard Interrupt signal (ctrl+c) you can catch the exception "KeyboardInterrupt" and cleanup before exiting. Like this:
try:
start_thread()
except (KeyboardInterrupt, SystemExit):
cleanup_stop_thread()
sys.exit()
This way you can control what to do whenever the program is abruptly terminated.
You can also use the built-in signal module that lets you setup signal handlers (in your specific case the SIGINT signal): http://docs.python.org/library/signal.html
Try to enable the sub-thread as daemon-thread.
Recommended:
from threading import Thread
t = Thread(target=desired_method)
t.daemon = True # Dies when main thread (only non-daemon thread) exits.
t.start()
Inline:
t = Thread(target=desired_method, daemon=True).start()
Old API:
t.setDaemon(True)
t.start()
When your main thread terminates (e.g. Ctrl+C keystrokes), other threads will also be killed by the instructions above.
Use the atexit module of Python's standard library to register "termination" functions that get called (on the main thread) on any reasonably "clean" termination of the main thread, including an uncaught exception such as KeyboardInterrupt. Such termination functions may (though inevitably in the main thread!) call any stop function you require; together with the possibility of setting a thread as daemon, that gives you the tools to properly design the system functionality you need.
If you spawn a Thread like so - myThread = Thread(target = function) - and then do myThread.start(); myThread.join(). When CTRL-C is initiated, the main thread doesn't exit because it is waiting on that blocking myThread.join() call. To fix this, simply put in a timeout on the .join() call. The timeout can be as long as you wish. If you want it to wait indefinitely, just put in a really long timeout, like 99999. It's also good practice to do myThread.daemon = True so all the threads exit when the main thread(non-daemon) exits.
Daemon threads are killed ungracefully so any finalizer instructions are not executed.
A possible solution is to check is main thread is alive instead of infinite loop.
E.g. for Python 3:
while threading.main_thread().isAlive():
do.you.subthread.thing()
gracefully.close.the.thread()
See Check if the Main Thread is still alive from another thread.
What difference makes it when I set a python thread as a daemon, using thread.setDaemon(True)?
A daemon thread will not prevent the application from exiting. The program ends when all non-daemon threads (main thread included) are complete.
So generally, if you're doing something in the background, you might want to set the thread as daemon so you don't have to explicitly have that thread's function return before the app can exit.
For example, if you are writing a GUI application and the user closes the main window, the program should quit. But if you have non-daemon threads hanging around, it won't.
From the docs: http://docs.python.org/library/threading.html#threading.Thread.daemon
Its initial value is inherited from
the creating thread; the main thread
is not a daemon thread and therefore
all threads created in the main thread
default to daemon = False.
The entire Python program exits when
no alive non-daemon threads are left.