This question already has answers here:
python multithreading wait till all threads finished
(9 answers)
Closed 7 years ago.
I need my main program, at some point, to do nothing - forever (there are threads started earlier which do some work).
I was wondering what was the pythonic way to do so. I particularly would like to avoid losing CPU cycles on this nothingness.
My immediate guess was time.sleep(100000) where 100000 is large enough for the life of my program. It works but is not aesthetically pleasant. I checked the CPU load of a python session running this: not measurable (close to zero).
I also tried
while True:
pass
It looks nicer but the hit on the CPU is enormous (~25%).
Is there one-- and preferably only one --obvious way to do it?
If you would like to wait for the thread to exit, use thread.join():
join([timeout])
Wait until the thread terminates. This blocks the
calling thread until the thread whose join() method is called
terminates – either normally or through an unhandled exception – or
until the optional timeout occurs.
Example usage:
import threading
t = threading.Thread(name='non-daemon', target=non_daemon)
t.start() # This will not block
t.join() # This is blocking
Related
This question already has answers here:
Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
(2 answers)
Closed 2 years ago.
I am using Python's ProcessPoolExecutor to run multiple processes in parallel and process them as any of them finishes. Then I look at their output and as soon as at least one of them gives satisfying answer I want to exit the program.
However, this is not possible since upon calling pool.shutdown(wait=False) I will have to wait for all active tasks in the pool to finish before I can exit my script.
Is there a way to kill all the remaining active children and exit?
Also, is there a better way to stop as soon as at least one child returns the answer we are waiting for?
What you're doing is not quite clear, but executors strike me as the wrong tool entirely.
multiprocessing.Pool seems much more suitable, and allows doing exactly what you're asking for: you can iterate on imap_unordered (or apply_async and poll the results), and once you have what you were looking for just terminate() the pool then join() it.
This question already has answers here:
Is there any way to kill a Thread?
(31 answers)
Closed 8 years ago.
I want only one Thread to be active, here is the algorithm:
def myfunction(update):
if threading.activeCount >=1:
# kill all previous threads
while true:
# some execution
while true:
t = threading.Thread(myfunction, args=([update]))
t.start()
so In here thread goes in Infinite loop in myfunction, so before starting new one i need to close previous one, please guide me
You can't kill threads, your function has to return somehow. The trick, then, is to write your function in such a way that it knows when to stop what its doing and return. That's very context dependent of course. Its common to implement your code as a class that has a 'close' method that knows what to do. For instance,
if the thread is waiting on a subprocess, the close could kill the subprocess.
If the thread does a sleep, you could use an event instead - the close would set the event.
If your thread is waiting on a queue, you could have a special method that tells it to return.
This question already has answers here:
Add timeout argument to python's Queue.join()
(4 answers)
Closed 7 years ago.
In python, I have multiple threads running and I need the main process to wait until they are done, so I did this with Queue class .join() method. However, I wanted to implement SIGINT but the handler for it wouldn't execute because join() was blocking it(the threading processes run for at least 5 minutes for what I have them doing). So I modified Queue's .join() and placed a time out in wait():
class CustomQueue(Queue.Queue):
#Can not use .join() because it would block any processing
#for SIGINT untill threads are done. To counter this,
# wait() is given a time out along with while not kill_received
#to be checked
def join(self):
self.all_tasks_done.acquire()
try:
while not kill_received and self.unfinished_tasks:
self.all_tasks_done.wait(10.0)
finally:
self.all_tasks_done.release()
This works beautifully and perfect for me.
But what I don't understand is the time out in wait(). For instance, I should not be able to send a SIGINT and have it process for at least 10 seconds. But, I am able to in less than 10 seconds. It doesn't matter what the seconds are, the SIGINT handler function is able to process without being blocked. Why is this? I should have to wait at least 10 seconds for the wait to time out and self.all_tasks_done.release() to run so the SIGINT function will process...rather than the SIGINT function processing before the wait() time out.
We're missing information here that may be important:
Which version of Python?
Which OS?
It may be important because mixing threads with signals is a cross-platform mess. CPython tries to make some sense of it all, but has had various degrees of success across various Python versions and OSes.
Anyway, the answer to your question may be simple ;-) Queue.all_tasks_done is a threading.Condition, and Condition implements .wait(timeout) (eventually, drilling down) using time.sleep(). As documented,
time.sleep(secs)
Suspend execution for the given number of seconds. ... The actual
suspension time may be less than that requested because any caught
signal will terminate the sleep() following execution of that
signal’s catching routine. ...
By default SIGINT raises KeyboardInterrupt. So if you don't have a handler installed for SIGINT, SIGINT terminates the sleep() early and raises KeyboardInterrupt. If you do have a SIGINT handler installed, SIGINT will still terminate the sleep() early, but what happens after that depends on what your handler does.
This question already has answers here:
How do you create a daemon in Python?
(16 answers)
Closed 9 years ago.
I am new with Daemons and I was wondering how can I make my main script a daemon?
I have my main script which I wish to make a Daemon and run in the background:
main.py
def requestData(information):
return currently_crunched_data()
while True:
crunchData()
I would like to be able to use the requestData function to this daemon while the loop is running. I am not too familiar with Daemons or how to convert my script into one.
However I am guessing I would have to make two threads, one for my cruncData loop and one for the Daemon request receiever since the Daemon has its own loop (daemon.requestLoop()).
I am currently looking into Pyro to do this. Does anyone know how I can ultimately make a background running while loop have the ability to receive requests from other processes (like a Daemon I suppose) ?
There are already a number of questions on creating a daemon in Python, like this one, which answer that part nicely.
So, how do you have your daemon do background work?
As you suspected, threads are an obvious answer. But there are three possible complexities.
First, there's shutdown. If you're lucky, your crunchData function can be summarily killed at any time with no corrupted data or (too-significant) lost work. In that case:
def worker():
while True:
crunchData()
# ... somewhere in the daemon startup code ...
t = threading.Thread(target=worker)
t.daemon = True
t.start()
Notice that t.daemon. A "daemon thread" has nothing to do with your program being a daemon; it means that you can just quit the main process, and it will be summarily killed.
But what if crunchData can't be killed? Then you'll need to do something like this:
quitflag = False
quitlock = threading.Lock()
def worker():
while True:
with quitlock:
if quitflag:
return
crunchData()
# ... somewhere in the daemon startup code ...
t = threading.Thread(target=worker)
t.start()
# ... somewhere in the daemon shutdown code ...
with quitlock:
quitflag = True
t.join()
I'm assuming each iteration of crunchData doesn't take that long. If it does, you may need to check quitFlag periodically within the function itself.
Meanwhile, you want your request handler to access some data that the background thread is producing. You'll need some kind of synchronization there as well.
The obvious thing is to just use another Lock. But there's a good chance that crunchData is writing to its data frequently. If it holds the lock for 10 seconds at a time, the request handler may block for 10 seconds. But if it grabs and releases the lock a million times, that could take longer than the actual work.
One alternative is to double-buffer your data: Have crunchData write into a new copy, then, when it's done, briefly grab the lock and set currentData = newData.
Depending on your use case, a Queue, a file, or something else might be even simpler.
Finally, crunchData is presumably doing a lot of CPU work. You need to make sure that the request handler does very little CPU work, or each request will slow things down quite a bit as the two threads fight over the GIL. Usually this is no problem. If it is, use a multiprocessing.Process instead of a Thread (which makes sharing or passing the data between the two processes slightly more complicated, but still not too bad).
This question already has answers here:
Is there any way to kill a Thread?
(31 answers)
Closed 6 years ago.
I start a thread using the following code.
t = thread.start_new_thread(myfunction)
How can I kill the thread t from another thread. So basically speaking in terms of code, I want to be able to do something like this.
t.kill()
Note that I'm using Python 2.4.
In Python, you simply cannot kill a Thread.
If you do NOT really need to have a Thread (!), what you can do, instead of using the threading package (http://docs.python.org/2/library/threading.html), is to use the multiprocessing package (http://docs.python.org/2/library/multiprocessing.html). Here, to kill a process, you can simply call the method:
yourProcess.terminate() # kill the process!
Python will kill your process (on Unix through the SIGTERM signal, while on Windows through the TerminateProcess() call). Pay attention to use it while using a Queue or a Pipe! (it may corrupt the data in the Queue/Pipe)
Note that the multiprocessing.Event and the multiprocessing.Semaphore work exactly in the same way of the threading.Event and the threading.Semaphore respectively. In fact, the first ones are clones of the latters.
If you REALLY need to use a Thread, there is no way to kill your threads directly. What you can do, however, is to use a "daemon thread". In fact, in Python, a Thread can be flagged as daemon:
yourThread.daemon = True # set the Thread as a "daemon thread"
The main program will exit when no alive non-daemon threads are left. In other words, when your main thread (which is, of course, a non-daemon thread) will finish its operations, the program will exit even if there are still some daemon threads working.
Note that it is necessary to set a Thread as daemon before the start() method is called!
Of course you can, and should, use daemon even with multiprocessing. Here, when the main process exits, it attempts to terminate all of its daemonic child processes.
Finally, please, note that sys.exit() and os.kill() are not choices.
If your thread is busy executing Python code, you have a bigger problem than the inability to kill it. The GIL will prevent any other thread from even running whatever instructions you would use to do the killing. (After a bit of research, I've learned that the interpreter periodically releases the GIL, so the preceding statement is bogus. The remaining comment stands, however.)
Your thread must be written in a cooperative manner. That is, it must periodically check in with a signalling object such as a semaphore, which the main thread can use to instruct the worker thread to voluntarily exit.
while not sema.acquire(False):
# Do a small portion of work…
or:
for item in work:
# Keep working…
# Somewhere deep in the bowels…
if sema.acquire(False):
thread.exit()
You can't kill a thread from another thread. You need to signal to the other thread that it should end. And by "signal" I don't mean use the signal function, I mean that you have to arrange for some communication between the threads.