This question already has answers here:
Add timeout argument to python's Queue.join()
(4 answers)
Closed 7 years ago.
In python, I have multiple threads running and I need the main process to wait until they are done, so I did this with Queue class .join() method. However, I wanted to implement SIGINT but the handler for it wouldn't execute because join() was blocking it(the threading processes run for at least 5 minutes for what I have them doing). So I modified Queue's .join() and placed a time out in wait():
class CustomQueue(Queue.Queue):
#Can not use .join() because it would block any processing
#for SIGINT untill threads are done. To counter this,
# wait() is given a time out along with while not kill_received
#to be checked
def join(self):
self.all_tasks_done.acquire()
try:
while not kill_received and self.unfinished_tasks:
self.all_tasks_done.wait(10.0)
finally:
self.all_tasks_done.release()
This works beautifully and perfect for me.
But what I don't understand is the time out in wait(). For instance, I should not be able to send a SIGINT and have it process for at least 10 seconds. But, I am able to in less than 10 seconds. It doesn't matter what the seconds are, the SIGINT handler function is able to process without being blocked. Why is this? I should have to wait at least 10 seconds for the wait to time out and self.all_tasks_done.release() to run so the SIGINT function will process...rather than the SIGINT function processing before the wait() time out.
We're missing information here that may be important:
Which version of Python?
Which OS?
It may be important because mixing threads with signals is a cross-platform mess. CPython tries to make some sense of it all, but has had various degrees of success across various Python versions and OSes.
Anyway, the answer to your question may be simple ;-) Queue.all_tasks_done is a threading.Condition, and Condition implements .wait(timeout) (eventually, drilling down) using time.sleep(). As documented,
time.sleep(secs)
Suspend execution for the given number of seconds. ... The actual
suspension time may be less than that requested because any caught
signal will terminate the sleep() following execution of that
signal’s catching routine. ...
By default SIGINT raises KeyboardInterrupt. So if you don't have a handler installed for SIGINT, SIGINT terminates the sleep() early and raises KeyboardInterrupt. If you do have a SIGINT handler installed, SIGINT will still terminate the sleep() early, but what happens after that depends on what your handler does.
Related
This question already has answers here:
python multithreading wait till all threads finished
(9 answers)
Closed 7 years ago.
I need my main program, at some point, to do nothing - forever (there are threads started earlier which do some work).
I was wondering what was the pythonic way to do so. I particularly would like to avoid losing CPU cycles on this nothingness.
My immediate guess was time.sleep(100000) where 100000 is large enough for the life of my program. It works but is not aesthetically pleasant. I checked the CPU load of a python session running this: not measurable (close to zero).
I also tried
while True:
pass
It looks nicer but the hit on the CPU is enormous (~25%).
Is there one-- and preferably only one --obvious way to do it?
If you would like to wait for the thread to exit, use thread.join():
join([timeout])
Wait until the thread terminates. This blocks the
calling thread until the thread whose join() method is called
terminates – either normally or through an unhandled exception – or
until the optional timeout occurs.
Example usage:
import threading
t = threading.Thread(name='non-daemon', target=non_daemon)
t.start() # This will not block
t.join() # This is blocking
This question already has answers here:
Is there any way to kill a Thread?
(31 answers)
Closed 6 years ago.
I start a thread using the following code.
t = thread.start_new_thread(myfunction)
How can I kill the thread t from another thread. So basically speaking in terms of code, I want to be able to do something like this.
t.kill()
Note that I'm using Python 2.4.
In Python, you simply cannot kill a Thread.
If you do NOT really need to have a Thread (!), what you can do, instead of using the threading package (http://docs.python.org/2/library/threading.html), is to use the multiprocessing package (http://docs.python.org/2/library/multiprocessing.html). Here, to kill a process, you can simply call the method:
yourProcess.terminate() # kill the process!
Python will kill your process (on Unix through the SIGTERM signal, while on Windows through the TerminateProcess() call). Pay attention to use it while using a Queue or a Pipe! (it may corrupt the data in the Queue/Pipe)
Note that the multiprocessing.Event and the multiprocessing.Semaphore work exactly in the same way of the threading.Event and the threading.Semaphore respectively. In fact, the first ones are clones of the latters.
If you REALLY need to use a Thread, there is no way to kill your threads directly. What you can do, however, is to use a "daemon thread". In fact, in Python, a Thread can be flagged as daemon:
yourThread.daemon = True # set the Thread as a "daemon thread"
The main program will exit when no alive non-daemon threads are left. In other words, when your main thread (which is, of course, a non-daemon thread) will finish its operations, the program will exit even if there are still some daemon threads working.
Note that it is necessary to set a Thread as daemon before the start() method is called!
Of course you can, and should, use daemon even with multiprocessing. Here, when the main process exits, it attempts to terminate all of its daemonic child processes.
Finally, please, note that sys.exit() and os.kill() are not choices.
If your thread is busy executing Python code, you have a bigger problem than the inability to kill it. The GIL will prevent any other thread from even running whatever instructions you would use to do the killing. (After a bit of research, I've learned that the interpreter periodically releases the GIL, so the preceding statement is bogus. The remaining comment stands, however.)
Your thread must be written in a cooperative manner. That is, it must periodically check in with a signalling object such as a semaphore, which the main thread can use to instruct the worker thread to voluntarily exit.
while not sema.acquire(False):
# Do a small portion of work…
or:
for item in work:
# Keep working…
# Somewhere deep in the bowels…
if sema.acquire(False):
thread.exit()
You can't kill a thread from another thread. You need to signal to the other thread that it should end. And by "signal" I don't mean use the signal function, I mean that you have to arrange for some communication between the threads.
In *nix python signaling allows me to stop sleep before it is ready. Is there any similar mechanism available in windows -- seems that all the methods end up intercepting code only after the sleep?
Code example:
from time import sleep
.. do something that will intercept the sleep
try:
sleep(60)
finally:
print 'how to get here under 60 seconds in windows?'
Similar question that doesn't have an answer for windows:
break/interrupt a time.sleep() in python
The Python documentation for signal says:
• Although Python signal handlers are called asynchronously as far as
the Python user is concerned, they can only occur between the “atomic”
instructions of the Python interpreter. This means that signals
arriving during long calculations implemented purely in C (such as
regular expression matches on large bodies of text) may be delayed for
an arbitrary amount of time.
whereas time says:
The actual suspension time may be less than that requested because any
caught signal will terminate the sleep() following execution of that
signal’s catching routine.
On Windows, apparently time.sleep() is not implemented in accordance with the documentation, because the handler for a signal received during sleep is not acted upon until the full duration of the sleep is finished. The following example:
import signal, time
def handler(signum, frame):
print('Signal handler called with signal', signum)
signal.signal(signal.SIGINT, handler)
print("signal set, sleeping")
time.sleep(10)
print("sleep done")
prints:
signal set, sleeping
Signal handler called with signal 2
sleep done
with the first line occurring immediately and the second two after 10 seconds, regardless of when the interrupt occurs.
As Thomas K suggests, a better strategy would be to use threads and synchronization.
I'm not sure if this will do it for you, but here's a little workaround you could do:
for i in range(60):
if checkIfIShouldInterruptSleep():
break
time.sleep(1)
print "Finished Sleeping!"
It's only accurate to the second, but it might serve your purpose. Happy coding!
I am trying to write a unix client program that is listening to a socket, stdin, and reading from file descriptors. I assign each of these tasks to an individual thread and have them successfully communicating with the "main" application using synchronized queues and a semaphore. The problem is that when I want to shutdown these child threads they are all blocking on input. Also, the threads cannot register signal handlers in the threads because in Python only the main thread of execution is allowed to do so.
Any suggestions?
There is no good way to work around this, especially when the thread is blocking.
I had a similar issue ( Python: How to terminate a blocking thread) and the only way I was able to stop my threads was to close the underlying connection. Which resulted in the thread that was blocking to raise and exception and then allowed me to check the stop flag and close.
Example code:
class Example(object):
def __init__(self):
self.stop = threading.Event()
self.connection = Connection()
self.mythread = Thread(target=self.dowork)
self.mythread.start()
def dowork(self):
while(not self.stop.is_set()):
try:
blockingcall()
except CommunicationException:
pass
def terminate():
self.stop.set()
self.connection.close()
self.mythread.join()
Another thing to note is commonly blocking operations generally offer up a timeout. If you have that option I would consider using it. My last comment is that you could always set the thread to deamonic,
From the pydoc :
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.
Also, the threads cannot register signal handlers
Signals to kill threads is potentially horrible, especially in C, especially if you allocate memory as part of the thread, since it won't be freed when that particular thread dies (as it belongs to the heap of the process). There is no garbage collection in C, so if that pointer goes out of scope, it's gone out of scope, the memory remains allocated. So just be careful with that one - only do it that way in C if you're going to actually kill all the threads and end the process so that the memory is handed back to the OS - adding and removing threads from a threadpool for example will give you a memory leak.
The problem is that when I want to shutdown these child threads they are all blocking on input.
Funnily enough I've been fighting with the same thing recently. The solution is literally don't make blocking calls without a timeout. So, for example, what you want ideally is:
def threadfunc(running):
while running:
blockingcall(timeout=1)
where running is passed from the controlling thread - I've never used threading but I have used multiprocessing and with this you actually need to pass an Event() object and check is_set(). But you asked for design patterns, that's the basic idea.
Then, when you want this thread to end, you run:
running.clear()
mythread.join()
and your main thread should then allow your client thread to handle its last call, and return, and the whole program folds up nicely.
What do you do if you have a blocking call without a timeout? Use the asynchronous option, and sleep (as in call whatever method you have to suspend the thread for a period of time so you're not spinning) if you need to. There's no other way around it.
See these answers:
Python SocketServer
How to exit a multithreaded program?
Basically, don't block on recv() by using select() with a timeout to check for readability of the socket, and poll a quit flag when select() times out.
I have been looking all over the place for a good timeout script that can kill a thread if it's been active for more than X seconds, but all the examples I've seen have flaws that don't always stop the thread. Using thread.join(x) ends up defeating the purpose of it being a thread.
The only decent example I have found is Timeout on a function call and that one is not without its flaws..
Anyone know of a better way to do this?
See my answer to python: how to send packets in multi thread and then the thread kill itself - there is a fragment with InterruptableThread class and example that kill another thread after timeout - exactly what you want.
There is also similar Python recipe at activestate.
I know this might not be what you want, but have you considered the signal approach?
Timeout on a function call
http://docs.python.org/library/signal.html#example
You can set an alarm signal at the beginning of the thread execution, and then stop the thread in the signal handler.