I would like to kill a thread in python upon certain events.
For example, I have a class within the class there is a neccessary function, for example this:
class exampleClass:
def neccessary_function(self):
try:
do_something()
return
except:
kill_this_thread()
I have multiple threads running simultaneously and I only want to kill that specific thread not all of them.
I can't return the function or anything like that, I need to either stop the thread doing anything or kill it. I currently have in the except section:
while True:
time.sleep(300)
But I feel as though that is not the best way to do it.
It is generally considered unsafe to kill a thread - it could still be holding onto some resources and can lead to deadlocks.
You can look at ways to kill a thread, although as long as you want to be safe, your options are just to "ask the thread kindly to stop when it's ready".
I would suggest you take a look at multiprocessing or asyncio - both APIs provide a rather simple way to cancel an async operation.
Related
Kind all, I'm really new to python and I'm facing a task which I can't completely grasp.
I've created an interface with Tkinter which should accomplish a couple of apparently easy feats.
By clicking a "Start" button two threads/processes will be started (each calling multiple subfunctions) which mainly read data from a serial port (one port per process, of course) and write them to file.
The I/O actions are looped within a while loop with a very high counter to allow them to go onward almost indefinitely.
The "Stop" button should stop the acquisition and essentially it should:
Kill the read/write Thread
Close the file
Close the serial port
Unfortunately I still do not understand how to accomplish point 1, i.e.: how to create killable threads without killing the whole GUI. Is there any way of doing this?
Thank you all!
First, you have to choose whether you are going to use threads or processes.
I will not go too much into differences, google it ;) Anyway, here are some things to consider: it is much easier to establish communication between threads than betweeween processes; in Python, all threads will run on the same CPU core (see Python GIL), but subprocesses may use multiple cores.
Processes
If you are using subprocesses, there are two ways: subprocess.Popen and multiprocessing.Process. With Popen you can run anything, whereas Process gives a simpler thread-like interface to running python code which is part of your project in a subprocess.
Both can be killed using terminate method.
See documentation for multiprocessing and subprocess
Of course, if you want a more graceful exit, you will want to send an "exit" message to the subprocess, rather than just terminate it, so that it gets a chance to do the clean-up. You could do that e.g. by writing to its stdin. The process should read from stdin and when it gets message "exit", it should do whatever you need before exiting.
Threads
For threads, you have to implement your own mechanism for stopping, rather than using something as violent as process.terminate().
Usually, a thread runs in a loop and in that loop you check for a flag which says stop. Then you break from the loop.
I usually have something like this:
class MyThread(Thread):
def __init__(self):
super(Thread, self).__init__()
self._stop_event = threading.Event()
def run(self):
while not self._stop_event.is_set():
# do something
self._stop_event.wait(SLEEP_TIME)
# clean-up before exit
def stop(self, timeout):
self._stop_event.set()
self.join(timeout)
Of course, you need some exception handling etc, but this is the basic idea.
EDIT: Answers to questions in comment
thread.start_new_thread(your_function) starts a new thread, that is correct. On the other hand, module threading gives you a higher-level API which is much nicer.
With threading module, you can do the same with:
t = threading.Thread(target=your_function)
t.start()
or you can make your own class which inherits from Thread and put your functionality in the run method, as in the example above. Then, when user clicks the start button, you do:
t = MyThread()
t.start()
You should store the t variable somewhere. Exactly where depends on how you designed the rest of your application. I would probably have some object which hold all active threads in a list.
When user clicks stop, you should:
t.stop(some_reasonable_time_in_which_the_thread_should_stop)
After that, you can remove the t from your list, it is not usable any more.
First you can use subprocess.Popen() to spawn child processes, then later you can use Popen.terminate() to terminate them.
Note that you could also do everything in a single Python thread, without subprocesses, if you want to. It's perfectly possible to "multiplex" reading from multiple ports in a single event loop.
So I have this library that I use and within one of my functions I call a function from that library, which happens to take a really long time. Now, at the same time I have another thread running where I check for different conditions, what I want is that if a condition is met, I want to cancel the execution of the library function.
Right now I'm checking the conditions at the start of the function, but if the conditions happen to change while the library function is running, I don't need its results, and want to return from it.
Basically this is what I have now.
def my_function():
if condition_checker.condition_met():
return
library.long_running_function()
Is there a way to run the condition check every second or so and return from my_function when the condition is met?
I've thought about decorators, coroutines, I'm using 2.7 but if this can only be done in 3.x I'd consider switching, it's just that I can't figure out how.
You cannot terminate a thread. Either the library supports cancellation by design, where it internally would have to check for a condition every once in a while to abort if requested, or you have to wait for it to finish.
What you can do is call the library in a subprocess rather than a thread, since processes can be terminated through signals. Python's multiprocessing module provides a threading-like API for spawning forks and handling IPC, including synchronization.
Or spawn a separate subprocess via subprocess.Popen if forking is too heavy on your resources (e.g. memory footprint through copying of the parent process).
I can't think of any other way, unfortunately.
Generally, I think you want to run your long_running_function in a separate thread, and have it occasionally report its information to the main thread.
This post gives a similar example within a wxpython program.
Presuming you are doing this outside of wxpython, you should be able to replace the wx.CallAfter and wx.Publisher with threading.Thread and PubSub.
It would look something like this:
import threading
import time
def myfunction():
# subscribe to the long_running_function
while True:
# subscribe to the long_running_function and get the published data
if condition_met:
# publish a stop command
break
time.sleep(1)
def long_running_function():
for loop in loops:
# subscribe to main thread and check for stop command, if so, break
# do an iteration
# publish some data
threading.Thread(group=None, target=long_running_function, args=()) # launches your long_running_function but doesn't block flow
myfunction()
I haven't used pubsub a ton so I can't quickly whip up the code but it should get you there.
As an alternative, do you know the stop criteria before you launch the long_running_function? If so, you can just pass it as an argument and check whether it is met internally.
How can I modify this code (that uses twisted) so that CTRL+C will cause it to exit? I expect the problem is that doWork does not yield control back to the reactor, so the reactor is not able to terminate its execution.
def loop_forever():
i = 0
while True:
yield i
i += 1
time.sleep(5)
def doWork():
for i in loop_forever():
print i
def main():
threads.deferToThread(doWork)
reactor.run()
Note that this code:
def main():
try:
threads.deferToThread(doWork)
reactor.run()
except KeyboardInterrupt:
print "user interrupted task"
does catch the exception on windows, but not on ubuntu
Twisted uses Python's threading library to implement deferToThread. All of the rules that apply to Python threads apply to the threads you get with deferToThread. One rule is that signals and threads are a bad combination (Ctrl-C sends SIGINT on Linux).
The basic idea for solving this problem is to put some logic into doWork so that it will stop. Perhaps this means setting a global flag that it checks once per iteration. You can probably find lots of information elsewhere regarding strategies for getting a long-running thread to cooperate with shutdown.
You may also want to not use deferToThread for this. If you expect your job to run for most of the lifetime of the process then you may just want to use the stdlib threading module directly. Consider that a job like this is using up one of the thread pool slots. If you have enough of these then your thread pool will be full and other work will not be able to proceed.
You may also want to take doWork out of the thread. It doesn't look like it does a lot of blocking. Instead, run doWork in the reactor thread and only run iterations of loop_forever with deferToThread. Now you no longer have a long-running operation in a thread and several of your problems will probably go away.
My python script creates alot of threads, they are all daemon threads, I find that I get an error saying "out of memory".
How do I kill a daemon thread whilst my script/application is running?
I understand the concept of daemon threads, that they destroy themselves when my process(script or application) closes/finishes. But I want to kill some of my daemon threads whilst my script is still running to avoid the "out of memory" error.
Will my thread below kill itself when there are no more tasks in the queue?
class ParsePageThread(threading.Thread):
THREAD_NUM = 0
def __init__(self, _queue):
threading.Thread.__init__(self)
self.queue = _queue
def run(self):
while(True):
try:
url = self.queue.get()
except Queue.Empty,e:
return # WILL this kill the thread?
finally:
self.queue.task_done()
I'll answer your second question first because it is easier. Yes, returning from the run method will indeed stop the thread. A detailed explanation is threading: Thread Objects doc.
To stop a thread that is running before it's natural completion you have to get a little more creative. There is no direct kill method on a thread object. What you need to do is use a shared variable to define the state of the thread.
alive = True
class MyThread(threading.Thread):
def run():
while(alive):
#do work here
In some other piece of code, when you detect a condition for stopping that thread, the other thread simply sets alive to False:
alive = False
This is a simple example, I'll leave it to you to scale to multiple threads.
DANGER
This example works because reading and setting a boolean variable are atomic actions in python because of the Global Interpreter Lock. Here is an excellent tutorial for lower level python threading. You should stick to using the Queue object because that's exactly what it's for.
If you do anything more than reading and setting simple variables from multiple threads you should use Locks or alternatively Reentrant Locks depending on your design and needs. Even something as simple as a compare and swap without a lock can cause problems in your program that are very difficult to debug.
Another piece of advice for python multithreading is to never do any significant work in the interpreter thread. It should setup and start all the other threads and then sleep or wait on a condition object until the program exits. The reason for this is no other python thread can receive operating system signals. This means that no other thread can deal with Ctrl+C aka KeyboardInterrupt exceptions. It can be a good practice to have the main thread handle the KeyboardInterrupt exception and then set all the alive variables to False so you can exit your program quickly. This is especially helpful while developing so you don't have to constantly kill things when you make a mistake.
I am trying to write a unix client program that is listening to a socket, stdin, and reading from file descriptors. I assign each of these tasks to an individual thread and have them successfully communicating with the "main" application using synchronized queues and a semaphore. The problem is that when I want to shutdown these child threads they are all blocking on input. Also, the threads cannot register signal handlers in the threads because in Python only the main thread of execution is allowed to do so.
Any suggestions?
There is no good way to work around this, especially when the thread is blocking.
I had a similar issue ( Python: How to terminate a blocking thread) and the only way I was able to stop my threads was to close the underlying connection. Which resulted in the thread that was blocking to raise and exception and then allowed me to check the stop flag and close.
Example code:
class Example(object):
def __init__(self):
self.stop = threading.Event()
self.connection = Connection()
self.mythread = Thread(target=self.dowork)
self.mythread.start()
def dowork(self):
while(not self.stop.is_set()):
try:
blockingcall()
except CommunicationException:
pass
def terminate():
self.stop.set()
self.connection.close()
self.mythread.join()
Another thing to note is commonly blocking operations generally offer up a timeout. If you have that option I would consider using it. My last comment is that you could always set the thread to deamonic,
From the pydoc :
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.
Also, the threads cannot register signal handlers
Signals to kill threads is potentially horrible, especially in C, especially if you allocate memory as part of the thread, since it won't be freed when that particular thread dies (as it belongs to the heap of the process). There is no garbage collection in C, so if that pointer goes out of scope, it's gone out of scope, the memory remains allocated. So just be careful with that one - only do it that way in C if you're going to actually kill all the threads and end the process so that the memory is handed back to the OS - adding and removing threads from a threadpool for example will give you a memory leak.
The problem is that when I want to shutdown these child threads they are all blocking on input.
Funnily enough I've been fighting with the same thing recently. The solution is literally don't make blocking calls without a timeout. So, for example, what you want ideally is:
def threadfunc(running):
while running:
blockingcall(timeout=1)
where running is passed from the controlling thread - I've never used threading but I have used multiprocessing and with this you actually need to pass an Event() object and check is_set(). But you asked for design patterns, that's the basic idea.
Then, when you want this thread to end, you run:
running.clear()
mythread.join()
and your main thread should then allow your client thread to handle its last call, and return, and the whole program folds up nicely.
What do you do if you have a blocking call without a timeout? Use the asynchronous option, and sleep (as in call whatever method you have to suspend the thread for a period of time so you're not spinning) if you need to. There's no other way around it.
See these answers:
Python SocketServer
How to exit a multithreaded program?
Basically, don't block on recv() by using select() with a timeout to check for readability of the socket, and poll a quit flag when select() times out.