I have a code where in I have two types of threads. 3 threads are spawned from the second. I wanted to know if there is a function which I can call, which will terminate the three spawned threads of the second type but still keeping the first one running.
A common solution is to have a global variable that the threads check if they should terminate or not.
Edit: An example of one way of doing it:
class MyThread(Thread):
def __init__(self):
self.keep_running = True
def run(self):
while self.keep_running:
# Do stuff
my_thread = MyThread()
my_thread.start()
# Do some other stuff
my_thread.keep_running = False
my_thread.join()
You can keep a thread pool for each type of thread and then terminate them accordingly. For instance, you can keep them in a Queue.Queue globally and then .stop() each as needed.
Edit// You can join every child thread you wish to stop to its parent with .join()
Related
I'm trying to run a function after my thread has completed but the function is not called. Code structure:
class():
def functiontocall() # uses data calculated in thread for plotting. only works when thread is complete
do something with self.A
def watchthread():
thread()
functiontocall()
# since this function depends on variable A, it throws an error.
# I tried: if thread.join == True: functiontocall but this did not call the function.
def thread():
def run():
pythoncom.CoInitialize()
--- do stuff --
for i in 1000:
thousands of calculations while updating state in GUI ---
A = result
self.A = A
thread = threading.Thread(target=run)
thread.start()
note: i removed 'self' for simplicity.
thread.join should tell me when the thread has finished but for some reason i still cant run the functiontocall.
Is this a bad way of organizing threads in general?
Edit: I can call the function after the thread is finished but I cannot access variables when the thread is running. e.g. 0-100% progress for a progress bar in my GUI. when I use:
def watchthread():
thread()
thread.join()
functiontocall()
I cannot update the status of the thread in my GUI. It just waits until the calculations are finished then runs functiontocall().
Because you're using threads, once the thread had started Python will move onto the next thing, it will not wait for the thread to finish unless you've asked it to.
With your code, if you want to wait for the thread function to finish before moving on then it doesn't sound like you need threading, a normal function would run, complete, and then Python will move onto running functiontocall()
If there's a reason you need to use threads which isn't coming across in the example then I would suggest using a thread.join()
threads = [] # list to hold threads if you have more than one
t = threading.Thread(target=run)
threads.append(t)
for thread in threads: # wait for all threads to finish
thread.join()
functiontocall() # will only run after all threads are done
Again, I'd suggest relooking at whether threads is what you need to use here as it doesn't seem apparent.
To update this answer based on the new information, this may be the way you want to have a variable be accessible. In this case the threads would all update your class variable A, your GUI update function also reads this periodically and updates your GUI.
class ThisClass():
def __init__(self):
self.A = 0
def function_to_call(self):
while self.A != 100: # assuming this is a progress bar to 100%
# update in GUI
def run(self):
# does calculations
with lock: # to prevent issues with threads accessing variable at the same time
self.A += calculations
def progress(self):
threads = [] # list to hold threads if you have more than one
t = threading.Thread(target=run)
threads.append(t)
f = threading.Thread(target=self.function_to_call)
threads.append(f)
for thread in threads:
thread.start()
for thread in threads: # wait for all threads to finish
thread.join()
My code runs N number of threads. I want to stop specific threads on some condition but the remaining threads should continue running. I am doing some operation once each thread finishes its job. Is there a way to stop running thread in Python 3.
My current code is implemented in Python2 which does this by "_Thread__stop()". Is there any identical thing in Python3?
The practice is to "signal" the thread that it is time to finish and then the thread needs to exit. This is not killing like you kill a process but a regular state machine behavior of your thread function.
For example, suppose your thread is lopping. You should insert an if statement inside the loop that instructing the thread function to break or return if stop is True. The stop variable should be a shared variable with the main thread (or the thread who need to stop out thread) that will change it to True. usually after this, the stopper thread will want to wait for the thread completion by join()
It's a bad habit to kill a thread, better is to create a "flag" which will tell you when your thread made its work done.
Consider the following example:
import threading
import random
class CheckSomething(threading.Thread):
def __init__(self, variable):
super(CheckSomething, self).__init__()
self.start_flag = threading.Event()
self.variable = variable
def check_position(self, variable):
x = random.randint(100)
if variable == x:
self.stop_checking()
def run(self):
while True:
self.check_position(self.variable)
def stop_checking():
self.start_flag.set()
def stopped():
return self.start_flag.is_set()
The set() method of Event() set its status to True. More you can read in docs: https://docs.python.org/3.5/library/threading.html
So you need to call stop_checking() when you meet a condition where you want exit.
I have searched for python barrier but there is very few related issues. I am still confused about barrier.wait(), even my code works.
I utilize python barrier to implement such a function: A main thread and n sub-threads. In each round, the main thread waits for all the sub-threads finishing their current work, and then all the threads go to the next round until some condition we meet. Thus, I found that barrier is proper to implement this function, here is my code for main thread.
def superstep(self):
workers = []
barrier = threading.Barrier(self.num_workers+1)
for vertex in self.vertices:
worker = Worker(vertex, barrier)
workers.append(worker)
worker.start()
while self.flag:
barrier.wait()
self.distributeMessages()
self.count += 1
print ("superstep: ", self.count)
self.flag = self.isTerminated()
for worker in workers:
worker.flag = False
for worker in workers:
worker.join()
the first 'for' loop creates n threads named worker and stored in a list workers.
the 'while' loop is the main thread that waits for other sub-threads, and breaks when self.flag is False.
the second 'for' loop used for setting flag to False in each worker(sub-threads), telling them to exit loop.
here is my Worker class.
class Worker(threading.Thread):
def __init__(self, vertex, barrier):
threading.Thread.__init__(self)
self.vertex = vertex
self.flag = True
self.barrier = barrier
def run(self):
while self.flag:
self.barrier.wait()
do something
The code works well, all the threads can join(). But as I looked at python barrier, all the threads will release simultaneously when all the threads call wait(). What if the main threads breaks from its while loop and all the other threads are waiting for it just right, in this case, the second 'for' loop is useless and sub-threads will never join().
So how this code works, is there any other way to exit the barrier instead of raising BrokenBarrierError? In addition, if I add some code in the second 'for' loop, print some information or something else, the procedure is blocked. I guess there must be some sub-threads that are in wait() and have no chance to check flag, so they cannot exit from run() of threads.
If you don't want to use abort you could have two calls to Barrier.wait in each thread. This would break the operation to two parts. In first part worker threads would do their work and main thread would update the flag status. Then on second part every thread would check the flag status and exit the loop if necessary.
On code level it would look something like this:
# Main
def superstep(self):
workers = []
barrier = threading.Barrier(self.num_workers+1)
for vertex in self.vertices:
worker = Worker(vertex, barrier)
workers.append(worker)
worker.start()
while self.flag:
barrier.wait()
self.distributeMessages()
self.count += 1
print ("superstep: ", self.count)
self.flag = self.isTerminated()
for worker in workers:
worker.flag = self.flag
barrier.wait()
for worker in workers:
worker.join()
# Worker
def run(self):
while self.flag:
self.barrier.wait()
# do something
self.barrier.wait()
You can call
self.barrier.abort()
to release the waiting workers after the second for loop, and catch BrokenBarrierError in the worker's run() method.
I have some code that needs to run against several other systems that may hang or have problems not under my control. I would like to use python's multiprocessing to spawn child processes to run independent of the main program and then when they hang or have problems terminate them, but I am not sure of the best way to go about this.
When terminate is called it does kill the child process, but then it becomes a defunct zombie that is not released until the process object is gone. The example code below where the loop never ends works to kill it and allow a respawn when called again, but does not seem like a good way of going about this (ie multiprocessing.Process() would be better in the __init__()).
Anyone have a suggestion?
class Process(object):
def __init__(self):
self.thing = Thing()
self.running_flag = multiprocessing.Value("i", 1)
def run(self):
self.process = multiprocessing.Process(target=self.thing.worker, args=(self.running_flag,))
self.process.start()
print self.process.pid
def pause_resume(self):
self.running_flag.value = not self.running_flag.value
def terminate(self):
self.process.terminate()
class Thing(object):
def __init__(self):
self.count = 1
def worker(self,running_flag):
while True:
if running_flag.value:
self.do_work()
def do_work(self):
print "working {0} ...".format(self.count)
self.count += 1
time.sleep(1)
You might run the child processes as daemons in the background.
process.daemon = True
Any errors and hangs (or an infinite loop) in a daemon process will not affect the main process, and it will only be terminated once the main process exits.
This will work for simple problems until you run into a lot of child daemon processes which will keep reaping memories from the parent process without any explicit control.
Best way is to set up a Queue to have all the child processes communicate to the parent process so that we can join them and clean up nicely. Here is some simple code that will check if a child processing is hanging (aka time.sleep(1000)), and send a message to the queue for the main process to take action on it:
import multiprocessing as mp
import time
import queue
running_flag = mp.Value("i", 1)
def worker(running_flag, q):
count = 1
while True:
if running_flag.value:
print(f"working {count} ...")
count += 1
q.put(count)
time.sleep(1)
if count > 3:
# Simulate hanging with sleep
print("hanging...")
time.sleep(1000)
def watchdog(q):
"""
This check the queue for updates and send a signal to it
when the child process isn't sending anything for too long
"""
while True:
try:
msg = q.get(timeout=10.0)
except queue.Empty as e:
print("[WATCHDOG]: Maybe WORKER is slacking")
q.put("KILL WORKER")
def main():
"""The main process"""
q = mp.Queue()
workr = mp.Process(target=worker, args=(running_flag, q))
wdog = mp.Process(target=watchdog, args=(q,))
# run the watchdog as daemon so it terminates with the main process
wdog.daemon = True
workr.start()
print("[MAIN]: starting process P1")
wdog.start()
# Poll the queue
while True:
msg = q.get()
if msg == "KILL WORKER":
print("[MAIN]: Terminating slacking WORKER")
workr.terminate()
time.sleep(0.1)
if not workr.is_alive():
print("[MAIN]: WORKER is a goner")
workr.join(timeout=1.0)
print("[MAIN]: Joined WORKER successfully!")
q.close()
break # watchdog process daemon gets terminated
if __name__ == '__main__':
main()
Without terminating worker, attempt to join() it to the main process would have blocked forever since worker has never finished.
The way Python multiprocessing handles processes is a bit confusing.
From the multiprocessing guidelines:
Joining zombie processes
On Unix when a process finishes but has not been joined it becomes a zombie. There should never be very many because each time a new process starts (or active_children() is called) all completed processes which have not yet been joined will be joined. Also calling a finished process’s Process.is_alive will join the process. Even so it is probably good practice to explicitly join all the processes that you start.
In order to avoid a process to become a zombie, you need to call it's join() method once you kill it.
If you want a simpler way to deal with the hanging calls in your system you can take a look at pebble.
I only know basic concepts for multi-threading, and I currently encounter a situation that needs some help.
I have two tasks to finish, and both should be executed continuously. The thing is that the second task should start only after the first thread did some jobs first. Right now the two thread classes look roughly like the following:
finished = False # shared flag
class first(threading.Thread):
def __init__(self, cond, finished):
threading.Thread.__init__(self)
self.cond = cond
self.finished = finished
def run(self):
self.cond.aquire()
do_something()
self.finished = True #change the flag
self.cond.notify()
self.cond.release()
do_something_else()
class second(threading.Thread):
def __init__(self, cond, finished):
threading.Thread.__init__(self)
self.cond = cond
self.finished = finished
def run(self):
self.cond.aquire()
while self.finished == False:
self.cond.wait()
self.cond.release()
do_something()
However, the fact is that the program still executes randomly regardless of the wait() and notify(). Can anybody help me with this issue? Thanks.
self.finished in class first is a copy of the value of the global finished, not a reference to it, so it has no live relationship to the self.finished of class second.
You should probably create a global Queue object (which is designed to be used with the threading module). Have both classes refer to the queue, and have the first thread write a go-ahead signal to the queue, and the second thread block until it reads the go-ahead.
You can avoid synchronization altogether. Use 3 threads instead of 2.
Thread 1a 'does some job' and terminates.
Thread 1b starts where 1a ended, and
Thread 2 starts independently.
(Also I suppose you know that you cannot effectively share CPU with Python threads; these are only good for I/O waiting in parallel. When you need CPU-bound parallelization, you use multiprocessing.)