I'm trying to run a function after my thread has completed but the function is not called. Code structure:
class():
def functiontocall() # uses data calculated in thread for plotting. only works when thread is complete
do something with self.A
def watchthread():
thread()
functiontocall()
# since this function depends on variable A, it throws an error.
# I tried: if thread.join == True: functiontocall but this did not call the function.
def thread():
def run():
pythoncom.CoInitialize()
--- do stuff --
for i in 1000:
thousands of calculations while updating state in GUI ---
A = result
self.A = A
thread = threading.Thread(target=run)
thread.start()
note: i removed 'self' for simplicity.
thread.join should tell me when the thread has finished but for some reason i still cant run the functiontocall.
Is this a bad way of organizing threads in general?
Edit: I can call the function after the thread is finished but I cannot access variables when the thread is running. e.g. 0-100% progress for a progress bar in my GUI. when I use:
def watchthread():
thread()
thread.join()
functiontocall()
I cannot update the status of the thread in my GUI. It just waits until the calculations are finished then runs functiontocall().
Because you're using threads, once the thread had started Python will move onto the next thing, it will not wait for the thread to finish unless you've asked it to.
With your code, if you want to wait for the thread function to finish before moving on then it doesn't sound like you need threading, a normal function would run, complete, and then Python will move onto running functiontocall()
If there's a reason you need to use threads which isn't coming across in the example then I would suggest using a thread.join()
threads = [] # list to hold threads if you have more than one
t = threading.Thread(target=run)
threads.append(t)
for thread in threads: # wait for all threads to finish
thread.join()
functiontocall() # will only run after all threads are done
Again, I'd suggest relooking at whether threads is what you need to use here as it doesn't seem apparent.
To update this answer based on the new information, this may be the way you want to have a variable be accessible. In this case the threads would all update your class variable A, your GUI update function also reads this periodically and updates your GUI.
class ThisClass():
def __init__(self):
self.A = 0
def function_to_call(self):
while self.A != 100: # assuming this is a progress bar to 100%
# update in GUI
def run(self):
# does calculations
with lock: # to prevent issues with threads accessing variable at the same time
self.A += calculations
def progress(self):
threads = [] # list to hold threads if you have more than one
t = threading.Thread(target=run)
threads.append(t)
f = threading.Thread(target=self.function_to_call)
threads.append(f)
for thread in threads:
thread.start()
for thread in threads: # wait for all threads to finish
thread.join()
Related
Problem description:
I am working with the simulator to extract some dataset from it. The idea is to run multiple processes to perform various task. For example: moving the vehicle using one process and data collection using another process. In the data collection process, 3 threads are running to record three different data types and the recording has to occur periodically. Also, the data should be recorded synchronously.
Sample code is provided without details.
import threading
import multiprocessing
import time
class DataRecorder:
def __init__(self):
"""
some parameters
"""
pass
def move_vehicle(self, path):
pass
def record_data1(self):
pass
def record_data2(self):
pass
def record_data3(self):
pass
def record_data():
t1 = threading.Thread(target=self.record_data1)
t2 = threading.Thread(target=self.record_data2)
t3 = threading.Thread(target=self.record_data3)
threads = [t1, t2, t3]
for thread in threads:
thread.start()
while (True):
for thread in threads:
if not thread.is_alive()
thread.start() # leads to threads can only be started once
for thread in threads:
if thread.is_alive()
thread.join()
time.sleep(1)
def stop_recording(self, p1):
if p1.is_alive():
p1.terminate()
def move_and_record():
P1 = multiprocessing.Process(target=self.record_data)
P1.start()
self.move_vehicle(path)
self.stop_recording(P1)
The problem:
RuntimeError: threads can only be started once.
And inside while loop, the threads gets stopped after 1st iteration. I have tried both without .join() part and with .join() part.
I am also looking for another alternative to solve this problem.
I have the following script (don't refer to the contents):
import _thread
def func1(arg1, arg2):
print("Write to CLI")
def verify_result():
func1()
for _ in range (4):
_thread.start_new_thread(func1, (DUT1_CLI, '0'))
verify_result()
I want to concurrently execute (say 4 threads) func1() which in my case includes a function call that can take time to execute. Then, only after the last thread finished its work I want to execute verify_result().
Currently, the result I get is that all threads finish their job, but verify_result() is executed before all threads finish their job.
I have even tried to use the following code (of course I imported threading) under the for loop but that didn't do the work (don't refer to the arguments)
t = threading.Thread(target = Enable_WatchDog, args = (URL_List[x], 180, Terminal_List[x], '0'))
t.start()
t.join()
Your last threading example is close, but you have to collect the threads in a list, start them all at once, then wait for them to complete all at once. Here's a simplified example:
import threading
import time
# Lock to serialize console output
output = threading.Lock()
def threadfunc(a,b):
for i in range(a,b):
time.sleep(.01) # sleep to make the "work" take longer
with output:
print(i)
# Collect the threads
threads = []
for i in range(10,100,10):
# Create 9 threads counting 10-19, 20-29, ... 90-99.
thread = threading.Thread(target=threadfunc,args=(i,i+10))
threads.append(thread)
# Start them all
for thread in threads:
thread.start()
# Wait for all to complete
for thread in threads:
thread.join()
Say you have a list of threads.
You loop(each_thread) over them -
for each_thread in thread_pool:
each_thread.start()
within the loop to start execution of the run function within each thread.
The same way, you write another loop after you start all threads and have
for each_thread in thread_pool:
each_thread.join()
what join does is that it will wait for thread i to finish execution before letting i+1th thread to finish execution.
The threads would run concurrently, join() would just synchronize the way each thread returns its results.
In your case specifically, you can the join() loop and the run verify_result() function.
My code runs N number of threads. I want to stop specific threads on some condition but the remaining threads should continue running. I am doing some operation once each thread finishes its job. Is there a way to stop running thread in Python 3.
My current code is implemented in Python2 which does this by "_Thread__stop()". Is there any identical thing in Python3?
The practice is to "signal" the thread that it is time to finish and then the thread needs to exit. This is not killing like you kill a process but a regular state machine behavior of your thread function.
For example, suppose your thread is lopping. You should insert an if statement inside the loop that instructing the thread function to break or return if stop is True. The stop variable should be a shared variable with the main thread (or the thread who need to stop out thread) that will change it to True. usually after this, the stopper thread will want to wait for the thread completion by join()
It's a bad habit to kill a thread, better is to create a "flag" which will tell you when your thread made its work done.
Consider the following example:
import threading
import random
class CheckSomething(threading.Thread):
def __init__(self, variable):
super(CheckSomething, self).__init__()
self.start_flag = threading.Event()
self.variable = variable
def check_position(self, variable):
x = random.randint(100)
if variable == x:
self.stop_checking()
def run(self):
while True:
self.check_position(self.variable)
def stop_checking():
self.start_flag.set()
def stopped():
return self.start_flag.is_set()
The set() method of Event() set its status to True. More you can read in docs: https://docs.python.org/3.5/library/threading.html
So you need to call stop_checking() when you meet a condition where you want exit.
I have been trying to use Threads in python. I am working on a Pi hardware project.
Here's the problem:
When I create a thread, and call it like this, the loop keeps creating new threads before the old ones are completed. Hence, slowing the program down... (printing 'threading.active_count' displays 20+ active threads).
while True:
t4 = Thread(target = myFunc, args=())
t4.start()
print("Hello World")
I need a threading process that runs the same function over and over on a SINGLE thread without affecting or delaying my main program. i.e. when a thread has completed executing the function, run it again... but my main should still be printing "Hello World" as normal.
I've found one way to stop it crashing, which is to sit and "wait" until the thread is finished, and then start again. However, this is a blocking approach, and completely defeats the purpose of threading.
while True:
t4 = Thread(target = myFunc, args=())
t4.start()
t4.join()
print("Hello World")
Any suggestions?
You can use a multiprocessing.pool.ThreadPool to manage both the starting of new threads and limiting the maximum number of them executing concurrently.
from multiprocessing.pool import ThreadPool
from random import randint
import threading
import time
MAX_THREADS = 5 # Number of threads that can run concurrently.
print_lock = threading.Lock() # Prevent overlapped printing from threads.
def myFunc():
time.sleep(random.uniform(0, 1)) # Pause a variable amount of time.
with print_lock:
print('myFunc')
def test():
pool = ThreadPool(processes=MAX_THREADS)
for _ in range(100): # Submit as many tasks as desired.
pool.apply_async(myFunc, args=())
pool.close() # Done adding tasks.
pool.join() # Wait for all tasks to complete.
print('done')
if __name__ == '__main__':
test()
I need a threading process that runs the same function over and over on a SINGLE thread
This snippet creates a single thread that continually calls myFunc().
def threadMain() :
while True :
myFunc()
t4 = Thread(target = threadMain, args=())
t4.start()
setDaemon(True) from threading.Thread class more here https://docs.python.org/2/library/threading.html#threading.Thread.daemon
Make a delegate thread - i.e. a thread to run your other threads in sequence:
def delegate(*args):
while True:
t = Thread(target=myFunc, args=args) # or just call myFunc(*args) instead of a thread
t.start()
t.join()
t = Thread(target=delegate, args=())
t.start()
while True:
print("Hello world!")
Or even better, redesign your myFunc() to run its logic within a while True: ... loop and start the thread only once.
I'd also advise you to add some sort of a delay (e.g. time.sleep()) if you're not performing any work in your threads to help with context switching.
I have a large dataset in a list that I need to do some work on.
I want to start x amounts of threads to work on the list at any given time, until everything in that list has been popped.
I know how to start x amounts of threads (lets say 20) at a given time (by using thread1....thread20.start())
but how do I make it start a new thread when one of the first 20 threads finish? so at any given time there are 20 threads running, until the list is empty.
what I have so far:
class queryData(threading.Thread):
def __init__(self,threadID):
threading.Thread.__init__(self)
self.threadID = threadID
def run(self):
global lst
#Get trade from list
trade = lst.pop()
tradeId=trade[0][1][:6]
print tradeId
thread1 = queryData(1)
thread1.start()
Update
I have something going with the following code:
for i in range(20):
threads.append(queryData(i))
for thread in threads:
thread.start()
while len(lst)>0:
for iter,thread in enumerate(threads):
thread.join()
lock.acquire()
threads[iter] = queryData(i)
threads[iter].start()
lock.release()
Now it starts 20 threads in the beginning...and then keeps starting a new thread when one finishes.
However, it is not efficient, as it waits for the first one in the list to finish, and then the second..and so on.
Is there a better way of doing this?
Basically I need:
-Start 20 threads:
-While list is not empty:
-wait for 1 of the 20 threads to finish
-reuse or start a new thread
As I suggested in a comment, I think using a multiprocessing.pool.ThreadPool would be appropriate — because it would handle much of the thread management you're manually doing in your code automatically. Once all the threads are queued-up for processing via ThreadPool's apply_async() method calls, the only thing that needs to be done is wait until they've all finished execution (unless there's something else your code could be doing, of course).
I've translated the code in my linked answer to another related question so it's more similar to what you appear to be doing to make it easier to understand in the current context.
from multiprocessing.pool import ThreadPool
from random import randint
import threading
import time
MAX_THREADS = 5
print_lock = threading.Lock() # Prevent overlapped printing from threads.
def query_data(trade):
trade_id = trade[0][1][:6]
time.sleep(randint(1, 3)) # Simulate variable working time for testing.
with print_lock:
print(trade_id)
def process_trades(trade_list):
pool = ThreadPool(processes=MAX_THREADS)
results = []
while(trade_list):
trade = trade_list.pop()
results.append(pool.apply_async(query_data, (trade,)))
pool.close() # Done adding tasks.
pool.join() # Wait for all tasks to complete.
def test():
trade_list = [[['abc', ('%06d' % id) + 'defghi']] for id in range(1, 101)]
process_trades(trade_list)
if __name__ == "__main__":
test()
You can wait for a thread to complete with : thread.join(). This call will block until that thread completes, at which point you can create a new one.
However, instead of respawning a Thread each time, why not recycle your existing threads ?
This can be done by the use of tasks for example. You keep a list of tasks in a shared collection, and when one of your threads finishes a task, it retrieves another one from that collection.