From different thread/interface my class getting work,my class has to process the work with configured delay time.
def getJob(job):
work = self._getNextWorkToRun(job)
if work is None:
return {}
#proceed to do work
job sends by different package to this class. I wanted to call _getNextWorkToRun() method every five minutes once only. but the job comes every seconds/less than seconds. So I have to wait until 5 minutes to call _getNextWorkToRun() once again with new job. Every job has reference (JOB1,JOB2...etc.,) and all the jobs have to complete with the delay of 5 mins.
What is the best way to achieve this.
below is an example of using threads, jobs will be added anytime to job queue from any other function and a get_job() function will run continuously to monitor jobs and process them on fixed interval until get a stop flag
from threading import Thread
from queue import Queue
import time
from random import random
jobs = Queue() # queue safely used between threads to pass jobs
run_flag = True
def job_feeder():
for i in range(10):
# adding a job to jobs queue, job could be anything, here we just add a string for simplicity
jobs.put(f'job-{i}')
print(f'adding job-{i}')
time.sleep(random()) # simulate adding jobs randomly
print('job_feeder() finished')
def get_job():
while run_flag:
if jobs.qsize(): # check if there is any jobs in queue first
job = jobs.get() # getting the job
print(f'executing {job}')
time.sleep(3)
print('get_job finished')
t1 = Thread(target=job_feeder)
t2 = Thread(target=get_job)
t1.start()
t2.start()
# we can make get_job() thread quit anytime by setting run_flag
time.sleep(20)
run_flag = False
# waiting for threads to quit
t1.join()
t2.join()
print('all clear')
output:
adding job-0
executing job-0
adding job-1
adding job-2
adding job-3
adding job-4
adding job-5
adding job-6
adding job-7
executing job-1
adding job-8
adding job-9
job_feeder() finished
executing job-2
executing job-3
executing job-4
executing job-5
executing job-6
get_job finished
all clear
note get_job() processed only 6 jobs because we send quit signal after 20 seconds
Related
I'm trying to launch a function (my_function) and stop its execution after a certain time is reached.
So i challenged multiprocessing library and everything works well. Here is the code, where my_function() has been changed to only create a dummy message.
from multiprocessing import Queue, Process
from multiprocessing.queues import Empty
import time
timeout=1
# timeout=3
def my_function(something):
time.sleep(2)
return f'my message: {something}'
def wrapper(something, queue):
message ="too late..."
try:
message = my_function(something)
return message
finally:
queue.put(message)
try:
queue = Queue()
params = ("hello", queue)
child_process = Process(target=wrapper, args=params)
child_process.start()
output = queue.get(timeout=timeout)
print(f"ok: {output}")
except Empty:
timeout_message = f"Timeout {timeout}s reached"
print(timeout_message)
finally:
if 'child_process' in locals():
child_process.kill()
You can test and verify that depending on timeout=1 or timeout=3, i can trigger an error or not.
My main problem is that the real my_function() is a torch model inference for which i would like to limit the number of threads (to 4 let's say)
One can easily do so if my_function were in the main process, but in my example i tried a lot of tricks to limit it in the child process without any success (using threadpoolctl.threadpool_limits(4), torch.set_num_threads(4), os.environ["OMP_NUM_THREADS"]=4, os.environ["MKL_NUM_THREADS"]=4).
I'm completely open to other solution that can monitor the time execution of a function while limiting the number of threads used by this function.
thanks
Regards
You can limit simultaneous process with Pool. (https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool)
You can set max tasks done per child. Check it out.
Here you have a sample from superfastpython by Jason Brownlee:
# SuperFastPython.com
# example of limiting the number of tasks per child in the process pool
from time import sleep
from multiprocessing.pool import Pool
from multiprocessing import current_process
# task executed in a worker process
def task(value):
# get the current process
process = current_process()
# report a message
print(f'Worker is {process.name} with {value}', flush=True)
# block for a moment
sleep(1)
# protect the entry point
if __name__ == '__main__':
# create and configure the process pool
with Pool(2, maxtasksperchild=3) as pool:
# issue tasks to the process pool
for i in range(10):
pool.apply_async(task, args=(i,))
# close the process pool
pool.close()
# wait for all tasks to complete
pool.join()
I'm new to Python threading and what I'm trying to do is :
1 Thread with While loop that will execute a GET request to an API each N seconds to refresh the data
A second Thread with a While loop that will use the data (ip addresses) to ping targets each N seconds.
So I was looking for a way to start the first Thread then only start the second after the first API call and then share these data to the second Thread so it can execute its logic.
Anyone can help me pls ? Thanks.
As per you requirements, here is a simple boilerplate code you might want to try,
import time
import threading
available = False
def thread1():
global available
while True:
# TODO: call API
# --------------
available = True # set available True after API call
time.sleep(5) # perform API calls after every 5 seconds
def thread2():
while True:
# TODO: perform ping
# --------------
# perform ping request after every 5 seconds
time.sleep(5)
if __name__ == "__main__":
t1 = threading.Thread(target=thread1, name="thread1")
t2 = threading.Thread(target=thread2, name="thread2")
t1.start()
while not available:
time.sleep(0.1)
else:
t2.start()
i m writing a script in which threads that do simple tasks start every 3 minutes . I m using the threading and schedule modules .
Because of the nature of multi-threading threads are using the same resources .
What i need to achieve ?
When creating a new thread i would like to check if there is any runningthread; and if there is , then wait until the running thread terminates , then start the new thread .
What i have tried ?
import threading
def run_threaded(job_fn):
job_thread = threading.Thread(target=job_fn)
bot.logger.info(" --------------No of active threads : "+threading.activeCount())
job_thread.start()
job_thread.join()
bot.logger.info(" --------------No of active threads : " + threading.activeCount())
schedule.every(3).minutes.do(run_threaded, job)
while True:
schedule.run_pending()
Note: On the example above every job_thread needs 5 minutes to complete . Thus it creates 1 thread every 6 minutes .
From what i understand , the job_thread.join() line is joining the main thread ( with any other active threads) . Although the schedule , is blocked thus no any other thread can be instantiated before the previous thread is finished . Is that correct ? And if yes , is this a good practice of doing this .?
And for the record .. Can the script execute other blocks of code while running a thread ? Or can it instantiate other threads, before the previous thread is finished , if they are going to execute an other job let's say job2 ?
Here's a simple example showing several threads being started, each of which will terminate itself at a different time and how the main thread can determine when each has terminated.
#!/usr/bin/env python3
import threading
import time
import queue
class Worker(threading.Thread):
def __init__(self, duration, tqueue):
self.duration = duration
self.tqueue = tqueue
super().__init__()
def run(self):
# Do real work here instead of just sleeping
time.sleep(self.duration)
# Tell parent we are gone by putting our instance to the queue
self.tqueue.put(self)
def main():
thr_duration = [2.0, 1.5, 0.5, 2.7, 1.25]
workers = []
thr_queue = queue.Queue()
# Start all threads
for dur in thr_duration:
worker = Worker(dur, thr_queue)
worker.start()
workers.append(worker)
print("Started thread {}, duration {}".format(worker.name, dur))
# Wait for all threads to terminate
while workers:
worker = thr_queue.get()
worker.join()
print("Reaped thread {}".format(worker.name))
workers.remove(worker)
if __name__ == '__main__':
main()
This is the problem I have: I'm using Python 2.7, and I have a code which runs in a thread, which has a critical region that only one thread should execute at the time. That code currently has no mutex mechanisms, so I wanted to inquire what I could use for my specific use case, which involves "dropping" of "queued" functions. I've tried to simulate that behavior with the following minimal working example:
useThreading=False # True
if useThreading: from threading import Thread, Lock
else: from multiprocessing import Process, Lock
mymutex = Lock()
import time
tstart = None
def processData(data):
#~ mymutex.acquire()
try:
print('thread {0} [{1:.5f}] Do some stuff'.format(data, time.time()-tstart))
time.sleep(0.5)
print('thread {0} [{1:.5f}] 1000'.format(data, time.time()-tstart))
time.sleep(0.5)
print('thread {0} [{1:.5f}] done'.format(data, time.time()-tstart))
finally:
#~ mymutex.release()
pass
# main:
tstart = time.time()
for ix in xrange(0,3):
if useThreading: t = Thread(target = processData, args = (ix,))
else: t = Process(target = processData, args = (ix,))
t.start()
time.sleep(0.001)
Now, if you run this code, you get a printout like this:
thread 0 [0.00173] Do some stuff
thread 1 [0.00403] Do some stuff
thread 2 [0.00642] Do some stuff
thread 0 [0.50261] 1000
thread 1 [0.50487] 1000
thread 2 [0.50728] 1000
thread 0 [1.00330] done
thread 1 [1.00556] done
thread 2 [1.00793] done
That is to say, the three threads quickly get "queued" one after another (something like 2-3 ms after each other). Actually, they don't get queued, they simply start executing in parallel after 2-3 ms after each other.
Now, if I enable the mymutex.acquire()/.release() commands, I get what would be expected:
thread 0 [0.00174] Do some stuff
thread 0 [0.50263] 1000
thread 0 [1.00327] done
thread 1 [1.00350] Do some stuff
thread 1 [1.50462] 1000
thread 1 [2.00531] done
thread 2 [2.00547] Do some stuff
thread 2 [2.50638] 1000
thread 2 [3.00706] done
Basically, now with locking, the threads don't run in parallel, but they run one after another thanks to the lock - as long as one thread is working, the others will block at the .acquire(). But this is not exactly what I want to achieve, either.
What I want to achieve is this: let's assume that when .acquire() is first triggered by a thread function, it registers an id of a function (say a pointer to it) in a queue. After that, the behavior is basically the same as with the Lock - while the one thread works, the others block at .acquire(). When the first thread is done, it goes in the finally: block - and here, I'd like to check to see how many threads are waiting in the queue; then I'd like to delete/drop all waiting threads except for the very last one - and finally, I'd .release() the lock; meaning that after this, what was the last thread in the queue would execute next. I'd imagine, I would want to write something like the following pseudocode:
...
finally:
if (len(mymutex.queue) > 2): # more than this instance plus one other waiting:
while (len(mymutex.queue) > 2):
mymutex.queue.pop(1) # leave alone [0]=this instance, remove next element
# at this point, there should be only queue[0]=this instance, and queue[1]= what was the last thread queued previously
mymutex.release() # once we releace, queue[0] should be gone, and the next in the queue should acquire the mutex/lock..
pass
...
With that, I'd expect a printout like this:
thread 0 [0.00174] Do some stuff
thread 0 [0.50263] 1000
thread 0 [1.00327] done
# here upon lock release, thread 1 would be deleted - and the last one in the queue, thread 2, would acquire the lock next:
thread 2 [1.00350] Do some stuff
thread 2 [1.50462] 1000
thread 2 [2.00531] done
What would be the most straightforward way to achieve this in Python?
Seems like you want a queue-like behaviour, so why not use Queue?
import threading
from Queue import Queue
import time
# threads advertise to this queue when they're waiting
wait_queue = Queue()
# threads get their task from this queue
task_queue = Queue()
def do_stuff():
print "%s doing stuff" % str(threading.current_thread())
time.sleep(5)
def queue_thread(sleep_time):
# advertise current thread waiting
time.sleep(sleep_time)
wait_queue.put("waiting")
# wait for permission to pass
message = task_queue.get()
print "%s got task: %s" % (threading.current_thread(), message)
# unregister current thread waiting
wait_queue.get()
if message == "proceed":
do_stuff()
# kill size-1 threads waiting
for _ in range(wait_queue.qsize() - 1):
task_queue.put("die")
# release last
task_queue.put("proceed")
if message == "die":
print "%s died without doing stuff" % threading.current_thread()
pass
t1 = threading.Thread(target=queue_thread, args=(1, ))
t2 = threading.Thread(target=queue_thread, args=(2, ))
t3 = threading.Thread(target=queue_thread, args=(3, ))
t4 = threading.Thread(target=queue_thread, args=(4, ))
# allow first thread to pass
task_queue.put("proceed")
t1.start()
t2.start()
t3.start()
t4.start()
thread-1 arrives first and "acquires" the section, other threads come later to wait at the queue (and advertise they're waiting). Then, when thread-1 leaves it gives permission to the last thread at the queue by telling all other thread to die, and the last thread to proceed.
You can have finer control using different messages, a typical one would be a thread-id in the wait_queue (so you know who is waiting, and the order in which it arrived).
You can probably utilize non-blocking operations (queue.put(block=False) and queue.get(block=False)) in your favour when you're set on what you need.
I want to make an synchronized simulation of graph coloring. To create the graph (tree) I am using igraph package and to synchronization I am using for the first time multiprocessing package. I built a graph where each node has attributes: label, color and parentColor. To color the tree I excecute the following function (I am not giving the full code because it is very long, and I think not necessary to solve my problem):
def sixColor(self):
root = self.graph.vs.find("root")
root["color"] = self.takeColorFromList(root["label"])
self.sendToChildren(root)
lista = []
for e in self.graph.vs():
lista.append(e.index)
p = multiprocessing.Pool(len(lista))
p.map(fun, zip([self]*len(lista), lista),chunksize=300)
def process_sixColor(self, id):
v = self.graph.vs.find(id)
if not v["name"] == "root":
while True:
if v["received"] == True:
v["received"] = False
#------------Part 1-----------
self.sendToChildren(v)
self.printInfo()
#-----------Part 2-------------
diffIdx = self.compareLabelWithParent(v)
if not diffIdx == -1:
diffIdxStr = str(bin(diffIdx))[2:]
charAtPos = (v["label"][::-1])[diffIdx]
newLabel = diffIdxStr + charAtPos
v["label"] = newLabel
self.sendToChildren(v)
colorNum = int(newLabel,2)
if colorNum in sixColorList:
v["color"] = self.takeColorFromList(newLabel)
self.printGraph()
break
I want to have that each node (except root) is calling function process_sixColor synchronously in parallel and will not evaluate Part 2before Part 1 will be made by all nodes. But I notice that this is not working properly and some nodes are evaluating before every other node will execute Part 1. How can I solve that problem?
You can use a combination of a multiprocessing.Queue and a multiprocessing.Event object to synchronize the workers. Make the main process create a Queue and an Event and pass both to all the workers. The Queue will be used by the workers to let the main process know that they are finished with part 1. The Event will be used by the main process to let all the workers know that all the workers are finished with part 1. Basically,
the workers will call queue.put() to let the main process know that they have reached part 2 and then call event.wait() to wait for the main process to give the green light.
the main process will repeatedly call queue.get() until it receives as many messages as there are workers in the worker pool and then call event.set() to give the green light for the workers to start with part 2.
This is a simple example:
from __future__ import print_function
from multiprocessing import Event, Process, Queue
def worker(identifier, queue, event):
# Part 1
print("Worker {0} reached part 1".format(identifier))
# Let the main process know that we have finished part 1
queue.put(identifier)
# Wait for all the other processes
event.wait()
# Start part 2
print("Worker {0} reached part 2".format(identifier))
def main():
queue = Queue()
event = Event()
processes = []
num_processes = 5
# Create the worker processes
for identifier in range(num_processes):
process = Process(target=worker, args=(identifier, queue, event))
processes.append(process)
process.start()
# Wait for "part 1 completed" messages from the processes
while num_processes > 0:
queue.get()
num_processes -= 1
# Set the event now that all the processes have reached part 2
event.set()
# Wait for the processes to terminate
for process in processes:
process.join()
if __name__ == "__main__":
main()
If you want to use this in a production environment, you should think about how to handle errors that occur in part 1. Right now if an exception happens in part 1, the worker will never call queue.put() and the main process will block indefinitely waiting for the message from the failed worker. A production-ready solution should probably wrap the entire part 1 in a try..except block and then send a special error signal in the queue. The main process can then exit immediately if the error signal is received in the queue.