I have some programm in which multiple processes try to finish some function. My aim now is to stop all the other processes after one process has successfully finished the function.
The python program shown below unfortunately waits until all the processes successfully solved the question given in find function. How can I fix my problem?
import multiprocessing
import random
FIND = 50
MAX_COUNT = 100000
INTERVAL = range(10)
def find(process, initial, return_dict):
succ = False
while succ == False:
start=initial
while(start <= MAX_COUNT):
if(FIND == start):
return_dict[process] = f"Found: {process}, start: {initial}"
succ = True
break;
i = random.choice(INTERVAL)
start = start + i
print(start)
processes = []
manager = multiprocessing.Manager()
return_code = manager.dict()
for i in range(5):
process = multiprocessing.Process(target=find, args=(f'computer_{i}', i, return_code))
processes.append(process)
process.start()
for process in processes:
process.join()
print(return_code.values())
output can be for example:
['Found: computer_0, start: 0', 'Found: computer_4, start: 4', 'Found: computer_2, start: 2', 'Found: computer_1, start: 1', 'Found: computer_3, start: 3']
But this output shows me the program is waiting until all processes are finished ...
Use an Event to govern if the processes should keep running.
Basically, it replaces succ with something that works over all processes.
import multiprocessing
import random
FIND = 50
MAX_COUNT = 1000
def find(process, initial, return_dict, run):
while run.is_set():
start = initial
while start <= MAX_COUNT:
if FIND == start:
return_dict[process] = f"Found: {process}, start: {initial}"
run.clear() # Stop running.
break
start += random.randrange(0, 10)
print(start)
if __name__ == "__main__":
processes = []
manager = multiprocessing.Manager()
return_code = manager.dict()
run = manager.Event()
run.set() # We should keep running.
for i in range(5):
process = multiprocessing.Process(
target=find, args=(f"computer_{i}", i, return_code, run)
)
processes.append(process)
process.start()
for process in processes:
process.join()
print(return_code.values())
Note that using __name__ is mandatory for multiprocessing to work properly on ms-windows and macOS.
On those systems, the main module is imported into newly created Python processes. This needs to happen without side effects such as starting a process, and the __name__ mechanism ensures that.
You can do this using multiprocessing.Queue and multiprocessing.Queue.get. How this works is that get by default blocks until there's something in the queue. So it will return the first result that gets appended to the queue, i.e. one of the processes finishing the search. After that, we can iterate over the processes and terminate each one (note that terminating a process doesn't kill child processes spawned by the process unless daemon is set to True).
import multiprocessing
import random
import time
FIND = 50
MAX_COUNT = 100000
INTERVAL = range(10)
queue = multiprocessing.Queue(maxsize=1)
def find(process, initial):
succ = False
while succ == False:
start=initial
while(start <= MAX_COUNT):
if(FIND == start):
queue.put(f"Found: {process}, start: {initial}")
break;
i = random.choice(INTERVAL)
start = start + i
print(process, start)
processes = []
manager = multiprocessing.Manager()
for i in range(5):
process = multiprocessing.Process(target=find, args=(f'computer_{i}', i))
processes.append(process)
process.start()
ret = queue.get()
for i in range(5):
process = processes[i]
process.terminate()
print(f'terminated {i}')
print(ret)
You might also want to look into setting the daemon, which kills the processes after the main process exits.
Related
I'm using multiprocessing to run workers on different files in parallel. Worker's results are put into queue. A listener gets the results from the queue and writes them to the file.
Sometimes listener might run into errors (of various origins). In this case, the listener silently dies, but all other processes continue running (rather surprisingly, worker errors causes all processes to terminate).
I would like to stop all processes (workers, listener, e.t.c.) when listener catches an error. How this can be done?
The scheme of my code is as follows:
def worker(file_path, q):
## do something
q.put(1.)
return True
def listener(q):
while True:
m = q.get()
if m == 'kill':
break
else:
try:
# do something and write to file
except Exception as err:
# raise error
tb = sys.exc_info()[2]
raise err.with_traceback(tb)
def main():
manager = mp.Manager()
q = manager.Queue(maxsize=3)
with mp.Pool(5) as pool:
watcher = pool.apply_async(listener, (q,))
files = ['path_1','path_2','path_3']
jobs = [ pool.apply_async(worker, (p,q,)) for p in files ]
# fire off workers
for job in jobs:
job.get()
# kill the listener when done
q.put('kill')
# run
if __name__ == "__main__":
main()
I tried introducing event = manager.Event() and using it as a flag in main():
## inside the pool, after starting workers
while True:
if event.is_set():
for job in jobs:
job.terminate()
No success. Calling os._exit(1) in listener exception block rises broken pipe error, but processes are not killed.
I also tried setting daemon = True,
for job in jobs:
job.daemon = True
Did not help.
In fact, to handle listener exceptions, I'm using a callable, as required by apply_async (so that they are not entirely silenced). This complicates the situation, but not much.
Thank you in advance.
As always there are many ways to accomplish what you're after, but I would probably suggest using an Event to signal that the processes should quit. I also would not use a Pool in this instance, as it only really simplifies things for simple cases where you need something like map. More complicated use cases quickly make it easier to just build you own "pool" with the functionality you need.
from multiprocessing import Process, Queue, Event
from random import random
def might_fail(a):
assert(a > .001)
def worker(args_q: Queue, result_q: Queue, do_quit: Event):
try:
while not do_quit.is_set():
args = args_q.get()
if args is None:
break
else:
# do something
result_q.put(random())
finally: #signal that worker is exiting even if exception is raised
result_q.put(None) #signal listener that worker is exiting
def listener(result_q: Queue, do_quit: Event, n_workers: int):
n_completed = 0
while n_workers > 0:
res = result_q.get()
if res is None:
n_workers -= 1
else:
n_completed += 1
try:
might_fail(res)
except:
do_quit.set() #let main continue
print(n_completed)
raise #reraise error after we signal others to stop
do_quit.set() #let main continue
print(n_completed)
if __name__ == "__main__":
args_q = Queue()
result_q = Queue()
do_quit = Event()
n_workers = 4
listener_p = Process(target=listener, args=(result_q, do_quit, n_workers))
listener_p.start()
for _ in range(n_workers):
worker_p = Process(target=worker, args=(args_q, result_q, do_quit))
worker_p.start()
for _ in range(1000):
args_q.put("some/file.txt")
for _ in range(n_workers):
args_q.put(None)
do_quit.wait()
print('done')
In python 3, I have to find prime numbers in an interval [a,b] using multiprocessing pipes in specific... Here is my code using the pool.map method:
import multiprocessing
import time
def isprime(num):
if num < 2:
return None
for i in range(2, num):
if (num % i) == 0:
return None
else:
return num
if __name__ == "__main__":
pool = multiprocessing.Pool(3)
start_time = time.perf_counter()
result = list(filter(lambda x: x is not None, pool.map(isprime, range(0,30))))
finish_time = time.perf_counter()
print(f"Program finished in {finish_time-start_time} seconds")
print(result)
i didnt fully understand the pipe method here is the code tht i found as an example while doing my research
from multiprocessing import Process, Pipe
def parentData(parent):
''' This function sends the data for the child process '''
parent.send(['Hello'])
parent.close()
def childData(child):
''' This function sends the data for the parent process '''
child.send(['Bye'])
child.close()
if __name__ == '__main__':
parent, child = Pipe() # Create Pipe
process1 = Process(target = parentData, args = (parent, )) # Create a process for handling parent data
process2 = Process(target = childData, args = (child, )) # Create a process for handling child data
process1.start() # Start the parent process
process2.start() # Start the child process
print(parent.recv()) # Display data received from child (BYE)
print(child.recv()) # Display data received from parent (HELLO)
process1.join() # Wait till the process completes its execution
process2.join()
Can anyone show me an example of doing the multiprocessing of prime numbers with pipes?
I'm learning python multiprocessing and I tried this code :
def f(name):
n = 0
print('running ', name)
for i in range(1000):
for j in range(1000):
for k in range(100):
n += 1
print(n)
if __name__ == '__main__':
tab = []
for i in range(10):
p = Process(target=f, args=(i,))
p.start()
tab.append(p)
for t in tab:
t.join()
It worked well, and I saw in the monitor that CPU were all running 100%.
But when I switched to my application, which is using a shared variable for multiprocessing, I saw that CPU were all running at 40-50% and execution times are worst than without multiprocessing.
def repeat_optimization(self, cycle, nb_cycles, res):
start = time.time()
res['a'].append(a())
res['b'].append(b())
end = time.time()
print("Benchmark iteration : " + str(cycle + 1) + "/" + str(nb_cycles) + " completed in " + str(round(end - start, 2)) + "s")
def parallelize_repeat_optimization(self, nb_cycles):
manager = Manager()
res = manager.dict()
res['a'] = []
res['b'] = []
tab = []
for cycle in range(nb_cycles):
p = Process(target=self.repeat_optimization, args=(cycle, nb_cycles, res))
p.start()
tab.append(p)
for t in tab:
t.join()
return res
What's wrong with my code, since I do not see a main difference with the first example ?
In any multiprocessing situation you will always have higher utilization if each process is completely independent. In this case, there are hidden costs to using a Manager to synchronize the two processes. Under the hood, the processes have to communicate with each other to make sure they do not overwrite each others' changes, and this means each process will waste a lot of time waiting for responses from the other process.
I want to kill a certain process or all processes in the middle of its execution. My sample code is as follows. How can I do that? Here, I want to kill the processes based on the current time. If the time is divisible by 2, I want to kill the processes, otherwise not.
import time
from multiprocessing import Process
def runTests(a, b):
time.sleep(10)
return a + b
def main(kill_processes):
print(kill_processes)
processes = []
for i in range(3):
print(i)
proc = Process(target=runTests, args=(2, 4,))
processes.append(proc)
proc.start()
for proc in processes:
proc.join()
if kill_processes:
print("killing")
proc.terminate()
if __name__ == "__main__":
if round(time.time()) % 2 == 0:
main(True)
else:
main(False)
This illustrates how to do it by using multiple threads in a manner similar to what's in the answer to the question multiprocessing in stoppable multithreading that I suggested you have a look at.
Basically all that is going on is all the join() calls are done in separate threads, so they won't block the main thread in the main process — which makes it possible for it to kill them.
import threading
import time
from multiprocessing import Process
def runTests(a, b):
time.sleep(10)
return a + b
def create_process(lock, i):
proc = Process(target=runTests, args=(2, 4,))
print(f'{proc.name} created')
proc.start()
with lock:
processes.append(proc)
proc.join()
def main(kill_processes):
global processes
N = 3
lock = threading.RLock()
processes = []
print(f'main({kill_processes=})')
for i in range(N):
thread = threading.Thread(target=create_process, args=(lock, i))
thread.start()
while True: # Wait for all processes to have been created.
with lock:
if len(processes) == N:
break
else:
time.sleep(.001)
if kill_processes:
print("Killing the processes")
for proc in processes:
proc.terminate()
print(f'process {proc} terminated')
if __name__ == "__main__":
main(True)
# if round(time.time()) % 2 == 0:
# main(True)
# else:
# main(False)
You can use multiprocessing.Event to signal termination condition to your child processes. Don't join child processes in the main process. Instead let main process and child processes run in their own loop. Check the termination condition in the main loop, and signal it using the multiprocessing.Event to the child processes.
The Event object is passed as an argument to child process. Child process continuously checks if the event is set, and stops its work if so. Main loop checks the termination condition and sets the Event if condition is met (in the below example main loop waits for Ctrl+c).
import multiprocessing as mp
import os
import time
def do_work(a, b, stop_event):
while not stop_event.is_set():
try:
time.sleep(2)
print(f"worker {os.getpid()}: working ...", a + b)
a += 1
b += 1
except KeyboardInterrupt:
print(f"worker {os.getpid()}: received SIGINT. ignore.")
pass
print(f"worker {os.getpid()}: stop_event is set. exit.")
if __name__ == "__main__":
stop_event = mp.Event()
procs = []
for i in range(3):
# p = mp.Process(target=do_work, args=(1, 2, stop_event), daemon=True)
p = mp.Process(target=do_work, args=(1, 2, stop_event))
p.start()
procs.append(p)
while True:
try:
print("main: waiting for termination signal")
time.sleep(1)
except KeyboardInterrupt:
print("main: received termination signal")
stop_event.set()
# wait for the processes to stop
for p in procs:
p.join()
for p in procs:
print(f"worker {p.pid} is terminated: {not p.is_alive()}")
# exit the main loop
break
print("main: bye")
If you want to terminate based on time, use the timeout paramter of join. One way is to set a stop time and as each process is joined, use time remaining as its timeout.
import time
from multiprocessing import Process
def runTests(a, b):
time.sleep(10)
return a + b
def main(kill_processes):
print(kill_processes)
processes = []
end = time.time() + 10 # wait 10 seconds
for i in range(3):
print(i)
proc = Process(target=runTests, args=(2, 4,))
processes.append(proc)
proc.start()
for proc in processes:
if kill_processes:
delta = end - time.time()
else:
delta = None
proc.join(delta)
if kill_processes:
print("killing")
proc.terminate()
proc.join(1)
if proc.is_alive():
proc.kill()
if __name__ == "__main__":
if round(time.time()) % 2 == 0:
main(True)
else:
main(False)
My multi-threading script raising this error:
thread.error : can't start new thread
when it reached 460 threads:
threading.active_count() = 460
I assume the old threads keeps stack up, since the script didn't kill them. This my code:
import threading
import Queue
import time
import os
import csv
def main(worker):
#Do Work
print worker
return
def threader():
while True:
worker = q.get()
main(worker)
q.task_done()
def main_threader(workers):
global q
global city
q = Queue.Queue()
for x in range(20):
t = threading.Thread(target=threader)
t.daemon = True
print "\n\nthreading.active_count() = " + str(threading.active_count()) + "\n\n"
t.start()
for worker in workers:
q.put(worker)
q.join()
How do I kill the old threads when their job is done? (Is the function returning not enough?)
Python threading API doesn't have any function to kill a thread (nothing like threading.kill(PID)).
That said, you should code some thread-stopping algorithm yourself. For example, your thread should somehow decide that is should terminate (e.g. check some global variable or check whether some signal has been sent) and simply return.
For example:
import threading
nthreads = 7
you_should_stop = [0 for _ in range(nthreads)]
def Athread(number):
while True:
if you_should_stop[number]:
print "Thread {} stopping...".format(number)
return
print "Running..."
for x in range(nthreads):
threading.Thread(target = Athread, args = (x, )).start()
for x in range(nthreads):
you_should_stop[x] = 1
print "\nStopped all threads!"