Develop Reference

getting the return value of a function used in multiprocess

Say I have the below code, a function that does something, which is initiated in a Process, and returns a value.
from multiprocessing import Process
def my_func(arg):
return 'Hello, ' + arg
p1 = Process(target=my_func, args=('John',)
p1.start()
p1.join()
How do I get the return value of the function?

Answer
from multiprocessing import Process, Queue
Q = Queue()
def my_func(arg):
Q.put('Hello, ' + arg)
p1 = Process(target=my_func, args=('John',))
p1.start()
print(Q.get())
p1.join()

You can pass Queue of multiprocessing to my_func() as shown below:
from multiprocessing import Process, Queue
def my_func(arg, q):
q.put('Hello, ' + arg)
queue = Queue() # Here
p1 = Process(target=my_func, args=('John', queue))
p1.start()
print(queue.get())
p1.join()
This is the result below:
Hello, John
Be careful, if using queue module below with process, the program doesn't work properly:
import queue
queue = queue.Queue()
So, just use Queue of multiprocessing module with process as I used in the code above:
from multiprocessing import Queue
queue = Queue()

Concurrent methods inside another function in python

I am new with multiprocessing in python and so far all the example I've seen are this kind (with one or more methods in the file and then 'main'):
from multiprocessing import Process
def f1(a):
#do something
def f2(b):
#do something
if __name__ == '__main__':
f1(a1)
p = Process(target=f2, args=(b2,))
p.start()
p.join()
If I have instead a method who calls 2 functions in another file to be concurrent like in the following lines,
def function():
#do something
file2.f1(a) #first concurrent method
file2.f2(b) #second concurrent method
how should I do?
Can anyone make a simple example? I tried in this way, but it starts all the program again after the first loop :
def function():
#do something
for i in range(3):
p1 = Process(target=file2.f1, args=(a)) #first concurrent method
p2 = Process(target=file2.f2, args=(b)) #second concurrent method
p1.start()
p2.start()
p1.join()
p2.join()

The issue seems to be that args varialbe is incorrectly defined, it should be tuple and not a single variable:
def function():
#do something
for i in range(3):
p1 = Process(target=file2.f1, args=(a, )) #first concurrent method
p2 = Process(target=file2.f2, args=(b, )) #second concurrent method
p1.start()
p2.start()
p1.join()
p2.join()
If you the order of the executions is flexible, you can use the Pool class to trigger multiple calls:
from multiprocessing.pool import Pool
pool = Pool()
pool.map_async(f1, [(arg, )] * 3)
pool.map_async(f2, [(arg, )] * 3)
pool.close()
pool.join()

Python process to complete before downstream commands run

I have 2 process running and I want them to complete before further down command executes (at the end of script it prints out that the program has ended). How can I make sure the process completes before printing out that it has ended?
from multiprocessing import Process
import datetime
class foo:
def fun1():
do sthn
def fun2():
do sthn
ob = foo()
if __name__ == '__main__':
p1 = Process(target = ob.fun1)
p1.start()
p2 = Process(target = ob.fun2)
p2.start()
endTime=datetime.datetime.now()
print 'Program Ending time is: ', endTime

You would use the .join() method, which blocks until the process is complete.
p1.join()
p2.join()

Keep a process listening for intercommunication

I want to start 2 processes simultaneously. One will start processing immediately and the other will wait for the trigger (and the arguments) from the first process , for it to start processing.
Following is my code:-
Main.py
packet_obj = packet(i,30,30,30,30,0)
d = multiprocessing.Process(target = workers.send_trigger_to_controller, args = (packet_obj))
d.start()
# another process should start listening. Something like whenever a trigger is send
# from the first process, it should then start processing.
Workers.py
def send_trigger_to_controller(packet_obj):
while condition:
if condition:
d = multiprocessing.Process(target = send_packet_to_controller, args = (sensor_id,high_low,trigger_type))
d.start()
elif condition:
d = multiprocessing.Process(target = send_packet_to_controller, args = (sensor_id,high_low,trigger_type))
d.start()
elif condition:
d = multiprocessing.Process(target = send_packet_to_controller, args = (sensor_id,high_low,trigger_type))
d.start()
As of right now, I'm starting a new process for each condition satisfied. PS: All of these conditions are met, but at different interval of times, hence depending upon time instance, different argument values are passed.
I want to create a single process for all of this which will be listening for all of these. If any trigger is send, that process should listen and then process , rather than creating a complete new process.
how can I do it?

Start 2 processes and use a queue ( https://docs.python.org/2/library/multiprocessing.html ) to communicate.
Create 2 processes using multiprocessing.Process (one producer and one consumer process).
The producer is the one that start processing immediately and the consumer the one that waits until the producer process is ready.
The producer process when it finishes it puts the results of the computation to a queue.
The consumer process "listens" on the queue and when there is an item it starts processing.
Something like:
class ProducerProcess(Process):
def __init__(self, q, **kwargs):
Process.__init__(self,)
self.q = q
def run():
res = do_stuff()
q.put(res)
class ConsumerProcess(Process):
def __init__(self, q, **kwargs):
Process.__init__(self,)
self.q = q
def run():
while True:
args = q.get(block=True) # wait until there is an item in the queue
do_stuff(*args) # do stuff here
q = Queue()
p1 = ProducerProcess(q, **your_args)
p2 =ConsumerProcess(q, **extra_args)
p2.start()
p1.start()
# join the processes p1.join() p2.join() or use JoinableQueue depending what you need

python multithreading wait till all threads finished

This may have been asked in a similar context but I was unable to find an answer after about 20 minutes of searching, so I will ask.
I have written a Python script (lets say: scriptA.py) and a script (lets say scriptB.py)
In scriptB I want to call scriptA multiple times with different arguments, each time takes about an hour to run, (its a huge script, does lots of stuff.. don't worry about it) and I want to be able to run the scriptA with all the different arguments simultaneously, but I need to wait till ALL of them are done before continuing; my code:
import subprocess
#setup
do_setup()
#run scriptA
subprocess.call(scriptA + argumentsA)
subprocess.call(scriptA + argumentsB)
subprocess.call(scriptA + argumentsC)
#finish
do_finish()
I want to do run all the subprocess.call() at the same time, and then wait till they are all done, how should I do this?
I tried to use threading like the example here:
from threading import Thread
import subprocess
def call_script(args)
subprocess.call(args)
#run scriptA
t1 = Thread(target=call_script, args=(scriptA + argumentsA))
t2 = Thread(target=call_script, args=(scriptA + argumentsB))
t3 = Thread(target=call_script, args=(scriptA + argumentsC))
t1.start()
t2.start()
t3.start()
But I do not think this is right.
How do I know they have all finished running before going to my do_finish()?

Put the threads in a list and then use the Join method
threads = []
t = Thread(...)
threads.append(t)
...repeat as often as necessary...
# Start all threads
for x in threads:
x.start()
# Wait for all of them to finish
for x in threads:
x.join()

You need to use join method of Thread object in the end of the script.
t1 = Thread(target=call_script, args=(scriptA + argumentsA))
t2 = Thread(target=call_script, args=(scriptA + argumentsB))
t3 = Thread(target=call_script, args=(scriptA + argumentsC))
t1.start()
t2.start()
t3.start()
t1.join()
t2.join()
t3.join()
Thus the main thread will wait till t1, t2 and t3 finish execution.

In Python3, since Python 3.2 there is a new approach to reach the same result, that I personally prefer to the traditional thread creation/start/join, package concurrent.futures: https://docs.python.org/3/library/concurrent.futures.html
Using a ThreadPoolExecutor the code would be:
from concurrent.futures.thread import ThreadPoolExecutor
import time
def call_script(ordinal, arg):
print('Thread', ordinal, 'argument:', arg)
time.sleep(2)
print('Thread', ordinal, 'Finished')
args = ['argumentsA', 'argumentsB', 'argumentsC']
with ThreadPoolExecutor(max_workers=2) as executor:
ordinal = 1
for arg in args:
executor.submit(call_script, ordinal, arg)
ordinal += 1
print('All tasks has been finished')
The output of the previous code is something like:
Thread 1 argument: argumentsA
Thread 2 argument: argumentsB
Thread 1 Finished
Thread 2 Finished
Thread 3 argument: argumentsC
Thread 3 Finished
All tasks has been finished
One of the advantages is that you can control the throughput setting the max concurrent workers.
To use multiprocessing instead, you can use ProcessPoolExecutor.

I prefer using list comprehension based on an input list:
inputs = [scriptA + argumentsA, scriptA + argumentsB, ...]
threads = [Thread(target=call_script, args=(i)) for i in inputs]
[t.start() for t in threads]
[t.join() for t in threads]

You can have class something like below from which you can add 'n' number of functions or console_scripts you want to execute in parallel passion and start the execution and wait for all jobs to complete..
from multiprocessing import Process
class ProcessParallel(object):
"""
To Process the functions parallely
"""
def __init__(self, *jobs):
"""
"""
self.jobs = jobs
self.processes = []
def fork_processes(self):
"""
Creates the process objects for given function deligates
"""
for job in self.jobs:
proc = Process(target=job)
self.processes.append(proc)
def start_all(self):
"""
Starts the functions process all together.
"""
for proc in self.processes:
proc.start()
def join_all(self):
"""
Waits untill all the functions executed.
"""
for proc in self.processes:
proc.join()
def two_sum(a=2, b=2):
return a + b
def multiply(a=2, b=2):
return a * b
#How to run:
if __name__ == '__main__':
#note: two_sum, multiply can be replace with any python console scripts which
#you wanted to run parallel..
procs = ProcessParallel(two_sum, multiply)
#Add all the process in list
procs.fork_processes()
#starts process execution
procs.start_all()
#wait until all the process got executed
procs.join_all()

I just came across the same problem where I needed to wait for all the threads which were created using the for loop.I just tried out the following piece of code.It may not be the perfect solution but I thought it would be a simple solution to test:
for t in threading.enumerate():
try:
t.join()
except RuntimeError as err:
if 'cannot join current thread' in err:
continue
else:
raise

From the threading module documentation
There is a “main thread” object; this corresponds to the initial
thread of control in the Python program. It is not a daemon thread.
There is the possibility that “dummy thread objects” are created.
These are thread objects corresponding to “alien threads”, which are
threads of control started outside the threading module, such as
directly from C code. Dummy thread objects have limited functionality;
they are always considered alive and daemonic, and cannot be join()ed.
They are never deleted, since it is impossible to detect the
termination of alien threads.
So, to catch those two cases when you are not interested in keeping a list of the threads you create:
import threading as thrd
def alter_data(data, index):
data[index] *= 2
data = [0, 2, 6, 20]
for i, value in enumerate(data):
thrd.Thread(target=alter_data, args=[data, i]).start()
for thread in thrd.enumerate():
if thread.daemon:
continue
try:
thread.join()
except RuntimeError as err:
if 'cannot join current thread' in err.args[0]:
# catchs main thread
continue
else:
raise
Whereupon:
>>> print(data)
[0, 4, 12, 40]

Maybe, something like
for t in threading.enumerate():
if t.daemon:
t.join()

using only join can result in false-possitive interaction with thread. Like said in docs :
When the timeout argument is present and not None, it should be a
floating point number specifying a timeout for the operation in
seconds (or fractions thereof). As join() always returns None, you
must call isAlive() after join() to decide whether a timeout happened
– if the thread is still alive, the join() call timed out.
and illustrative piece of code:
threads = []
for name in some_data:
new = threading.Thread(
target=self.some_func,
args=(name,)
)
threads.append(new)
new.start()
over_threads = iter(threads)
curr_th = next(over_threads)
while True:
curr_th.join()
if curr_th.is_alive():
continue
try:
curr_th = next(over_threads)
except StopIteration:
break