I am trying to use python multiprocessing to fill a queue with strings and then print them back out but am having trouble. Could someone point out what I am doing wrong?
import multiprocessing
my_q = multiprocessing.Queue()
my_list =[i for i in range(0,100)]
def enqueue(q):
for data in my_list:
q.put(data)
def get_it(q):
while not q.empty():
item = q.get()
print(item)
p1 = multiprocessing.Process(target=enqueue, args=(my_q,))
p2 = multiprocessing.Process(target=get_it, args=(my_q,))
p1.start()
p2.start()
p1.join()
p2.join()
This program executes without printing anything.
If get_it is executed before the queue is populated, it will return immediately, printing nothing.
You need to make sure the queue is populated before the get_it is called.
For example, wait until enqueue is called until all items are enqueued:
...
p1 = multiprocessing.Process(target=enqueue, args=(my_q,))
p1.start()
p1.join()
p2 = multiprocessing.Process(target=get_it, args=(my_q,))
p2.start()
p2.join()
Or modifying get_it like below not to end too early:
...
def get_it(q):
while True:
item = q.get()
if item is None: # loop until sentinel value (None) appear.
break
print(item)
my_list.append(None) # sentinel value to denote end of input value
p1 = multiprocessing.Process(target=enqueue, args=(my_q,))
p2 = multiprocessing.Process(target=get_it, args=(my_q,))
p1.start()
p2.start()
p1.join()
p2.join()
or use multiprocess.pool.Pool.map instead:
import multiprocessing.pool
def get_it(item):
print(item)
pool = multiprocessing.pool.Pool()
pool.map(get_it, range(100))
Related
I am using two loops and cannot figure out how to properly update the dictionary in one loop and use it in the other loop.
In the first loop I am adding a new pair to the dictionary, but in the second loop I don't see any changes, how do I do it correctly?
import time
from multiprocessing import Process
dict_1 = {1:'1'}
def func_1():
while True:
dict_1.update({2:'2'})
print('Result func_1-',dict_1)
time.sleep(5)
def func_2():
while True:
print('Result func_2-',dict_1)
time.sleep(5)
if __name__ == '__main__':
p1 = Process(target=func_1)
p2 = Process(target=func_2)
p1.start()
p2.start()
p1.join()
p2.join()
Result func_1- {1: '1', 2: '2'} Result func_2- {1: '1'}
In the first cycle I see a new pair, but I do not see it in the second cycle.
You can solve this by using multiprocessing.Manager to create a managed dictionary for your purpose this way:
import time
from multiprocessing import Process, Manager
manager = Manager()
dict_1 = manager.dict({1:'1'})
def func_1():
while True:
dict_1.update({2:'2'})
print('Result func_1-',dict_1)
time.sleep(5)
def func_2():
while True:
print('Result func_2-',dict_1)
time.sleep(5)
if __name__ == '__main__':
p1 = Process(target=func_1)
p2 = Process(target=func_2)
p1.start()
p2.start()
p1.join()
p2.join()
Why while loop is ignored in work1? I would like to update value from string to another value in loop and output this value in process work2. Also already tried with Queue, but problem is I have only one variable which I would like to update in work1 and access to it at work2.
from multiprocessing import Process, Manager, Value
from ctypes import c_char_p
import time
def work1(string):
i = 2
string.value = i
# while True:
# print("work1")
# string.value = i + 1
# time.sleep(2)
def work2(string):
while True:
print("Value set in work1 " + str(string.value))
time.sleep(2)
if __name__ == '__main__':
manager = Manager()
string = manager.Value(int, 0);
p1=Process(target=work1, args=(string,))
p1.start()
p1.join()
p2=Process(target=work2, args=(string,))
p2.start()
p2.join()
That is because you didn't make your program parallel with two processes, but instead, two processes run in tandem. What you need to do is to start both process before any join. Like my modification below:
from multiprocessing import Process, Manager, Value
from ctypes import c_char_p
import time
def work1(string):
i = 2
string.value = i
while True:
i = i+1
string.value = i
print("work1 set value to "+str(string.value))
time.sleep(2)
def work2(string):
while True:
print("Value set in work1 " + str(string.value))
time.sleep(2)
if __name__ == '__main__':
manager = Manager()
string = manager.Value(int, 0, lock=False);
p1=Process(target=work1, args=(string,))
p2=Process(target=work2, args=(string,))
p1.start()
p2.start()
p2.join()
p1.join()
Indeed, if you write the code in this way, the join never happened due to the infinite while loop.
Based on this pretty useful tutorial I have tried to make a simple implementation of Python multiprocessing to measure its effectivity. The modules multi1, multi2, multi3 contain an ODE integration and exporting the calculated values in a csv (it does not matter, they are here for a script to do something).
import multiprocessing
import multi1
import multi2
import multi3
import time
t0 = time.time()
if __name__ == '__main__':
p1 = multiprocessing.Process(target = multi1.main(), args=())
p2 = multiprocessing.Process(target = multi2.main(), args=())
p3 = multiprocessing.Process(target = multi3.main(), args=())
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
t1 = time.time()
multi1.main()
multi2.main()
multi3.main()
t2 = time.time()
print t1-t0
print t2-t1
The problem is that the printed times are equal, so the multiprocessing didn't speed up the process. Why?
You called main in the main thread, and passed the return value (probably None) as the target, so no actual work is done in your worker processes. Remove the call parens, so you pass the function itself without calling it, e.g.:
p1 = multiprocessing.Process(target=multi1.main, args=())
p2 = multiprocessing.Process(target=multi2.main, args=())
p3 = multiprocessing.Process(target=multi3.main, args=())
This is the same basic problem seen in the threaded case.
I have 2 process running and I want them to complete before further down command executes (at the end of script it prints out that the program has ended). How can I make sure the process completes before printing out that it has ended?
from multiprocessing import Process
import datetime
class foo:
def fun1():
do sthn
def fun2():
do sthn
ob = foo()
if __name__ == '__main__':
p1 = Process(target = ob.fun1)
p1.start()
p2 = Process(target = ob.fun2)
p2.start()
endTime=datetime.datetime.now()
print 'Program Ending time is: ', endTime
You would use the .join() method, which blocks until the process is complete.
p1.join()
p2.join()
I'm trying to run 2 separate processes in my python application. So I have code like this:
from multiprocessing import Process
def f1():
while 1:
print('Hello')
def f2():
while 1:
print('Goodbye')
def main():
p1 = Process(target=f1, args=())
p1.start()
p1.join()
p2 = Process(target=f2, args=())
p2.start()
p2.join()
if __name__ == '__main__':
main()
This code does nothing on my machine, it doesn't produce any output. I thought initially that maybe it was an IDE-related problem, but it's the same on both my IDEs, PyScripter and IDLE.
Any ideas, why this doesn't print anything?
How about using Queue?
from multiprocessing import Process, Queue
def f1(q):
while 1:
q.put('Hello')
def f2(q):
while 1:
q.put('Goodbye')
def main():
q = Queue()
p1 = Process(target=f1, args=(q,))
p1.start()
p2 = Process(target=f2, args=(q,))
p2.start()
while True:
try:
print q.get()
except:
break
if __name__ == '__main__':
main()
You should save it and run outside the IDE:
C:\> python multi.py
then it infinitely prints out Hello. You should change your main to see both Hello and Goodbye:
def main():
p1 = Process(target=f1, args=())
p2 = Process(target=f2, args=())
p1.start()
p2.start()
p1.join()
p2.join()
Then you have a little happy race condition that constantly prints out GHoodbyeello because both processes use the same stdout resource concurrently.