Following program always hangs on Mac OS (Python 2.7.5) if I return big enough string on Mac OS. I can't says for sure what is the limit, but it works for smaller text.
It works fine on Ubuntu, but hangs on pipe_to_parent.send(result).
Does anybody know how to fix this? Is there anything wrong with the code bellow?
#!/usr/bin/python
import sys
from multiprocessing import Process, Pipe
def run(text, length):
return (text * ((length / len(text))+1))[:length]
def proc_func(pipe_to_parent):
result = {'status': 1, 'log': run('Hello World', 20000), 'details': {}, 'exception': ''}
pipe_to_parent.send(result)
sys.exit()
def call_run():
to_child, to_self = Pipe()
proc = Process(target=proc_func, args=(to_self,))
proc.start()
proc.join()
print(to_child.recv())
to_child.close()
to_self.close()
call_run()
The documentation shows an example that has some differences, as follows:
from multiprocessing import Process, Pipe
def f(conn):
conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
# This is the important part
# Note: conn.recv() is called _before_ process.join()
print parent_conn.recv() # prints "[42, None, 'hello']"
p.join()
In your example, you call .recv() after you've already called process.join().
...
proc = Process(target=proc_func, args=(to_self,))
proc.start()
proc.join()
print(to_child.recv())
...
To see exactly what is happening, we would have to look at the multiprocessing module code, but I'm guessing that the hanging is occurring because the pipe is attempting to begin a read from a closed end and blocking to wait for a response.
Related
I noticed data received through a multiprocess pipe can not be evaulated directly. In the example below, the code gets stuck in the child process.
import multiprocessing as mp
def child(conn):
while True:
if conn.recv()==1:
conn.send(1)
if conn.recv()==2:
conn.send(2)
conn.close()
def main():
parent_conn, child_conn = mp.Pipe()
p = mp.Process(target=child, args=(child_conn,))
p.start()
while True:
parent_conn.send(1)
print(parent_conn.recv())
p.join()
if __name__ == '__main__':
main()
But if I assign a variable to conn.recv() in the child process, as shown below. Then everything works.
def child(conn):
while True:
x = conn.recv()
if x==1:
conn.send(1)
if x==2:
conn.send(2)
conn.close()
I assume this is because the parent and child processes are running concurrently, so the data being passed should only be evaluated as they are received. Is this the the cause?
I am running Python 3.7 on Windows 10.
Below is my demo:
Thread xxx:
import xxx
import xxxx
def test_process1(hostname,proxy_param):
# just never run
try: # breakpoint 0
with open("/xxx","w+") as f: # breakpoint 1
f.write("something")
except Exception as e:
pass # just never run breakpoint 3
def test():
try:
a = Process(target=test_process1, args=(hostname,proxy_param))
a.start()
a.join() # you are blocking here. test_process1 not working and never quit
except Exception as e:
pass # breakpoint 4
function test_process1 just never run. No error, No breakpoint.
The test function code is in a big project, here is a demo.
Hope! this piece of code helps.
Workers list will get divided based on the number of processes in use.
Sample Code with ManagerList.
from subprocess import PIPE, Popen
from multiprocessing import Pool,Pipe
from multiprocessing import Process, Queue, Manager
def child_process(child_conn,output_list,messenger):
input_recvd = messenger["input"]
output_list.append(input_recvd)
print(input_recvd)
child_conn.close()
def parent_process(number_of_process=2):
workers_inputs = [{"input":"hello"}, {"input":"world"}]
with Manager() as manager:
processes = []
output_list = manager.list() # <-- can be shared between processes.
parent_conn, child_conn = Pipe()
for single_id_dict in workers_inputs:
pro_obj = Process(target=child_process, args=(child_conn,output_list,single_id_dict)) # Passing the list
pro_obj.start()
processes.append(pro_obj)
for p in processes:
p.join()
output_list = [single_feature for single_feature in output_list]
return output_list
parent_process()
OUTPUT:
hello
world
['hello', 'world']
ManagerList is useful to get the output from various parallel process it's like an inbuilt Queue Mechanism with easy to use and safe from deadlocks.
I have problems with python multiprocessing
python version 3.6.6
using Spyder IDE on windows 7
1.
queue is not being populated -> everytime I try to read it, its empty. Somewhere I read, that I have to get() it before process join() but it did not solve it.
from multiprocessing import Process,Queue
# define a example function
def fnc(i, output):
output.put(i)
if __name__ == '__main__':
# Define an output queue
output = Queue()
# Setup a list of processes that we want to run
processes = [Process(target=fnc, args=(i, output)) for i in range(4)]
print('created')
# Run processes
for p in processes:
p.start()
print('started')
# Exit the completed processes
for p in processes:
p.join()
print(output.empty())
print('finished')
>>>created
>>>started
>>>True
>>>finished
I would expect output to not be empty.
if I change it from .join() to
for p in processes:
print(output.get())
#p.join()
it freezes
2.
Next problem I have is with pool.map() - it freezes and has no chance to exceed memory limit. I dont even know how to debug such simple pieace of code.
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
pool = Pool(processes=4)
print('Pool created')
# print "[0, 1, 4,..., 81]"
print(pool.map(f, range(10))) # it freezes here
Hope its not a big deal to have two questions in one topic
Apperently the problem is Spyder's IPython console. When I run both from cmd, its executed properly.
Solution
for debugging in Spyder add .dummy to multiprocessing import
from multiprocessing.dummy import Process,Queue
It will not be executed by more processors, but you will get results and can actualy see the output. When debugging is done simply delete .dummy, place it in another file, import it and call it for example as function
multiprocessing_my.py
from multiprocessing import Process,Queue
# define a example function
def fnc(i, output):
output.put(i)
print(i)
def test():
# Define an output queue
output = Queue()
# Setup a list of processes that we want to run
processes = [Process(target=fnc, args=(i, output)) for i in range(4)]
print('created')
# Run processes
for p in processes:
p.start()
print('started')
# Exit the completed processes
for p in processes:
p.join()
print(output.empty())
print('finished')
# Get process results from the output queue
results = [output.get() for p in processes]
print('get results')
print(results)
test_mp.py
executed by selecting code and pressing ctrl+Enter
import multiprocessing_my
multiprocessing_my.test2()
...
In[9]: test()
created
0
1
2
3
started
False
finished
get results
[0, 1, 2, 3]
I have been learning how to use the Python multiprocessing module recently, and reading the official doc. In 16.6.1.2. Exchanging objects between processes there is a simple example about using pipe to exchange data.
And, in 16.6.2.4. Connection Objects, there is this statement, quoted "Raises EOFError if there is nothing left to receive and the other end was closed."
So, I revised the example as shown below. IMHO this should trigger an EOFError exception: nothing sent and the sending end is closed.
The revised code:
from multiprocessing import Process, Pipe
def f(conn):
#conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
#print parent_conn.recv() # prints "[42, None, 'hello']"
try:
print parent_conn.recv()
except EOFError:
pass
p.join()
But, when I tried the revised example on my Ubuntu 11.04 machine, Python 2.7.2, the script hang.
If anyone can point out to me what I missed, I would be very appreciative.
When you start a new process with mp.Process, the child process inherits the pipes of the parent. When the child closes conn, the parent process still has child_conn open, so the reference count for the pipe file descriptor is still greater than 0, and so EOFError is not raised.
To get the EOFError, close the end of the pipe in both the parent and child processes:
import multiprocessing as mp
def foo_pipe(conn):
conn.close()
def pipe():
conn = mp.Pipe()
parent_conn, child_conn = conn
proc = mp.Process(target = foo_pipe, args = (child_conn, ))
proc.start()
child_conn.close() # <-- Close the child_conn end in the main process too.
try:
print(parent_conn.recv())
except EOFError as err:
print('Got here')
proc.join()
if __name__=='__main__':
pipe()
I would like to create a shell which will control a separate process that I created with the multiprocessing module. Possible? How?
EDIT:
I have already achieved a way to send commands to the secondary process: I created a code.InteractiveConsole in that process, and attached it to an input queue and an output queue, so I can command the console from my main process. But I want it in a shell, probably a wx.py.shell.Shell, so a user of the program could use it.
First create the shell
Decouple the shell from your app by making its locals empty
Create your code string
Compile the code string and get a code object
Execute the code object in the shell
from wx.py.shell import Shell
frm = wx.Frame(None)
sh = Shell(frm)
frm.Show()
sh.interp.locals = {}
codeStr = """
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print q.get() # prints "[42, None, 'hello']"
p.join()
"""
code = compile(codeStr, '', 'exec')
sh.interp.runcode(code)
Note:
The codeStr I stole from the first poster may not work here due to some pickling issues. But the point is you can execute your own codeStr remotely in a shell.
You can create a Queue which you pass to the separate process. From the Python Docs:
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print q.get() # prints "[42, None, 'hello']"
p.join()
EXAMPLE: In the wx.py.shell.Shell Docs the constructur parameters are given as
__init__(self, parent, id, pos, size, style, introText, locals,
InterpClass, startupScript, execStartupScript, *args, **kwds)
I have not tried it, but locals might be a dictionary of local variables, which you can pass to the shell. So, I would try the following:
def f(cmd_queue):
shell = wx.py.shell.Shell(parent, id, pos, size, style, introText, locals(),
...)
q = Queue()
p = Process(target=f, args=(q,))
p.start()
Inside the shell, you should then be able to put your commands into cmd_queue which have then to be read in the parent process to be executed.