I am trying to start a process using the multiprocessing.Process example from the python documentation.
Here is the example code:
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
I would expect the console to show me the output of the function f('bob'), but I only get to see the output of info('mainline').
So I think the process doesn't even start??
I have never before worked with multiprocessing, I bet it's a silly mistake I'm making.
I have also tried to set the start method multiprocessing.set_start_method('spawn') (see here), as 'spawn' seems to be the only valid one for windows.
But I only get a
RuntimeError: context has already been set
At the moment I think, I can't get the process to start.
Any Ideas how to solve this?
P.S. I am working on windows 10 in spyder 4.2.5 (maybe this something with the ipython console? Because I have heared, this is no normal python console).
But I have also tried the same example in the normal python shell, and it also only showed the output of info('mainline').
SOLVED: by running the script from cmd
Related
A basic example of multiprocessing Process class runs when executed from file, but not from IDLE. Why is that and can it be done?
from multiprocessing import Process
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
Yes. The following works in that function f is run in a separate (third) process.
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
However, to see the print output, at least on Windows, one must start IDLE from a console like so.
C:\Users\Terry>python -m idlelib
hello bob
(Use idlelib.idle on 2.x.) The reason is that IDLE runs user code in a separate process. Currently the connection between the IDLE process and the user code process is via a socket. The fork done by multiprocessing does not duplicate or inherit the socket connection. When IDLE is started via an icon or Explorer (in Windows), there is nowhere for the print output to go. When started from a console with python (rather than pythonw), output goes to the console, as above.
Assume the following python-program from the official docs (The Process class):
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
Running this code against my 3.7.7 interpreter on my Windows machine works, as expected, without any problems. However, running the same code against a Subinterpreter created in C++ fails with the following error (no Exception actually, the following error just gets printed to the console):
unrecognised option '-c'
I assume that the reason for this error is to be found in spawn.py (within the multiprocessing module, line 89):
...
return [_python_exe] + opts + ['-c', prog, '--multiprocessing-fork']
...
I could create my new process via Popen. This works, but the spawned process should be a child-process, not a completely independent process.
My question:
Why does this error occur? Is there any way to spawn a child process within a Subinterpreter via multiprocessing.Process?
Thank you!
UPDATE 1
As suggested, adding freeze_support fixes the error, but a new one occurs:
unrecognised option '--multiprocessing-fork'
why does my print function (in python multiprocess) print nothing?
from multiprocessing import Process, Queue
import os, time, random
def write(q):
print('Process to write: %s' % os.getpid())
for value in ['A', 'B', 'C']:
print('Put %s to queue...' % value)
q.put(value)
time.sleep(random.random())
def read(q):
print('Process to read: %s' % os.getpid())
while True:
value = q.get(True)
print('Get %s from queue.' % value)
if __name__=='__main__':
q = Queue()
pw = Process(target=write, args=(q,))
pr = Process(target=read, args=(q,))
pw.start()
print('start')
pr.start()
pw.join()
pr.terminate()
print('end')
I run it on spyder (windows 10 system).
My result on IPython console of Spyder:
runfile('C:/Users/Dust/Desktop/programs/crawl/test.py', wdir='C:/Users/Dust/Desktop/programs/crawl')
start
end
Result on Python console of Spyder:
>>> runfile('C:/Users/Dust/Desktop/programs/crawl/test.py', wdir='C:/Users/Dust/Desktop/programs/crawl')
start
Process to write: 12824
Put A to queue...
Put B to queue...
Put C to queue...
end
It is really weird. The results are different, but both are not what I want.
Could anyone help me find where is the problem in my program. Thanks a lot
The multiprocessing module uses fork to spawn child processes for parallelism. Windows does not implement fork, and Python emulation of it on Windows is incomplete. One of the missing effects is that stdout (used by print()) is not inherited by the child processes. Rather, child processes use the default stdout.
This works fine in a command window because it uses default stdout. This shows nothing in Spyder because a different pipe is used for output to the Spyder IPython console window.
See this question ("Simple Multiprocessing function in Spyder doesn't output results") and this answer for additional details and potential mitigations.
A basic example of multiprocessing Process class runs when executed from file, but not from IDLE. Why is that and can it be done?
from multiprocessing import Process
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
Yes. The following works in that function f is run in a separate (third) process.
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
However, to see the print output, at least on Windows, one must start IDLE from a console like so.
C:\Users\Terry>python -m idlelib
hello bob
(Use idlelib.idle on 2.x.) The reason is that IDLE runs user code in a separate process. Currently the connection between the IDLE process and the user code process is via a socket. The fork done by multiprocessing does not duplicate or inherit the socket connection. When IDLE is started via an icon or Explorer (in Windows), there is nowhere for the print output to go. When started from a console with python (rather than pythonw), output goes to the console, as above.
I have a problem with the code below, and with any code that uses the print function in the child processes. I can't see any printed statements, even if I use sys.std[err|out].write('worker') instead of print.
This is the code (from the official python documentation):
from multiprocessing import Process
def f(name):
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
The output is blank.
Note: The following code uses the threading module and it prints the output:
import threading
def f(name):
print 'hello', name
if __name__ == '__main__':
p = threading.Thread(target=f, args=('bob',))
p.start()
p.join()
Output: hello bob
Can you please point me to the solution? Thanks in advance.
Try this:
from multiprocessing import Process
import sys
def f(name):
print 'hello', name
sys.stdout.flush()
...
AFAIK the standard output of processed spawned by the multiprocessing module is buffered, hence you will see the output only if the buffer becomes full or you explicitly flush sys.stdout.
The docs for multiprocessing clearly explain why this won't work!
"Note: Functionality within this package requires that the __main__ method be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter."
Having run into this issue myself, sometimes this can be because the child process is actually silently failing before ever getting to the print statement. If this is the case, wrapping the child process code in a try-except block and returning the exception object (to be printed in the parent process) is an effective way to debug this.
I was using PyCharm IDE, and by checking the "Emulate terminal in output console" field in Run/Debug Configurations, it printed the desired result.
Hope it helps if you're using PyCharm.