I have a problem with the code below, and with any code that uses the print function in the child processes. I can't see any printed statements, even if I use sys.std[err|out].write('worker') instead of print.
This is the code (from the official python documentation):
from multiprocessing import Process
def f(name):
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
The output is blank.
Note: The following code uses the threading module and it prints the output:
import threading
def f(name):
print 'hello', name
if __name__ == '__main__':
p = threading.Thread(target=f, args=('bob',))
p.start()
p.join()
Output: hello bob
Can you please point me to the solution? Thanks in advance.
Try this:
from multiprocessing import Process
import sys
def f(name):
print 'hello', name
sys.stdout.flush()
...
AFAIK the standard output of processed spawned by the multiprocessing module is buffered, hence you will see the output only if the buffer becomes full or you explicitly flush sys.stdout.
The docs for multiprocessing clearly explain why this won't work!
"Note: Functionality within this package requires that the __main__ method be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter."
Having run into this issue myself, sometimes this can be because the child process is actually silently failing before ever getting to the print statement. If this is the case, wrapping the child process code in a try-except block and returning the exception object (to be printed in the parent process) is an effective way to debug this.
I was using PyCharm IDE, and by checking the "Emulate terminal in output console" field in Run/Debug Configurations, it printed the desired result.
Hope it helps if you're using PyCharm.
Related
A basic example of multiprocessing Process class runs when executed from file, but not from IDLE. Why is that and can it be done?
from multiprocessing import Process
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
Yes. The following works in that function f is run in a separate (third) process.
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
However, to see the print output, at least on Windows, one must start IDLE from a console like so.
C:\Users\Terry>python -m idlelib
hello bob
(Use idlelib.idle on 2.x.) The reason is that IDLE runs user code in a separate process. Currently the connection between the IDLE process and the user code process is via a socket. The fork done by multiprocessing does not duplicate or inherit the socket connection. When IDLE is started via an icon or Explorer (in Windows), there is nowhere for the print output to go. When started from a console with python (rather than pythonw), output goes to the console, as above.
I am trying to start a process using the multiprocessing.Process example from the python documentation.
Here is the example code:
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
I would expect the console to show me the output of the function f('bob'), but I only get to see the output of info('mainline').
So I think the process doesn't even start??
I have never before worked with multiprocessing, I bet it's a silly mistake I'm making.
I have also tried to set the start method multiprocessing.set_start_method('spawn') (see here), as 'spawn' seems to be the only valid one for windows.
But I only get a
RuntimeError: context has already been set
At the moment I think, I can't get the process to start.
Any Ideas how to solve this?
P.S. I am working on windows 10 in spyder 4.2.5 (maybe this something with the ipython console? Because I have heared, this is no normal python console).
But I have also tried the same example in the normal python shell, and it also only showed the output of info('mainline').
SOLVED: by running the script from cmd
If I have the following code:
def f():
print 'ok!'
import sys
sys.exit()
if __name__=='__main__':
import billiard
billiard.forking_enable(0)
p = billiard.Process( target=f)
p.start()
while p.is_alive():
pass
The script behaves as expected, printing "ok!" and ending. But if I omit the if __name__=='__main__': line and de-indent the following lines, my machine (OS X) goes crazy, continually spawning tons of Python processes until I killall Python. Any idea what's going on here?
(To those marking this as a duplicate, note that while the other question asks the purpose of if __name__=='__main__' generally, I'm specifically asking why failure to use it here causes dramatically unexpected behaviour)
You're disabling fork support with the line:
billiard.forking_enable(0)
That means that the library will need to spawn (instead of fork) your child process, and have it re-import the __main__ module to run f, just like Windows does. Without the if __name__ ... guard, re-importing the __main__ module in the children will also mean re-running your code that creates the billiard.Process, which creates an infinite loop.
If you leave fork enabled, the re-import in the child process isn't necessary, so everything works fine with or without the if __name__ ... guard.
I am not a Python programmer, but I read a lot of Python scripts that import sub-packages from whole packages already imported.
For example:
import multiprocessing
from multiprocessing import Process
What exactly is the purpose of importing Process specifically when its entire parent package, multiprocessing, has already been imported?
It makes the namespace/module accessible.
Then you can write :
p = Process(target=f, args=('bob',))
p.start()
p.join()
Otherwise, if you had not used the line from multiprocessing import Process, you would have written:
p = multiprocessing.Process(target=f, args=('bob',))
p.start()
p.join()
The above code works, given you have defined the function f:
def f(name):
print 'hello', name
In 2nd case multiprocessing.Process is aliased to Process
The first line, import multiprocessing, initializes the package and imports the name into the local namespace. This allows access to multiprocess.Process, multiprocess.Pipe, multiprocess.Queue, etc.
The second line, from multiprocessing import Process, simply imports the name multiprocess.Process directly into the current namespace, so that it can be referred to as Process rather than the longer multiprocess.Process. It could be replaced with
Process = multiprocessing.Process
since it only creates a new name for multprocessing.Process.
Python doesn't have a syntax for importing both an entire package/module's name and one or more names from that package/module with a single import statement.
I'm trying to use subprocessing in Python, but I don't understand if it can be used with a function, because in the official Python documentation, the syntax of subprocess.Popen takes arguments, but I don't see anything that can be related to a function. Or is there another way to spawn a new process dedicated to a function without subprocess ?
What you need is multiprocessing not subprocessing
http://docs.python.org/library/multiprocessing.html#the-process-class
from multiprocessing import Process
def my_function(name):
print "My name is %s" % name
if __name__ == '__main__':
p = Process(target=my_function, args=('Yarkee', ))
p.start()
p.join() # this blocks until the process terminates
Take a look at the multiprocessing module. This may have what you're looking for
The python subprocess module can be used to call external programs like this:
import subprocess
subprocess.call(["ls", "-l"])
While multiprocessing, as mentioned by #Yarkee, is bound to a function so its probably what you are looking for.