I'm new to the subprocess module and the documentation leaves me wondering what the difference is between subprocess.popen and subprocess.run. Is there a difference in what the command does? Is one just newer? Which is better to use?
subprocess.run() was added in Python 3.5 as a simplification over subprocess.Popen when you just want to execute a command and wait until it finishes, but you don't want to do anything else in the mean time. For other cases, you still need to use subprocess.Popen.
The main difference is that subprocess.run() executes a command and waits for it to finish, while with subprocess.Popen you can continue doing your stuff while the process finishes and then just repeatedly call Popen.communicate() yourself to pass and receive data to your process. Secondly, subprocess.run() returns subprocess.CompletedProcess.
subprocess.run() just wraps Popen and Popen.communicate() so you don't need to make a loop to pass/receive data or wait for the process to finish.
Check the official documentation for info on which params subprocess.run() pass to Popen and communicate().
Both available in Python by default.
The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.
-Subprocess.run:
import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "print('ocean')"])
-Subprocess.popen: run multiple command line with subprocess, communicate method waits for the process to finish and finally prints the stdout and stderr as a tuple
EX:
import subprocess
process = subprocess.Popen(shell_cmd,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
text = True,
shell = True
)
std_out, std_err = process.communicate()
std_out.strip(), std_err
Related
My python script (python 3.4.3) calls a bash script via subprocess.
OutPST = subprocess.check_output(cmd,shell=True)
It works, but the problem is, that I only get half of the data. The subprocess I call, calls a different subprocess and I have the guess, that if the "sub subprocess" sends the EOF, my programm thinks, that that´s it and ends the check_output.
Has someone an idea how to get all the data?
You should use subprocess.run() unless you really need that fine grained of control over talking to the processing via its stdin (or doing something else while the process is running instead of blocking for it to finish). It makes capturing output super easy:
from subprocess import run, PIPE
result = run(cmd, stdout=PIPE, stderr=PIPE)
print(result.stdout)
print(result.stderr)
If you want to merge stdout and stderr (like how you'd see it in your terminal if you didn't do any redirection), you can use the special destination STDOUT for stderr:
from subprocess import STDOUT
result = run(cmd, stdout=PIPE, stderr=STDOUT)
print(result.stdout)
I try to run simple script in windows in the same shell.
When I run
subprocess.call(["python.exe", "a.py"], shell=False)
It works fine.
But when I run
subprocess.Popen(["python.exe", "a.py"], shell=False)
It opens new shell and the shell=false has no affect.
a.py just print message to the screen.
First calling Popen with shell=False doesn't mean that the underlying python won't try to open a window/console. It's just that the current python instance executes python.exe directly and not in a system shell (cmd or sh).
Second, Popen returns a handle on the process, and you have to perform a wait() on this handle for it to end properly or you could generate a defunct process (depending on the platform you're running on). I suggest that you try
p = subprocess.Popen(["python.exe", "a.py"], shell=False)
return_code = p.wait()
to wait for process termination and get return code.
Note that Popen is a very bad way to run processes in background. The best way would be to use a separate thread
import subprocess
import threading
def run_it():
subprocess.call(["python.exe", "a.py"], shell=False)
t = threading.Thread(target=run_it)
t.start()
# do your stuff
# in the end
t.join()
Hi everyone can anyone elaborate on the difference between
subprocess.Popen
subprocess.call
subprocess.check_output
and also if possible then please explain difference between
x.readlines() versus x.communicate()?
i.e difference between
import subprocess
from subprocess import PIPE
ls = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
**out = ls.stdout.readlines()**
print out
and
import subprocess
from subprocess import PIPE
ls = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
out = ls.communicate()
print out
call and check_output (along with check_call) are just utility functions which call Popen under the hood.
call returns the exit code of child process
check_call raises CalledProcessError error if exit code was non zero
check_output same as above but also returns output.
The difference between readlines and communicate is that readlines is simply a function made on the buffer (stdout) while communicate is a method of process class so it can handle different exceptions, you can pass input in it, and it waits for the process to finish.
Read more here
I'm using the following command to run a shell command (creating a subprocess):
cmd = "ls"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
Then, I want to get its return code when it's finished. I should use wait() or poll()? It seems to me wait() is equal to a poll() enclosed in a busy wait. Something like:
while process.poll() == None:
time.sleep(0.5)
I read that wait() could generate a deadlock if stdout/stderr buffer is filled. process.poll() used like above also could generate a deadlock? If this is true,
I should use process.comunicate() to solve the problem? Nowadays, I only use
process.comunicate() when I'm interested in the subprocess stdout/stderr.
Thanks in advance.
Yes. subprocess.poll, when used in a loop like that, will cause a deadlock if the stdout is piped into your process and you aren't reading from it. If you don't pipe stdout or you're consistently reading from it, neither poll nor wait will deadlock. subprocess.communicate will solve the deadlock in the cases it would occur. However, if you just want to run a command, check its return code, and get its output, use subprocess.check_output, which wraps all of that.
When I execute a python script using subprocess.Popen(script, shell=True) in another python script, is it possible to alert python when the script completes running before executing other functions?
On a side note, can I get real-time output of the executed python script?
I can only get output from it doing command>output.txt but that's only after the whole process ends. stdout does not grep any ouput.
When you create a subprocess with Popen, it returns a subprocess.Popen object that has several methods for accessing subprocess status and data:
You can use poll() to determine whether a subprocess has finished. None indicates that the process has ended.
Output from a script while its running can be retrieved with communicate().
You can combine these two to create a script that monitors output from a subprocess and waits until its ready as follows:
import subprocess
p = subprocess.Popen((["python", "script.py"]), stdout=subprocess.PIPE)
while p.poll() is None:
(stdout, stderr) = p.communicate()
print stdout
You want to wait for the Popen to end? have you tried simply this:
popen = subprocess.Popen(script, shell=True)
popen.wait()
Have you considered using the external python script importing it as a module instead of spawning a subprocess?
As for the real-time output: try python -u ...