Develop Reference

Got different result when using subprocess.popen and subprocess.run

When I execute below program, it list file correctly.
import subprocess
foo = subprocess.run("ls /home/my_home",
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.decode("utf-8")
But when execute below program, there is nothing in stdout.
import subprocess
foo = subprocess.Popen(["ls /home/my_home"],
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.read().decode("utf-8")
I wonder is there anything wrong with my second part program?
Thankyou in advance!

From python docs:
"communicate() returns a tuple (stdout_data, stderr_data). The data will be strings if streams were opened in text mode; otherwise, bytes."
Therefore, if you'd like to get output via Popen, you have to unpack the retruned tuple from communicate() like this:
out, err = foo.communicate()
In [150]: out
Out[150]: b''
In [151]: err
Out[151]: b"ls: cannot access '/home/my_home': No such file or directory\n"

I think the bash command and the path should be placed between quotes each when you use brackets like the following
import subprocess foo = subprocess.Popen(["ls", "/home/my_home"], shell=True, executable=/bin/bash, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) my_std_out = foo.stdout.read().decode("utf-8")

Python Pandoc Subprocess Not Printing STDOUT When Exception Occurs

I'm trying to run a process with subprocess and print its entire output if and only if an exception occurs.
Where I was before:
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
text=True,
)
except subprocess.CalledProcessError as error:
print(error.output)
This did not work.
Output when subprocess.CalledProcessError occurs:
b''
Replacing capture_output with stdout=subprocess.PIPE resulted in the output of everything regardless whether an exception occurred or not, error.output was still empty.
So I experimented:
This prints everything I would see if I executed the command in the command-line.
subprocess.run(
command,
stdout=subprocess.PIPE,
)
This prints out nothing.
proc = subprocess.run(
command,
capture_output=True,
)
print(proc.stdout.decode())
I also tried subprocess.check_output() which to my information does the same as subprocess.run() with the flags I set in the first code snippet.
What am I missing here? Thanks.
Addendum
import subprocess
command = ['pandoc', 'file']
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
)
except subprocess.CalledProcessError as error:
print('Exception:')
print(error.output)
This is an MWE with the specific process I want to run (pandoc)
Output
$ pandoc file
pandoc: file: openBinaryFile: does not exist (No such file or directory)
$ ./samplecode.py
Exception:
b''
So the exception gets triggered, but the output object is empty.

It seems that the error message is present in error.stderr and not in error.output. I tried your example (with a ls of non-existent file) :
import subprocess
command = ['ls', 'file']
try:
proc = subprocess.run(
command,
check=True,
capture_output=True,
text=True
)
except subprocess.CalledProcessError as error:
print('Exception:')
print('output : ' + error.output)
print('stderr : ' + error.stderr)
The output is the following :
Exception:
output :
stderr : ls: file: No such file or directory
Hope it helps.

I believe what you're meaning to run is stderr=subprocess.PIPE. This should print the relevant error code to the standard console error output.
Example:
process = subprocess.Popen(['ls', 'myfile.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output,error) = process.communicate()
if error:
print error

How to use subprocess.Popen with built-in command on Windows

In my old python script, I use the following code to show the result for Windows cmd command:
print(os.popen("dir c:\\").read())
As the python 2.7 document said os.popen is obsolete and subprocess is recommended. I follow the documentation as:
result = subprocess.Popen("dir c:\\").stdout
And I got error message:
WindowsError: [Error 2] The system cannot find the file specified
Can you tell me the correct way to use the subprocess module?

You should use call subprocess.Popen with shell=True as below:
import subprocess
result = subprocess.Popen("dir c:", shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,error = result.communicate()
print (output)
More info on subprocess module.

This works in Python 3.7:
from subprocess import Popen, PIPE
args = ["echo", "realtime abc"]
p = Popen(args, stdout=PIPE, stderr=PIPE, shell=True, text=True)
for line in p.stdout:
print("O=:", line)
.
Output:
O=: "realtime abc"

Unable to read response when Subprocess shell=False

In below code if execute in windows platform i am getting output
import subprocess
COMMAND = " Application.exe arg1 arg2"
process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=None, shell=True)
while process.poll() is None:
output = process.stdout.readline()
print output,
Output> Some text
But if i use shell=False I am not getting output how to get response in this case .

When you set shell=False, you must provide COMMAND as list containing program name and arguments:
With shell=True:
COMMAND = "Application.exe arg1 arg2"
With shell=False:
COMMAND = ["Application.exe", "arg1", "arg2"]
I would recommend you to avoid using subprocess.Popen with shell=True in any case for security measures and use communicate() instead whenever possible:
>>> import subprocess
>>> COMMAND = " Application.exe arg1 arg2".strip().split()
>>> process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
>>> output, error = process.communicate()
>>> print output

Subprocess.Popen spits output on screen even with stdout=subprocess.PIPE)

I'm using multiple commands to run:
e.g. cd foo/bar; ../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg
Running with:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
(out, err) = p.communicate()
But this spits output on screen (Python 2.7.5)
And out is empty string.

You have shell=True, so you're basically reading the standard output of the shell spawned, not the standard output of the program you want to run.
I'm guessing you're using shell=True to accommodate the directory changing. Fortunately, subprocess can take care of that for you (by passing a directory via the cwd keyword argument):
import subprocess
import shlex
directory = 'foo/bar'
cmd = '../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg'
p = subprocess.Popen(shlex.split(cmd), cwd=directory, stdout=subprocess.PIPE)
(out, err) = p.communicate()

As per comment I added stderr too and that worked!:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)