Develop Reference

How to pass a block of commands to a python subprocess?

I have an external application called gmx which works like so:
gmx trjconv -f test.trr -s test.tpr -o test.xtc -center -fit progressive -ur compact << eof
1
1
0
eof
This commands creates the file test.xtc in the current directory.
I would like to call this from a python subprocess. I tried the following:
p = subprocess.Popen(
['gmx', 'trjconv', '-f test.trr', '-s test.tpr', '-o test.xtc', '-center', '-ur compact', '-fit progressive'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdin.write(b'1\n1\n0\n')
p.stdin.close()
But it fails silently in the sense that no error occurs but nothing happens. I read some posts about passing eof but I could not adapt them to my needs. Would you have any idea about how to do this ?

The problem was in the way my Popen was written. Rewriting it as:
p = subprocess.Popen(
['gmx', 'trjconv', '-f', 'test.trr', '-s', 'test.tpr', '-o', 'test.xtc', '-center', '-ur', 'compact', '-fit', 'progressive'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate(b'1\n1\n0')
did the job.
There were at least two mistakes:
the arguments list must contain the arguments one by one (e.g. not
'-f test.trr' but '-f', 'test.trr')
using communicate when using both stdin and stdout

Got different result when using subprocess.popen and subprocess.run

When I execute below program, it list file correctly.
import subprocess
foo = subprocess.run("ls /home/my_home",
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.decode("utf-8")
But when execute below program, there is nothing in stdout.
import subprocess
foo = subprocess.Popen(["ls /home/my_home"],
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.read().decode("utf-8")
I wonder is there anything wrong with my second part program?
Thankyou in advance!

From python docs:
"communicate() returns a tuple (stdout_data, stderr_data). The data will be strings if streams were opened in text mode; otherwise, bytes."
Therefore, if you'd like to get output via Popen, you have to unpack the retruned tuple from communicate() like this:
out, err = foo.communicate()
In [150]: out
Out[150]: b''
In [151]: err
Out[151]: b"ls: cannot access '/home/my_home': No such file or directory\n"

I think the bash command and the path should be placed between quotes each when you use brackets like the following
import subprocess foo = subprocess.Popen(["ls", "/home/my_home"], shell=True, executable=/bin/bash, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) my_std_out = foo.stdout.read().decode("utf-8")

Unable to read response when Subprocess shell=False

In below code if execute in windows platform i am getting output
import subprocess
COMMAND = " Application.exe arg1 arg2"
process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=None, shell=True)
while process.poll() is None:
output = process.stdout.readline()
print output,
Output> Some text
But if i use shell=False I am not getting output how to get response in this case .

When you set shell=False, you must provide COMMAND as list containing program name and arguments:
With shell=True:
COMMAND = "Application.exe arg1 arg2"
With shell=False:
COMMAND = ["Application.exe", "arg1", "arg2"]
I would recommend you to avoid using subprocess.Popen with shell=True in any case for security measures and use communicate() instead whenever possible:
>>> import subprocess
>>> COMMAND = " Application.exe arg1 arg2".strip().split()
>>> process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
>>> output, error = process.communicate()
>>> print output

Capturing *all* terminal output of a program called from Python

I have a program which can be execute as
./install.sh
This install bunch of stuff and has quite a lot of activity happening on screen..
Now, I am trying to execute it via
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
With the hope that all the activity happening on the screen is captured in out (or err). However, content is printed directly to the terminal while the process is running, and not captured into out or err, which are both empty after the process is run.
What could be happening here? How can this content be captured?

In general, what you're doing is already sufficient to channel all output to your variables.
One exception to that is if the program you're running is using /dev/tty to connect directly to its controlling terminal, and emitting output through that terminal rather than through stdout (FD 1) and stderr (FD 2). This is commonly done for security-sensitive IO such as password prompts, but rarely seen otherwise.
As a demonstration that this works, you can copy-and-paste the following into a Python shell exactly as given:
import subprocess
executable = ['/bin/sh', '-c', 'echo stdout; echo stderr >&2']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
print "stderr: ", err
...by contrast, for a demonstration of the case that doesn't work:
import subprocess
executable = ['/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
In this case, content is written to the TTY directly, not to stdout or stderr. This content cannot be captured without using a program (such as script or expect) that provides a fake TTY. So, to use script:
import subprocess
executable = ['script', '-q', '/dev/null',
'/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out

Trying to call a shell command with Python, gets nothing

I'm trying to run some integration tests on my web app, using casperjs. To do so, I have the following test class :
from django.conf import settings
import unittest
import subprocess
class MyTest(unittest.TestCase):
def test_something(self):
print "begin"
command = "../../n1k0-casperjs-76fc831/bin/casperjs test.js"
p = subprocess.Popen(command, shell=True, bufsize=0, stdout=subprocess.PIPE, universal_newlines=True)
p.wait()
output = p.stdout.read()
p.stdout.close()
print output
print "end"
When I execute this code in a django, or a classif python shell, I get the output of the js script. However, when it's called with django, I get this :
begin
end
Would you guys know why is this happening ?

Is it possible your shell command is actually outputting to stderr?
To redirect stderr to stdout, try this to call the command:
p = subprocess.Popen(command, shell=True, bufsize=0, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)