Develop Reference

How to pass a block of commands to a python subprocess?

I have an external application called gmx which works like so:
gmx trjconv -f test.trr -s test.tpr -o test.xtc -center -fit progressive -ur compact << eof
1
1
0
eof
This commands creates the file test.xtc in the current directory.
I would like to call this from a python subprocess. I tried the following:
p = subprocess.Popen(
['gmx', 'trjconv', '-f test.trr', '-s test.tpr', '-o test.xtc', '-center', '-ur compact', '-fit progressive'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdin.write(b'1\n1\n0\n')
p.stdin.close()
But it fails silently in the sense that no error occurs but nothing happens. I read some posts about passing eof but I could not adapt them to my needs. Would you have any idea about how to do this ?

The problem was in the way my Popen was written. Rewriting it as:
p = subprocess.Popen(
['gmx', 'trjconv', '-f', 'test.trr', '-s', 'test.tpr', '-o', 'test.xtc', '-center', '-ur', 'compact', '-fit', 'progressive'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate(b'1\n1\n0')
did the job.
There were at least two mistakes:
the arguments list must contain the arguments one by one (e.g. not
'-f test.trr' but '-f', 'test.trr')
using communicate when using both stdin and stdout

return output from terminal

I would like to ask if is there a way to return to python a terminated value from terminal? For example I'm using os.system to write this line to ubuntu terminal
cat /sys/devices/virtual/thermal/thermal_zone*/temp
to return a couple of values, and I would like to be able to access this values again in python code, is it possible?

os.system is being deprecated, or at least replaced by subprocess
You can capture output fairly easily with subprocess.run
result = subprocess.run(['cat', '/sys/devices/virtual/thermal/thermal_zone*/temp'], capture_output=True)
print(result.stdout)

Because you have * in command so you have to use shell=True and shell will put full name in place of *
import subprocess
process = subprocess.run('cat /sys/devices/virtual/thermal/thermal_zone*/temp', shell=True, capture_output=True)
print(process.stdout.decode())
Result on my Linux Mint:
66000
47000

I think your looking for something like this: Retrieving the output of subprocess.call()
example:
p = Popen(['ls', '-l'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode

If you're using an older version of Python(<3.5)
from subprocess import check_output
result = check_output(["cat", "/sys/devices/virtual/thermal/thermal_zone*/temp"])
print(result)

Unable to read response when Subprocess shell=False

In below code if execute in windows platform i am getting output
import subprocess
COMMAND = " Application.exe arg1 arg2"
process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=None, shell=True)
while process.poll() is None:
output = process.stdout.readline()
print output,
Output> Some text
But if i use shell=False I am not getting output how to get response in this case .

When you set shell=False, you must provide COMMAND as list containing program name and arguments:
With shell=True:
COMMAND = "Application.exe arg1 arg2"
With shell=False:
COMMAND = ["Application.exe", "arg1", "arg2"]
I would recommend you to avoid using subprocess.Popen with shell=True in any case for security measures and use communicate() instead whenever possible:
>>> import subprocess
>>> COMMAND = " Application.exe arg1 arg2".strip().split()
>>> process = subprocess.Popen(COMMAND, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
>>> output, error = process.communicate()
>>> print output

Subprocess.Popen spits output on screen even with stdout=subprocess.PIPE)

I'm using multiple commands to run:
e.g. cd foo/bar; ../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg
Running with:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
(out, err) = p.communicate()
But this spits output on screen (Python 2.7.5)
And out is empty string.

You have shell=True, so you're basically reading the standard output of the shell spawned, not the standard output of the program you want to run.
I'm guessing you're using shell=True to accommodate the directory changing. Fortunately, subprocess can take care of that for you (by passing a directory via the cwd keyword argument):
import subprocess
import shlex
directory = 'foo/bar'
cmd = '../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg'
p = subprocess.Popen(shlex.split(cmd), cwd=directory, stdout=subprocess.PIPE)
(out, err) = p.communicate()

As per comment I added stderr too and that worked!:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)

Python subprocess readlines()?

So I'm trying to move away from os.popen to subprocess.popen as recommended by the user guide. The only trouble I'm having is I can't seem to find a way of making readlines() work.
So I used to be able to do
list = os.popen('ls -l').readlines()
But I can't do
list = subprocess.Popen(['ls','-l']).readlines()

ls = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
out = ls.stdout.readlines()
or, if you want to read line-by-line (maybe the other process is more intensive than ls):
for ln in ls.stdout:
# whatever

With subprocess.Popen, use communicate to read and write data:
out, err = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE).communicate()
Then you can always split the string from the processes' stdout with splitlines().
out = out.splitlines()

Making a system call that returns the stdout output as a string:
lines = subprocess.check_output(['ls', '-l']).splitlines()

list = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE).communicate()[0].splitlines()
straight from the help(subprocess)

A more detailed way of using subprocess.
# Set the command
command = "ls -l"
# Setup the module object
proc = subprocess.Popen(command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# Communicate the command
stdout_value,stderr_value = proc.communicate()
# Once you have a valid response, split the return output
if stdout_value:
stdout_value = stdout_value.split()