Develop Reference

printing stdout pipe of sub processes ignore last lines

I'm trying to run a sub processes and watching his stdout until I find desirable string.
this is my code:
def waitForAppOutput(proc, word):
for stdout_line in iter(proc.stdout.readline, b''):
print stdout_line
if word in stdout_line.rstrip():
return;
p = Popen(["./app.sh"], shell=True, stdin=PIPE ,stdout=PIPE, stderr=PIPE)
waitForAppOutput(p,"done!")
the issue here is that for some reason the function waitForAppOutput stop printing stdout few lines before the "done!" which is the last line that should appear in the stdout . I assume iter(proc.stdout.readline, b'') is blocking and readline is not able to read the last lines of the stdout.
any idea what is the issue here?

You have a misspelling: it should be waitForAppOutput instead of waitForAppOutout. How does this even run at all? And when you are invoking a command using a shell, you should not be passing an array of strings but rather one single string.
Normally one should use the communicate method on the return subprocess object from the Popen call to prevent potential deadlocks (which seems to be what you are experiencing). This returns a tuple: (stdout, stderr), the stdout and stderr output strings:
from subprocess import Popen, PIPE
def waitForAppOutput(stdout_lines, word):
for stdout_line in stdout_lines:
print stdout_line
if word in stdout_line.rstrip():
return;
p = Popen("./app.sh", shell=True, stdout=PIPE, stderr=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = "command line 1\ncommand line 2\n"
stdout, stderr = p.communicate(expected_input)
stdout_lines = stdout.splitlines()
waitForAppOutput(stdout_lines, "done!")
The only issue is if the output strings are large (whatever your definition of large might be), for it might be memory-inefficient or even prohibitive to read the entire output into memory. If this is your situation, then I would try to avoid the deadlock by piping only stdout.
from subprocess import Popen, PIPE
def waitForAppOutput(proc, word):
for stdout_line in iter(proc.stdout.readline, ''):
print stdout_line
if word in stdout_line.rstrip():
return;
p = Popen("./app.sh", shell=True, stdout=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = "command line 1\ncommand line 2\n"
p.stdin.write(expected_input)
p.stdin.close()
waitForAppOutput(p, "done!")
for stdout_line in iter(p.stdout.readline, ''):
pass # read rest of output
p.wait() # wait for termination
Update
Here is an example using both techniques that runs the Windows sort command to sort a bunch of input lines. This works particularly well both ways because the sort command does not start output until all the input has been read, so it's a very simple protocol in which deadlocking is easy to avoid. Try running this with USE_COMMUNICATE set alternately to True and False:
from subprocess import Popen, PIPE
USE_COMMUNICATE = False
p = Popen("sort", shell=True, stdout=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = """q
w
e
r
t
y
u
i
o
p
"""
if USE_COMMUNICATE:
stdout_lines, stderr_lines = p.communicate(expected_input)
output = stdout_lines
else:
p.stdin.write(expected_input)
p.stdin.close()
output = iter(p.stdout.readline, '')
for stdout_line in output:
print stdout_line,
p.wait() # wait for termination
Prints:
e
i
o
p
q
r
t
u
w
y

read from subprocess output python

I am running a subprocess using 'Popen'. I need to block till this subprocess finishes and then read its output.
p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, encoding="utf-8")
p.communicate():
output = p.stdout.readline()
print(output)
I get an error that
ValueError: I/O operation on closed file.
How can I read the output after the subprocess finishes, I do not want to use poll() though as the subprocess takes time and I would need to wait for its completion anyway.

This should work:
p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, encoding="utf-8")
output, error = p.communicate()
print(output)
if error:
print('error:', error, file=sys.stderr)
However, subprocess.run() is preferred these days:
p = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("output:", p.stdout)
if proc.stderr:
print("error:", p.stderr, file=sys.stderr)

Use subprocess.check_output. It returns the output of the command.

python how to use subprocess pipe with linux shell

I have a python script search for logs, it continuously output the logs found and I want to use linux pipe to filter the desired output. example like that:
$python logsearch.py | grep timeout
The problem is the sort and wc are blocked until the logsearch.py finishes, while the logsearch.py will continuous output the result.
sample logsearch.py:
p = subprocess.Popen("ping google.com", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for line in p.stdout:
print(line)
UPDATE:
figured out, just change the stdout in subprocess to sys.stdout, python will handle the pipe for you.
p = subprocess.Popen("ping -c 5 google.com", shell=True, stdout=**sys.stdout**)
Thanks for all of you help!

And why use grep? Why don't do all the stuff in Python?
from subprocess import Popen, PIPE
p = Popen(['ping', 'google.com'], shell=False, stdin=PIPE, stdout=PIPE)
for line in p.stdout:
if 'timeout' in line.split():
# Process the error
print("Timeout error!!")
else:
print(line)
UPDATE:
I change the Popen line as recommended #triplee. Pros and cons in Actual meaning of 'shell=True' in subprocess

Subprocess.Popen spits output on screen even with stdout=subprocess.PIPE)

I'm using multiple commands to run:
e.g. cd foo/bar; ../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg
Running with:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
(out, err) = p.communicate()
But this spits output on screen (Python 2.7.5)
And out is empty string.

You have shell=True, so you're basically reading the standard output of the shell spawned, not the standard output of the program you want to run.
I'm guessing you're using shell=True to accommodate the directory changing. Fortunately, subprocess can take care of that for you (by passing a directory via the cwd keyword argument):
import subprocess
import shlex
directory = 'foo/bar'
cmd = '../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg'
p = subprocess.Popen(shlex.split(cmd), cwd=directory, stdout=subprocess.PIPE)
(out, err) = p.communicate()

As per comment I added stderr too and that worked!:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)

Subprocess arp -a yielding less results than cmd arp -a

The below code produces a lesser ip yield than doing arp -a in cmd
arpA_req = Popen('arp -a', stdin=PIPE, stdout=PIPE, stderr=STDOUT)
line = arpA_req.stdout.readline().decode('ascii').rsplit()
print(line)
Does anyone know why this may be? And if it's a common issue, how can I obtain a fuller ip list?

As wim pointed out, readline() only reads one line.
To read all the output, one way is to call communicate:
import subprocess
PIPE, STDOUT = subprocess.PIPE, subprocess.STDOUT
arpA_req = subprocess.Popen(
['arp', '-a'], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
out, err = arpA_req.communicate()
print(out)
Or, to process one line at a time, a standard idiom is to use iter(func, stop_value):
for line in iter(arpA_req.stdout.readline, ''):
print(line)