Develop Reference

Interacting with subprocess while reading stdout and stderr in real time

I need to write a wrapper around a shell command that prints its stdout and stderr in real time back to stdout and stderr respectively, while also allowing the user to interact and send stdin to it, if the process prompts for input.
Note that pexpect.interact() almost solves the problem, except that it combines all stdout and stderr and sends it back to stdout. And there appears no way to stop it doing that.
What I have so far, is a method to read stdout and stderr from a process via subprocess.Popen:
def _popen_command(
command: List[str]
) -> None:
"""
Run a shell command with Popen line by line
in real time without redirecting stdout or stderr.
"""
with subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
) as proc:
selector = selectors.DefaultSelector()
selector.register(proc.stdout, selectors.EVENT_READ)
selector.register(proc.stderr, selectors.EVENT_READ)
eof = False
while not eof:
for key, _ in selector.select():
data = key.fileobj.read1(1).decode()
if not data:
eof = True
if key.fileobj is proc.stdout:
print(data, end="")
else:
print(data, end="", file=sys.stderr)
I don't believe an answer exists with all of these features exists in Stack Overflow: handles stdout, stderr, in real time, line by line, and prints them back to stdout, stderr, respectively, and also allows arbitrary interaction with stdin.
Is it possible to do this?

Reading stdout from a subprocess in real time

Given this code snippet:
from subprocess import Popen, PIPE, CalledProcessError
def execute(cmd):
with Popen(cmd, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for line in p.stdout:
print(line, end='')
if p.returncode != 0:
raise CalledProcessError(p.returncode, p.args)
base_cmd = [
"cmd", "/c", "d:\\virtual_envs\\py362_32\\Scripts\\activate",
"&&"
]
cmd1 = " ".join(base_cmd + ['python -c "import sys; print(sys.version)"'])
cmd2 = " ".join(base_cmd + ["python -m http.server"])
If I run execute(cmd1) the output will be printed without any problems.
However, If I run execute(cmd2) instead nothing will be printed, why is that and how can I fix it so I could see the http.server's output in real time.
Also, how for line in p.stdout is been evaluated internally? is it some sort of endless loop till reaches stdout eof or something?
This topic has already been addressed few times here in SO but I haven't found a windows solution. The above snippet is code from this answer and I'm running http.server from a virtualenv (python3.6.2-32bits on win7)

If you want to read continuously from a running subprocess, you have to make that process' output unbuffered. Your subprocess being a Python program, this can be done by passing -u to the interpreter:
python -u -m http.server
This is how it looks on a Windows box.

With this code, you can`t see the real-time output because of buffering:
for line in p.stdout:
print(line, end='')
But if you use p.stdout.readline() it should work:
while True:
line = p.stdout.readline()
if not line: break
print(line, end='')
See corresponding python bug discussion for details
UPD: here you can find almost the same problem with various solutions on stackoverflow.

I think the main problem is that http.server somehow is logging the output to stderr, here I have an example with asyncio, reading the data either from stdout or stderr.
My first attempt was to use asyncio, a nice API, which exists in since Python 3.4. Later I found a simpler solution, so you can choose, both of em should work.
asyncio as solution
In the background asyncio is using IOCP - a windows API to async stuff.
# inspired by https://pymotw.com/3/asyncio/subprocesses.html
import asyncio
import sys
import time
if sys.platform == 'win32':
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
async def run_webserver():
buffer = bytearray()
# start the webserver without buffering (-u) and stderr and stdin as the arguments
print('launching process')
proc = await asyncio.create_subprocess_exec(
sys.executable, '-u', '-mhttp.server',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
print('process started {}'.format(proc.pid))
while 1:
# wait either for stderr or stdout and loop over the results
for line in asyncio.as_completed([proc.stderr.readline(), proc.stdout.readline()]):
print('read {!r}'.format(await line))
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(run_df())
finally:
event_loop.close()
redirecting the from stdout
based on your example this is a really simple solution. It just redirects the stderr to stdout and only stdout is read.
from subprocess import Popen, PIPE, CalledProcessError, run, STDOUT import os
def execute(cmd):
with Popen(cmd, stdout=PIPE, stderr=STDOUT, bufsize=1) as p:
while 1:
print('waiting for a line')
print(p.stdout.readline())
cmd2 = ["python", "-u", "-m", "http.server"]
execute(cmd2)

How for line in p.stdout is been evaluated internally? is it some sort of endless loop till reaches stdout eof or something?
p.stdout is a buffer (blocking). When you are reading from an empty buffer, you are blocked until something is written to that buffer. Once something is in it, you get the data and execute the inner part.
Think of how tail -f works on linux: it waits until something is written to the file, and when it does it echo's the new data to the screen. What happens when there is no data? it waits. So when your program gets to this line, it waits for data and process it.
As your code works, but when run as a model not, it has to be related to this somehow. The http.server module probably buffers the output. Try adding -u parameter to Python to run the process as unbuffered:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
Also, you might want to try change your loop to for line in iter(lambda: p.stdout.read(1), ''):, as this reads 1 byte at a time before processing.
Update: The full loop code is
for line in iter(lambda: p.stdout.read(1), ''):
sys.stdout.write(line)
sys.stdout.flush()
Also, you pass your command as a string. Try passing it as a list, with each element in its own slot:
cmd = ['python', '-m', 'http.server', ..]

You could implement the no-buffer behavior at the OS level.
In Linux, you could wrap your existing command line with stdbuf :
stdbuf -i0 -o0 -e0 YOURCOMMAND
Or in Windows, you could wrap your existing command line with winpty:
winpty.exe -Xallow-non-tty -Xplain YOURCOMMAND
I'm not aware of OS-neutral tools for this.

Python's subprocess with "1>&2" and stderr=STDOUT

I have this code from https://pymotw.com/2/subprocess/
I'm not sure how to interpret the code, in the check_output with 1>&2 output is redirected to stderr, but in the parameter, the stderr is back to stdout stderr=subprocess.STDOUT.
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True,
stderr=subprocess.STDOUT,
)
print "*****************"
print 'Have %d bytes in output' % len(output)
print output
Running the code, the print commands are not executed meaning nothing is captured.
What does this code trying to accomplish?
EDIT
From the answer and comment, I could run this code to get
try:
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True, # No such file or directory error without, maybe 1>&2 requires shell=True
stderr=subprocess.STDOUT,
)
except subprocess.CalledProcessError as e:
print "*****************"
print 'Have %d bytes in output' % len(e.output)
print e.output
this output:
*****************
Have 20 bytes in output
to stdout
to stderr
However, when I commented out the stderr=subprocess.STDOUT line, I got instead
to stderr
*****************
Have 10 bytes in output
to stdout
EDIT2
I tested more with stderr library (https://github.com/sickill/stderred) that helps a shell to show characters from stderr in red color.
When I execute this code (comment out the redirection), I can see the to stderr in BLACK color which implies it uses stdout.
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True,
#stderr=subprocess.STDOUT,
)
From this, I guess (correct me if I'm wrong) that Python's check_output method prints out the data into the stderr redirect to stdout so that it prints out the error message into stderr.

The 1 >&2 shell code applies only to the (echo) command it appears on. It is how to tell the shell to direct the output of that echo to the shell's stderr stream.
The python code stderr=subprocess.STDOUT tells the subprocess module that you want the process's stderr stream to be the same file descriptor as its stdout stream so that you will read whatever the process writes to either stream interleaved together in one stream.
The exit 1 in the shell command means that the shell exits with an error (non-zero) status.
The purpose of the code is to demonstrate that the python function subprocess.check_output will check the exit status and raise an exception when it is non-zero.
If the exit code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and output in the output attribute.
Your description of:
Running the code, the print commands are not executed
is a bit misleading since you neglect to mention the output that does occur:
Traceback (most recent call last):
File "t.py", line 6, in <module>
stderr=subprocess.STDOUT,
File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'echo to stdout; echo to stderr 1>&2; exit 1' returned non-zero exit status 1

Capturing *all* terminal output of a program called from Python

I have a program which can be execute as
./install.sh
This install bunch of stuff and has quite a lot of activity happening on screen..
Now, I am trying to execute it via
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
With the hope that all the activity happening on the screen is captured in out (or err). However, content is printed directly to the terminal while the process is running, and not captured into out or err, which are both empty after the process is run.
What could be happening here? How can this content be captured?

In general, what you're doing is already sufficient to channel all output to your variables.
One exception to that is if the program you're running is using /dev/tty to connect directly to its controlling terminal, and emitting output through that terminal rather than through stdout (FD 1) and stderr (FD 2). This is commonly done for security-sensitive IO such as password prompts, but rarely seen otherwise.
As a demonstration that this works, you can copy-and-paste the following into a Python shell exactly as given:
import subprocess
executable = ['/bin/sh', '-c', 'echo stdout; echo stderr >&2']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
print "stderr: ", err
...by contrast, for a demonstration of the case that doesn't work:
import subprocess
executable = ['/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
In this case, content is written to the TTY directly, not to stdout or stderr. This content cannot be captured without using a program (such as script or expect) that provides a fake TTY. So, to use script:
import subprocess
executable = ['script', '-q', '/dev/null',
'/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out

catching stdout in realtime from subprocess

I want to subprocess.Popen() rsync.exe in Windows, and print the stdout in Python.
My code works, but it doesn't catch the progress until a file transfer is done! I want to print the progress for each file in real time.
Using Python 3.1 now since I heard it should be better at handling IO.
import subprocess, time, os, sys
cmd = "rsync.exe -vaz -P source/ dest/"
p, line = True, 'start'
p = subprocess.Popen(cmd,
shell=True,
bufsize=64,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
for line in p.stdout:
print(">>> " + str(line.rstrip()))
p.stdout.flush()

Some rules of thumb for subprocess.
Never use shell=True. It needlessly invokes an extra shell process to call your program.
When calling processes, arguments are passed around as lists. sys.argv in python is a list, and so is argv in C. So you pass a list to Popen to call subprocesses, not a string.
Don't redirect stderr to a PIPE when you're not reading it.
Don't redirect stdin when you're not writing to it.
Example:
import subprocess, time, os, sys
cmd = ["rsync.exe", "-vaz", "-P", "source/" ,"dest/"]
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(">>> " + line.rstrip())
That said, it is probable that rsync buffers its output when it detects that it is connected to a pipe instead of a terminal. This is the default behavior - when connected to a pipe, programs must explicitly flush stdout for realtime results, otherwise standard C library will buffer.
To test for that, try running this instead:
cmd = [sys.executable, 'test_out.py']
and create a test_out.py file with the contents:
import sys
import time
print ("Hello")
sys.stdout.flush()
time.sleep(10)
print ("World")
Executing that subprocess should give you "Hello" and wait 10 seconds before giving "World". If that happens with the python code above and not with rsync, that means rsync itself is buffering output, so you are out of luck.
A solution would be to connect direct to a pty, using something like pexpect.

I know this is an old topic, but there is a solution now. Call the rsync with option --outbuf=L. Example:
cmd=['rsync', '-arzv','--backup','--outbuf=L','source/','dest']
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
print '>>> {}'.format(line.rstrip())

Depending on the use case, you might also want to disable the buffering in the subprocess itself.
If the subprocess will be a Python process, you could do this before the call:
os.environ["PYTHONUNBUFFERED"] = "1"
Or alternatively pass this in the env argument to Popen.
Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:
cmd = ["stdbuf", "-oL"] + cmd
See also here about stdbuf or other options.

On Linux, I had the same problem of getting rid of the buffering. I finally used "stdbuf -o0" (or, unbuffer from expect) to get rid of the PIPE buffering.
proc = Popen(['stdbuf', '-o0'] + cmd, stdout=PIPE, stderr=PIPE)
stdout = proc.stdout
I could then use select.select on stdout.
See also https://unix.stackexchange.com/questions/25372/

for line in p.stdout:
...
always blocks until the next line-feed.
For "real-time" behaviour you have to do something like this:
while True:
inchar = p.stdout.read(1)
if inchar: #neither empty string nor None
print(str(inchar), end='') #or end=None to flush immediately
else:
print('') #flush for implicit line-buffering
break
The while-loop is left when the child process closes its stdout or exits.
read()/read(-1) would block until the child process closed its stdout or exited.

Your problem is:
for line in p.stdout:
print(">>> " + str(line.rstrip()))
p.stdout.flush()
the iterator itself has extra buffering.
Try doing like this:
while True:
line = p.stdout.readline()
if not line:
break
print line

You cannot get stdout to print unbuffered to a pipe (unless you can rewrite the program that prints to stdout), so here is my solution:
Redirect stdout to sterr, which is not buffered. '<cmd> 1>&2' should do it. Open the process as follows: myproc = subprocess.Popen('<cmd> 1>&2', stderr=subprocess.PIPE)
You cannot distinguish from stdout or stderr, but you get all output immediately.
Hope this helps anyone tackling this problem.

To avoid caching of output you might wanna try pexpect,
child = pexpect.spawn(launchcmd,args,timeout=None)
while True:
try:
child.expect('\n')
print(child.before)
except pexpect.EOF:
break
PS : I know this question is pretty old, still providing the solution which worked for me.
PPS: got this answer from another question

p = subprocess.Popen(command,
bufsize=0,
universal_newlines=True)
I am writing a GUI for rsync in python, and have the same probelms. This problem has troubled me for several days until i find this in pyDoc.
If universal_newlines is True, the file objects stdout and stderr are opened as text files in universal newlines mode. Lines may be terminated by any of '\n', the Unix end-of-line convention, '\r', the old Macintosh convention or '\r\n', the Windows convention. All of these external representations are seen as '\n' by the Python program.
It seems that rsync will output '\r' when translate is going on.

if you run something like this in a thread and save the ffmpeg_time property in a property of a method so you can access it, it would work very nice
I get outputs like this:
output be like if you use threading in tkinter
input = 'path/input_file.mp4'
output = 'path/input_file.mp4'
command = "ffmpeg -y -v quiet -stats -i \"" + str(input) + "\" -metadata title=\"#alaa_sanatisharif\" -preset ultrafast -vcodec copy -r 50 -vsync 1 -async 1 \"" + output + "\""
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, shell=True)
for line in self.process.stdout:
reg = re.search('\d\d:\d\d:\d\d', line)
ffmpeg_time = reg.group(0) if reg else ''
print(ffmpeg_time)

Change the stdout from the rsync process to be unbuffered.
p = subprocess.Popen(cmd,
shell=True,
bufsize=0, # 0=unbuffered, 1=line-buffered, else buffer-size
stdin=subprocess.PIPE,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)

I've noticed that there is no mention of using a temporary file as intermediate. The following gets around the buffering issues by outputting to a temporary file and allows you to parse the data coming from rsync without connecting to a pty. I tested the following on a linux box, and the output of rsync tends to differ across platforms, so the regular expressions to parse the output may vary:
import subprocess, time, tempfile, re
pipe_output, file_name = tempfile.TemporaryFile()
cmd = ["rsync", "-vaz", "-P", "/src/" ,"/dest"]
p = subprocess.Popen(cmd, stdout=pipe_output,
stderr=subprocess.STDOUT)
while p.poll() is None:
# p.poll() returns None while the program is still running
# sleep for 1 second
time.sleep(1)
last_line = open(file_name).readlines()
# it's possible that it hasn't output yet, so continue
if len(last_line) == 0: continue
last_line = last_line[-1]
# Matching to "[bytes downloaded] number% [speed] number:number:number"
match_it = re.match(".* ([0-9]*)%.* ([0-9]*:[0-9]*:[0-9]*).*", last_line)
if not match_it: continue
# in this case, the percentage is stored in match_it.group(1),
# time in match_it.group(2). We could do something with it here...

In Python 3, here's a solution, which takes a command off the command line and delivers real-time nicely decoded strings as they are received.
Receiver (receiver.py):
import subprocess
import sys
cmd = sys.argv[1:]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in p.stdout:
print("received: {}".format(line.rstrip().decode("utf-8")))
Example simple program that could generate real-time output (dummy_out.py):
import time
import sys
for i in range(5):
print("hello {}".format(i))
sys.stdout.flush()
time.sleep(1)
Output:
$python receiver.py python dummy_out.py
received: hello 0
received: hello 1
received: hello 2
received: hello 3
received: hello 4