Related
I am trying to write a wrapper script for a command line program (svnadmin verify) that will display a nice progress indicator for the operation. This requires me to be able to see each line of output from the wrapped program as soon as it is output.
I figured that I'd just execute the program using subprocess.Popen, use stdout=PIPE, then read each line as it came in and act on it accordingly. However, when I ran the following code, the output appeared to be buffered somewhere, causing it to appear in two chunks, lines 1 through 332, then 333 through 439 (the last line of output)
from subprocess import Popen, PIPE, STDOUT
p = Popen('svnadmin verify /var/svn/repos/config', stdout = PIPE,
stderr = STDOUT, shell = True)
for line in p.stdout:
print line.replace('\n', '')
After looking at the documentation on subprocess a little, I discovered the bufsize parameter to Popen, so I tried setting bufsize to 1 (buffer each line) and 0 (no buffer), but neither value seemed to change the way the lines were being delivered.
At this point I was starting to grasp for straws, so I wrote the following output loop:
while True:
try:
print p.stdout.next().replace('\n', '')
except StopIteration:
break
but got the same result.
Is it possible to get 'realtime' program output of a program executed using subprocess? Is there some other option in Python that is forward-compatible (not exec*)?
I tried this, and for some reason while the code
for line in p.stdout:
...
buffers aggressively, the variant
while True:
line = p.stdout.readline()
if not line: break
...
does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now "Closed" as of Aug 29, 2018)
By setting the buffer size to 1, you essentially force the process to not buffer the output.
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.stdout.close()
p.wait()
You can direct the subprocess output to the streams directly. Simplified example:
subprocess.run(['ls'], stderr=sys.stderr, stdout=sys.stdout)
You can try this:
import subprocess
import sys
process = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
If you use readline instead of read, there will be some cases where the input message is not printed. Try it with a command the requires an inline input and see for yourself.
In Python 3.x the process might hang because the output is a byte array instead of a string. Make sure you decode it into a string.
Starting from Python 3.6 you can do it using the parameter encoding in Popen Constructor. The complete example:
process = subprocess.Popen(
'my_command',
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
encoding='utf-8',
errors='replace'
)
while True:
realtime_output = process.stdout.readline()
if realtime_output == '' and process.poll() is not None:
break
if realtime_output:
print(realtime_output.strip(), flush=True)
Note that this code redirects stderr to stdout and handles output errors.
Real Time Output Issue resolved:
I encountered a similar issue in Python, while capturing the real time output from C program. I added fflush(stdout); in my C code. It worked for me. Here is the code.
C program:
#include <stdio.h>
void main()
{
int count = 1;
while (1)
{
printf(" Count %d\n", count++);
fflush(stdout);
sleep(1);
}
}
Python program:
#!/usr/bin/python
import os, sys
import subprocess
procExe = subprocess.Popen(".//count", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
while procExe.poll() is None:
line = procExe.stdout.readline()
print("Print:" + line)
Output:
Print: Count 1
Print: Count 2
Print: Count 3
The Streaming subprocess stdin and stdout with asyncio in Python blog post by Kevin McCarthy shows how to do it with asyncio:
import asyncio
from asyncio.subprocess import PIPE
from asyncio import create_subprocess_exec
async def _read_stream(stream, callback):
while True:
line = await stream.readline()
if line:
callback(line)
else:
break
async def run(command):
process = await create_subprocess_exec(
*command, stdout=PIPE, stderr=PIPE
)
await asyncio.wait(
[
_read_stream(
process.stdout,
lambda x: print(
"STDOUT: {}".format(x.decode("UTF8"))
),
),
_read_stream(
process.stderr,
lambda x: print(
"STDERR: {}".format(x.decode("UTF8"))
),
),
]
)
await process.wait()
async def main():
await run("docker build -t my-docker-image:latest .")
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Depending on the use case, you might also want to disable the buffering in the subprocess itself.
If the subprocess will be a Python process, you could do this before the call:
os.environ["PYTHONUNBUFFERED"] = "1"
Or alternatively pass this in the env argument to Popen.
Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:
cmd = ["stdbuf", "-oL"] + cmd
See also here about stdbuf or other options.
(See also here for the same answer.)
Found this "plug-and-play" function here. Worked like a charm!
import subprocess
def myrun(cmd):
"""from
http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html
"""
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.readline()
stdout.append(line)
print line,
if line == '' and p.poll() != None:
break
return ''.join(stdout)
I ran into the same problem awhile back. My solution was to ditch iterating for the read method, which will return immediately even if your subprocess isn't finished executing, etc.
I used this solution to get realtime output on a subprocess. This loop will stop as soon as the process completes leaving out a need for a break statement or possible infinite loop.
sub_process = subprocess.Popen(my_command, close_fds=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while sub_process.poll() is None:
out = sub_process.stdout.read(1)
sys.stdout.write(out)
sys.stdout.flush()
You may use an iterator over each byte in the output of the subprocess. This allows inline update (lines ending with '\r' overwrite previous output line) from the subprocess:
from subprocess import PIPE, Popen
command = ["my_command", "-my_arg"]
# Open pipe to subprocess
subprocess = Popen(command, stdout=PIPE, stderr=PIPE)
# read each byte of subprocess
while subprocess.poll() is None:
for c in iter(lambda: subprocess.stdout.read(1) if subprocess.poll() is None else {}, b''):
c = c.decode('ascii')
sys.stdout.write(c)
sys.stdout.flush()
if subprocess.returncode != 0:
raise Exception("The subprocess did not terminate correctly.")
This is the basic skeleton that I always use for this. It makes it easy to implement timeouts and is able to deal with inevitable hanging processes.
import subprocess
import threading
import Queue
def t_read_stdout(process, queue):
"""Read from stdout"""
for output in iter(process.stdout.readline, b''):
queue.put(output)
return
process = subprocess.Popen(['dir'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
cwd='C:\\',
shell=True)
queue = Queue.Queue()
t_stdout = threading.Thread(target=t_read_stdout, args=(process, queue))
t_stdout.daemon = True
t_stdout.start()
while process.poll() is None or not queue.empty():
try:
output = queue.get(timeout=.5)
except Queue.Empty:
continue
if not output:
continue
print(output),
t_stdout.join()
if you just want to forward the log to console in realtime
Below code will work for both
p = subprocess.Popen(cmd,
shell=True,
cwd=work_dir,
bufsize=1,
stdin=subprocess.PIPE,
stderr=sys.stderr,
stdout=sys.stdout)
Complete solution:
import contextlib
import subprocess
# Unix, Windows and old Macintosh end-of-line
newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
stream = getattr(proc, stream)
with contextlib.closing(stream):
while True:
out = []
last = stream.read(1)
# Don't loop forever
if last == '' and proc.poll() is not None:
break
while last not in newlines:
# Don't loop forever
if last == '' and proc.poll() is not None:
break
out.append(last)
last = stream.read(1)
out = ''.join(out)
yield out
def example():
cmd = ['ls', '-l', '/']
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
# Make all end-of-lines '\n'
universal_newlines=True,
)
for line in unbuffered(proc):
print line
example()
Using pexpect with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app's output is getting buffered by the pipe, therefore you can't get to that output until the buffer fills or the process dies.
Here is what worked for me:
import subprocess
import sys
def run_cmd_print_output_to_console_and_log_to_file(cmd, log_file_path):
make_file_if_not_exist(log_file_path)
logfile = open(log_file_path, 'w')
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell = True)
for line in proc.stdout:
sys.stdout.write(line.decode("utf-8") )
print(line.decode("utf-8").strip(), file=logfile, flush=True)
proc.wait()
logfile.close()
(This solution has been tested with Python 2.7.15)
You just need to sys.stdout.flush() after each line read/write:
while proc.poll() is None:
line = proc.stdout.readline()
sys.stdout.write(line)
# or print(line.strip()), you still need to force the flush.
sys.stdout.flush()
Few answers suggesting python 3.x or pthon 2.x , Below code will work for both.
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout = []
while True:
line = p.stdout.readline()
if not isinstance(line, (str)):
line = line.decode('utf-8')
stdout.append(line)
print (line)
if (line == '' and p.poll() != None):
break
def run_command(command):
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
return rc
Yet another answer! I had the following requirements:
Run some command and print the output to stdout as though the user ran it
Display to the user any prompts from the command. E.g. pip uninstall numpy will prompt with ... Proceed (Y/n)? (which does not end in a newline)
Capture the output (that the user saw) as a string
This worked for me (only tested in Python 3.10 on Windows):
def run(*args: list[str]) -> str:
proc = subprocess.Popen(
*args,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
result = ""
while proc.poll() is None:
output = proc.stdout.read(1)
if output:
sys.stdout.write(output)
sys.stdout.flush()
result += output
return result
These are all great examples, but I've found they either (a) handle partial lines (eg "Are you sure (Y/n):") but are really slow or b) are quick but hang on partial lines.
I've worked on the following which:
provides real-time output for both stdout and stderr to their respective streams
is extremely fast as it works with stream buffering
allows for using timeouts as it never blocks on read()
efficiently saves stdout and stderr independently
handles text encoding (though easily adaptable to binary streams)
works on Python 3.6+
import os
import subprocess
import sys
import selectors
import io
def run_command(command: str) -> (int, str):
proc = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
sel = selectors.DefaultSelector()
for fobj in [ proc.stdout, proc.stderr ]:
os.set_blocking(fobj.fileno(), False)
sel.register(fobj, selectors.EVENT_READ)
out=io.StringIO()
err=io.StringIO()
# loop until all descriptors removed
while len(sel.get_map()) > 0:
events = sel.select()
if len(events) == 0:
# timeout or signal, kill to prevent wait hanging
proc.terminate()
break
for key, _ in events:
# read all available data
buf = key.fileobj.read().decode(errors='ignore')
if buf == '':
sel.unregister(key.fileobj)
elif key.fileobj == proc.stdout:
sys.stdout.write(buf)
sys.stdout.flush()
out.write(buf)
elif key.fileobj == proc.stderr:
sys.stderr.write(buf)
sys.stderr.flush()
err.write(buf)
sel.close()
proc.wait()
if proc.returncode != 0:
return (proc.returncode, err.getvalue())
return (0, out.getvalue())
I didn't include the timeout logic (as the subject is real-time output), but it's simple to add them to select()/wait() and no longer worry about infinite hangs.
I've timed cat '25MB-file' and compared to the .read(1) solutions, it's roughly 300 times faster.
I am running a subprocess using 'Popen'. I need to block till this subprocess finishes and then read its output.
p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, encoding="utf-8")
p.communicate():
output = p.stdout.readline()
print(output)
I get an error that
ValueError: I/O operation on closed file.
How can I read the output after the subprocess finishes, I do not want to use poll() though as the subprocess takes time and I would need to wait for its completion anyway.
This should work:
p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, encoding="utf-8")
output, error = p.communicate()
print(output)
if error:
print('error:', error, file=sys.stderr)
However, subprocess.run() is preferred these days:
p = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("output:", p.stdout)
if proc.stderr:
print("error:", p.stderr, file=sys.stderr)
Use subprocess.check_output. It returns the output of the command.
I am trying to write a wrapper script for a command line program (svnadmin verify) that will display a nice progress indicator for the operation. This requires me to be able to see each line of output from the wrapped program as soon as it is output.
I figured that I'd just execute the program using subprocess.Popen, use stdout=PIPE, then read each line as it came in and act on it accordingly. However, when I ran the following code, the output appeared to be buffered somewhere, causing it to appear in two chunks, lines 1 through 332, then 333 through 439 (the last line of output)
from subprocess import Popen, PIPE, STDOUT
p = Popen('svnadmin verify /var/svn/repos/config', stdout = PIPE,
stderr = STDOUT, shell = True)
for line in p.stdout:
print line.replace('\n', '')
After looking at the documentation on subprocess a little, I discovered the bufsize parameter to Popen, so I tried setting bufsize to 1 (buffer each line) and 0 (no buffer), but neither value seemed to change the way the lines were being delivered.
At this point I was starting to grasp for straws, so I wrote the following output loop:
while True:
try:
print p.stdout.next().replace('\n', '')
except StopIteration:
break
but got the same result.
Is it possible to get 'realtime' program output of a program executed using subprocess? Is there some other option in Python that is forward-compatible (not exec*)?
I tried this, and for some reason while the code
for line in p.stdout:
...
buffers aggressively, the variant
while True:
line = p.stdout.readline()
if not line: break
...
does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now "Closed" as of Aug 29, 2018)
By setting the buffer size to 1, you essentially force the process to not buffer the output.
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.stdout.close()
p.wait()
You can direct the subprocess output to the streams directly. Simplified example:
subprocess.run(['ls'], stderr=sys.stderr, stdout=sys.stdout)
You can try this:
import subprocess
import sys
process = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
If you use readline instead of read, there will be some cases where the input message is not printed. Try it with a command the requires an inline input and see for yourself.
In Python 3.x the process might hang because the output is a byte array instead of a string. Make sure you decode it into a string.
Starting from Python 3.6 you can do it using the parameter encoding in Popen Constructor. The complete example:
process = subprocess.Popen(
'my_command',
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
encoding='utf-8',
errors='replace'
)
while True:
realtime_output = process.stdout.readline()
if realtime_output == '' and process.poll() is not None:
break
if realtime_output:
print(realtime_output.strip(), flush=True)
Note that this code redirects stderr to stdout and handles output errors.
Real Time Output Issue resolved:
I encountered a similar issue in Python, while capturing the real time output from C program. I added fflush(stdout); in my C code. It worked for me. Here is the code.
C program:
#include <stdio.h>
void main()
{
int count = 1;
while (1)
{
printf(" Count %d\n", count++);
fflush(stdout);
sleep(1);
}
}
Python program:
#!/usr/bin/python
import os, sys
import subprocess
procExe = subprocess.Popen(".//count", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
while procExe.poll() is None:
line = procExe.stdout.readline()
print("Print:" + line)
Output:
Print: Count 1
Print: Count 2
Print: Count 3
The Streaming subprocess stdin and stdout with asyncio in Python blog post by Kevin McCarthy shows how to do it with asyncio:
import asyncio
from asyncio.subprocess import PIPE
from asyncio import create_subprocess_exec
async def _read_stream(stream, callback):
while True:
line = await stream.readline()
if line:
callback(line)
else:
break
async def run(command):
process = await create_subprocess_exec(
*command, stdout=PIPE, stderr=PIPE
)
await asyncio.wait(
[
_read_stream(
process.stdout,
lambda x: print(
"STDOUT: {}".format(x.decode("UTF8"))
),
),
_read_stream(
process.stderr,
lambda x: print(
"STDERR: {}".format(x.decode("UTF8"))
),
),
]
)
await process.wait()
async def main():
await run("docker build -t my-docker-image:latest .")
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Depending on the use case, you might also want to disable the buffering in the subprocess itself.
If the subprocess will be a Python process, you could do this before the call:
os.environ["PYTHONUNBUFFERED"] = "1"
Or alternatively pass this in the env argument to Popen.
Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:
cmd = ["stdbuf", "-oL"] + cmd
See also here about stdbuf or other options.
(See also here for the same answer.)
Found this "plug-and-play" function here. Worked like a charm!
import subprocess
def myrun(cmd):
"""from
http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html
"""
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.readline()
stdout.append(line)
print line,
if line == '' and p.poll() != None:
break
return ''.join(stdout)
I ran into the same problem awhile back. My solution was to ditch iterating for the read method, which will return immediately even if your subprocess isn't finished executing, etc.
I used this solution to get realtime output on a subprocess. This loop will stop as soon as the process completes leaving out a need for a break statement or possible infinite loop.
sub_process = subprocess.Popen(my_command, close_fds=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while sub_process.poll() is None:
out = sub_process.stdout.read(1)
sys.stdout.write(out)
sys.stdout.flush()
You may use an iterator over each byte in the output of the subprocess. This allows inline update (lines ending with '\r' overwrite previous output line) from the subprocess:
from subprocess import PIPE, Popen
command = ["my_command", "-my_arg"]
# Open pipe to subprocess
subprocess = Popen(command, stdout=PIPE, stderr=PIPE)
# read each byte of subprocess
while subprocess.poll() is None:
for c in iter(lambda: subprocess.stdout.read(1) if subprocess.poll() is None else {}, b''):
c = c.decode('ascii')
sys.stdout.write(c)
sys.stdout.flush()
if subprocess.returncode != 0:
raise Exception("The subprocess did not terminate correctly.")
This is the basic skeleton that I always use for this. It makes it easy to implement timeouts and is able to deal with inevitable hanging processes.
import subprocess
import threading
import Queue
def t_read_stdout(process, queue):
"""Read from stdout"""
for output in iter(process.stdout.readline, b''):
queue.put(output)
return
process = subprocess.Popen(['dir'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
cwd='C:\\',
shell=True)
queue = Queue.Queue()
t_stdout = threading.Thread(target=t_read_stdout, args=(process, queue))
t_stdout.daemon = True
t_stdout.start()
while process.poll() is None or not queue.empty():
try:
output = queue.get(timeout=.5)
except Queue.Empty:
continue
if not output:
continue
print(output),
t_stdout.join()
if you just want to forward the log to console in realtime
Below code will work for both
p = subprocess.Popen(cmd,
shell=True,
cwd=work_dir,
bufsize=1,
stdin=subprocess.PIPE,
stderr=sys.stderr,
stdout=sys.stdout)
Complete solution:
import contextlib
import subprocess
# Unix, Windows and old Macintosh end-of-line
newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
stream = getattr(proc, stream)
with contextlib.closing(stream):
while True:
out = []
last = stream.read(1)
# Don't loop forever
if last == '' and proc.poll() is not None:
break
while last not in newlines:
# Don't loop forever
if last == '' and proc.poll() is not None:
break
out.append(last)
last = stream.read(1)
out = ''.join(out)
yield out
def example():
cmd = ['ls', '-l', '/']
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
# Make all end-of-lines '\n'
universal_newlines=True,
)
for line in unbuffered(proc):
print line
example()
Using pexpect with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app's output is getting buffered by the pipe, therefore you can't get to that output until the buffer fills or the process dies.
Here is what worked for me:
import subprocess
import sys
def run_cmd_print_output_to_console_and_log_to_file(cmd, log_file_path):
make_file_if_not_exist(log_file_path)
logfile = open(log_file_path, 'w')
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell = True)
for line in proc.stdout:
sys.stdout.write(line.decode("utf-8") )
print(line.decode("utf-8").strip(), file=logfile, flush=True)
proc.wait()
logfile.close()
(This solution has been tested with Python 2.7.15)
You just need to sys.stdout.flush() after each line read/write:
while proc.poll() is None:
line = proc.stdout.readline()
sys.stdout.write(line)
# or print(line.strip()), you still need to force the flush.
sys.stdout.flush()
Few answers suggesting python 3.x or pthon 2.x , Below code will work for both.
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout = []
while True:
line = p.stdout.readline()
if not isinstance(line, (str)):
line = line.decode('utf-8')
stdout.append(line)
print (line)
if (line == '' and p.poll() != None):
break
def run_command(command):
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
return rc
Yet another answer! I had the following requirements:
Run some command and print the output to stdout as though the user ran it
Display to the user any prompts from the command. E.g. pip uninstall numpy will prompt with ... Proceed (Y/n)? (which does not end in a newline)
Capture the output (that the user saw) as a string
This worked for me (only tested in Python 3.10 on Windows):
def run(*args: list[str]) -> str:
proc = subprocess.Popen(
*args,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
result = ""
while proc.poll() is None:
output = proc.stdout.read(1)
if output:
sys.stdout.write(output)
sys.stdout.flush()
result += output
return result
These are all great examples, but I've found they either (a) handle partial lines (eg "Are you sure (Y/n):") but are really slow or b) are quick but hang on partial lines.
I've worked on the following which:
provides real-time output for both stdout and stderr to their respective streams
is extremely fast as it works with stream buffering
allows for using timeouts as it never blocks on read()
efficiently saves stdout and stderr independently
handles text encoding (though easily adaptable to binary streams)
works on Python 3.6+
import os
import subprocess
import sys
import selectors
import io
def run_command(command: str) -> (int, str):
proc = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
sel = selectors.DefaultSelector()
for fobj in [ proc.stdout, proc.stderr ]:
os.set_blocking(fobj.fileno(), False)
sel.register(fobj, selectors.EVENT_READ)
out=io.StringIO()
err=io.StringIO()
# loop until all descriptors removed
while len(sel.get_map()) > 0:
events = sel.select()
if len(events) == 0:
# timeout or signal, kill to prevent wait hanging
proc.terminate()
break
for key, _ in events:
# read all available data
buf = key.fileobj.read().decode(errors='ignore')
if buf == '':
sel.unregister(key.fileobj)
elif key.fileobj == proc.stdout:
sys.stdout.write(buf)
sys.stdout.flush()
out.write(buf)
elif key.fileobj == proc.stderr:
sys.stderr.write(buf)
sys.stderr.flush()
err.write(buf)
sel.close()
proc.wait()
if proc.returncode != 0:
return (proc.returncode, err.getvalue())
return (0, out.getvalue())
I didn't include the timeout logic (as the subject is real-time output), but it's simple to add them to select()/wait() and no longer worry about infinite hangs.
I've timed cat '25MB-file' and compared to the .read(1) solutions, it's roughly 300 times faster.
This is a follow up to this question, but if I want to pass an argument to stdin to subprocess, how can I get the output in real time? This is what I currently have; I also tried replacing Popen with call from the subprocess module and this just leads to the script hanging.
from subprocess import Popen, PIPE, STDOUT
cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT)
subfolders = '\n'.join(['subfolder1','subfolder2'])
output = p.communicate(input=subfolders)[0]
print output
In the former question where I did not have to pass stdin I was suggested to use p.stdout.readline, there there is no room there to pipe anything to stdin.
Addendum: This works for the transfer, but I see the output only at the end and I would like to see the details of the transfer while it's happening.
In order to grab stdout from the subprocess in real time you need to decide exactly what behavior you want; specifically, you need to decide whether you want to deal with the output line-by-line or character-by-character, and whether you want to block while waiting for output or be able to do something else while waiting.
It looks like it will probably suffice for your case to read the output in line-buffered fashion, blocking until each complete line comes in, which means the convenience functions provided by subprocess are good enough:
p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE)
# Grab stdout line by line as it becomes available. This will loop until
# p terminates.
while p.poll() is None:
l = p.stdout.readline() # This blocks until it receives a newline.
print l
# When the subprocess terminates there might be unconsumed output
# that still needs to be processed.
print p.stdout.read()
If you need to write to the stdin of the process, just use another pipe:
p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
# Send input to p.
p.stdin.write("some input\n")
p.stdin.flush()
# Now start grabbing output.
while p.poll() is None:
l = p.stdout.readline()
print l
print p.stdout.read()
Pace the other answer, there's no need to indirect through a file in order to pass input to the subprocess.
something like this I think
from subprocess import Popen, PIPE, STDOUT
p = Popen('c:/python26/python printingTest.py', stdout = PIPE,
stderr = PIPE)
for line in iter(p.stdout.readline, ''):
print line
p.stdout.close()
using an iterator will return live results basically ..
in order to send input to stdin you would need something like
other_input = "some extra input stuff"
with open("to_input.txt","w") as f:
f.write(other_input)
p = Popen('c:/python26/python printingTest.py < some_input_redirection_thing',
stdin = open("to_input.txt"),
stdout = PIPE,
stderr = PIPE)
this would be similar to the linux shell command of
%prompt%> some_file.o < cat to_input.txt
see alps answer for better passing to stdin
If you pass all your input before starting reading the output and if by "real-time" you mean whenever the subprocess flushes its stdout buffer:
from subprocess import Popen, PIPE, STDOUT
cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT, bufsize=1)
subfolders = '\n'.join(['subfolder1','subfolder2'])
p.stdin.write(subfolders)
p.stdin.close() # eof
for line in iter(p.stdout.readline, ''):
print line, # do something with the output here
p.stdout.close()
rc = p.wait()
I am trying to write a wrapper script for a command line program (svnadmin verify) that will display a nice progress indicator for the operation. This requires me to be able to see each line of output from the wrapped program as soon as it is output.
I figured that I'd just execute the program using subprocess.Popen, use stdout=PIPE, then read each line as it came in and act on it accordingly. However, when I ran the following code, the output appeared to be buffered somewhere, causing it to appear in two chunks, lines 1 through 332, then 333 through 439 (the last line of output)
from subprocess import Popen, PIPE, STDOUT
p = Popen('svnadmin verify /var/svn/repos/config', stdout = PIPE,
stderr = STDOUT, shell = True)
for line in p.stdout:
print line.replace('\n', '')
After looking at the documentation on subprocess a little, I discovered the bufsize parameter to Popen, so I tried setting bufsize to 1 (buffer each line) and 0 (no buffer), but neither value seemed to change the way the lines were being delivered.
At this point I was starting to grasp for straws, so I wrote the following output loop:
while True:
try:
print p.stdout.next().replace('\n', '')
except StopIteration:
break
but got the same result.
Is it possible to get 'realtime' program output of a program executed using subprocess? Is there some other option in Python that is forward-compatible (not exec*)?
I tried this, and for some reason while the code
for line in p.stdout:
...
buffers aggressively, the variant
while True:
line = p.stdout.readline()
if not line: break
...
does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now "Closed" as of Aug 29, 2018)
By setting the buffer size to 1, you essentially force the process to not buffer the output.
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.stdout.close()
p.wait()
You can direct the subprocess output to the streams directly. Simplified example:
subprocess.run(['ls'], stderr=sys.stderr, stdout=sys.stdout)
You can try this:
import subprocess
import sys
process = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
If you use readline instead of read, there will be some cases where the input message is not printed. Try it with a command the requires an inline input and see for yourself.
In Python 3.x the process might hang because the output is a byte array instead of a string. Make sure you decode it into a string.
Starting from Python 3.6 you can do it using the parameter encoding in Popen Constructor. The complete example:
process = subprocess.Popen(
'my_command',
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
encoding='utf-8',
errors='replace'
)
while True:
realtime_output = process.stdout.readline()
if realtime_output == '' and process.poll() is not None:
break
if realtime_output:
print(realtime_output.strip(), flush=True)
Note that this code redirects stderr to stdout and handles output errors.
Real Time Output Issue resolved:
I encountered a similar issue in Python, while capturing the real time output from C program. I added fflush(stdout); in my C code. It worked for me. Here is the code.
C program:
#include <stdio.h>
void main()
{
int count = 1;
while (1)
{
printf(" Count %d\n", count++);
fflush(stdout);
sleep(1);
}
}
Python program:
#!/usr/bin/python
import os, sys
import subprocess
procExe = subprocess.Popen(".//count", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
while procExe.poll() is None:
line = procExe.stdout.readline()
print("Print:" + line)
Output:
Print: Count 1
Print: Count 2
Print: Count 3
The Streaming subprocess stdin and stdout with asyncio in Python blog post by Kevin McCarthy shows how to do it with asyncio:
import asyncio
from asyncio.subprocess import PIPE
from asyncio import create_subprocess_exec
async def _read_stream(stream, callback):
while True:
line = await stream.readline()
if line:
callback(line)
else:
break
async def run(command):
process = await create_subprocess_exec(
*command, stdout=PIPE, stderr=PIPE
)
await asyncio.wait(
[
_read_stream(
process.stdout,
lambda x: print(
"STDOUT: {}".format(x.decode("UTF8"))
),
),
_read_stream(
process.stderr,
lambda x: print(
"STDERR: {}".format(x.decode("UTF8"))
),
),
]
)
await process.wait()
async def main():
await run("docker build -t my-docker-image:latest .")
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Depending on the use case, you might also want to disable the buffering in the subprocess itself.
If the subprocess will be a Python process, you could do this before the call:
os.environ["PYTHONUNBUFFERED"] = "1"
Or alternatively pass this in the env argument to Popen.
Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:
cmd = ["stdbuf", "-oL"] + cmd
See also here about stdbuf or other options.
(See also here for the same answer.)
Found this "plug-and-play" function here. Worked like a charm!
import subprocess
def myrun(cmd):
"""from
http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html
"""
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.readline()
stdout.append(line)
print line,
if line == '' and p.poll() != None:
break
return ''.join(stdout)
I ran into the same problem awhile back. My solution was to ditch iterating for the read method, which will return immediately even if your subprocess isn't finished executing, etc.
I used this solution to get realtime output on a subprocess. This loop will stop as soon as the process completes leaving out a need for a break statement or possible infinite loop.
sub_process = subprocess.Popen(my_command, close_fds=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while sub_process.poll() is None:
out = sub_process.stdout.read(1)
sys.stdout.write(out)
sys.stdout.flush()
You may use an iterator over each byte in the output of the subprocess. This allows inline update (lines ending with '\r' overwrite previous output line) from the subprocess:
from subprocess import PIPE, Popen
command = ["my_command", "-my_arg"]
# Open pipe to subprocess
subprocess = Popen(command, stdout=PIPE, stderr=PIPE)
# read each byte of subprocess
while subprocess.poll() is None:
for c in iter(lambda: subprocess.stdout.read(1) if subprocess.poll() is None else {}, b''):
c = c.decode('ascii')
sys.stdout.write(c)
sys.stdout.flush()
if subprocess.returncode != 0:
raise Exception("The subprocess did not terminate correctly.")
This is the basic skeleton that I always use for this. It makes it easy to implement timeouts and is able to deal with inevitable hanging processes.
import subprocess
import threading
import Queue
def t_read_stdout(process, queue):
"""Read from stdout"""
for output in iter(process.stdout.readline, b''):
queue.put(output)
return
process = subprocess.Popen(['dir'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
cwd='C:\\',
shell=True)
queue = Queue.Queue()
t_stdout = threading.Thread(target=t_read_stdout, args=(process, queue))
t_stdout.daemon = True
t_stdout.start()
while process.poll() is None or not queue.empty():
try:
output = queue.get(timeout=.5)
except Queue.Empty:
continue
if not output:
continue
print(output),
t_stdout.join()
if you just want to forward the log to console in realtime
Below code will work for both
p = subprocess.Popen(cmd,
shell=True,
cwd=work_dir,
bufsize=1,
stdin=subprocess.PIPE,
stderr=sys.stderr,
stdout=sys.stdout)
Complete solution:
import contextlib
import subprocess
# Unix, Windows and old Macintosh end-of-line
newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
stream = getattr(proc, stream)
with contextlib.closing(stream):
while True:
out = []
last = stream.read(1)
# Don't loop forever
if last == '' and proc.poll() is not None:
break
while last not in newlines:
# Don't loop forever
if last == '' and proc.poll() is not None:
break
out.append(last)
last = stream.read(1)
out = ''.join(out)
yield out
def example():
cmd = ['ls', '-l', '/']
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
# Make all end-of-lines '\n'
universal_newlines=True,
)
for line in unbuffered(proc):
print line
example()
Using pexpect with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app's output is getting buffered by the pipe, therefore you can't get to that output until the buffer fills or the process dies.
Here is what worked for me:
import subprocess
import sys
def run_cmd_print_output_to_console_and_log_to_file(cmd, log_file_path):
make_file_if_not_exist(log_file_path)
logfile = open(log_file_path, 'w')
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell = True)
for line in proc.stdout:
sys.stdout.write(line.decode("utf-8") )
print(line.decode("utf-8").strip(), file=logfile, flush=True)
proc.wait()
logfile.close()
(This solution has been tested with Python 2.7.15)
You just need to sys.stdout.flush() after each line read/write:
while proc.poll() is None:
line = proc.stdout.readline()
sys.stdout.write(line)
# or print(line.strip()), you still need to force the flush.
sys.stdout.flush()
Few answers suggesting python 3.x or pthon 2.x , Below code will work for both.
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout = []
while True:
line = p.stdout.readline()
if not isinstance(line, (str)):
line = line.decode('utf-8')
stdout.append(line)
print (line)
if (line == '' and p.poll() != None):
break
def run_command(command):
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
return rc
Yet another answer! I had the following requirements:
Run some command and print the output to stdout as though the user ran it
Display to the user any prompts from the command. E.g. pip uninstall numpy will prompt with ... Proceed (Y/n)? (which does not end in a newline)
Capture the output (that the user saw) as a string
This worked for me (only tested in Python 3.10 on Windows):
def run(*args: list[str]) -> str:
proc = subprocess.Popen(
*args,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
result = ""
while proc.poll() is None:
output = proc.stdout.read(1)
if output:
sys.stdout.write(output)
sys.stdout.flush()
result += output
return result
These are all great examples, but I've found they either (a) handle partial lines (eg "Are you sure (Y/n):") but are really slow or b) are quick but hang on partial lines.
I've worked on the following which:
provides real-time output for both stdout and stderr to their respective streams
is extremely fast as it works with stream buffering
allows for using timeouts as it never blocks on read()
efficiently saves stdout and stderr independently
handles text encoding (though easily adaptable to binary streams)
works on Python 3.6+
import os
import subprocess
import sys
import selectors
import io
def run_command(command: str) -> (int, str):
proc = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
sel = selectors.DefaultSelector()
for fobj in [ proc.stdout, proc.stderr ]:
os.set_blocking(fobj.fileno(), False)
sel.register(fobj, selectors.EVENT_READ)
out=io.StringIO()
err=io.StringIO()
# loop until all descriptors removed
while len(sel.get_map()) > 0:
events = sel.select()
if len(events) == 0:
# timeout or signal, kill to prevent wait hanging
proc.terminate()
break
for key, _ in events:
# read all available data
buf = key.fileobj.read().decode(errors='ignore')
if buf == '':
sel.unregister(key.fileobj)
elif key.fileobj == proc.stdout:
sys.stdout.write(buf)
sys.stdout.flush()
out.write(buf)
elif key.fileobj == proc.stderr:
sys.stderr.write(buf)
sys.stderr.flush()
err.write(buf)
sel.close()
proc.wait()
if proc.returncode != 0:
return (proc.returncode, err.getvalue())
return (0, out.getvalue())
I didn't include the timeout logic (as the subject is real-time output), but it's simple to add them to select()/wait() and no longer worry about infinite hangs.
I've timed cat '25MB-file' and compared to the .read(1) solutions, it's roughly 300 times faster.