I am using Python and it's subprocess library to check output from calls using strace, something in the matter of:
subprocess.check_output(["strace", str(processname)])
However, this only gives me the output after the called subprocess already finished, which is very limiting for my use-case.
I need a kind of "stream" or live-output from the process, so I need to read the output while the process is still running instead of only after it finished.
Is there a convenient way to achieve this using the subprocess library?
I'm thinking of a kind of poll every x seconds, but did not find any hints regarding on how to implement this in the documentation.
Many thanks in advance.
As of Python 3.2 (when context manager support was added to Popen), I have found this to be the most straightforward way to continuously stream output from a subprocess:
import subprocess
def run(args):
with subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
for line in process.stdout:
print(line.decode('utf8'))
Had some problems referencing the selected answer for streaming output from a test runner. The following worked better for me:
import subprocess
from time import sleep
def stream_process(process):
go = process.poll() is None
for line in process.stdout:
print(line)
return go
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while stream_process(process):
sleep(0.1)
According to the documentation:
Popen.poll()
Check if child process has terminated. Set and return returncode attribute.
So based on this you can:
process = subprocess.Popen('your_command_here',stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if process.poll() is not None and output == '':
break
if output:
print (output.strip())
retval = process.poll()
This will loop, reading the stdout, and display the output in real time.
This does not work in current versions of python. (At least) for Python 3.8.5 and newer you should replace output == '' with output == b''
Related
Here is my demo code. It contains two scripts.
The first is main.py, it will call print_line.py with subprocess module.
The second is print_line.py, it prints something to the stdout.
main.py
import subprocess
p = subprocess.Popen('python2 print_line.py',
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True,
shell=True,
universal_newlines=True)
while True:
line = p.stdout.readline()
if line:
print(line)
else:
break
print_line.py
from multiprocessing import Process, JoinableQueue, current_process
if __name__ == '__main__':
task_q = JoinableQueue()
def do_task():
while True:
task = task_q.get()
pid = current_process().pid
print 'pid: {}, task: {}'.format(pid, task)
task_q.task_done()
for _ in range(10):
p = Process(target=do_task)
p.daemon = True
p.start()
for i in range(100):
task_q.put(i)
task_q.join()
Before, print_line.py is written with threading and Queue module, everything is fine. But now, after changing to multiprocessing module, the main.py cannot get any output from print_line. I tried to use Popen.communicate() to get the output or set preexec_fn=os.setsid inPopen(). Neither of them work.
So, here is my question:
Why subprocess cannot get the output with multiprocessing? why it is ok with threading?
If I comment out stdout=subprocess.PIPE and stderr=subprocess.PIPE, the output is printed in my console. Why? How does this happen?
Is there any chance to get the output from print_line.py?
Curious.
In theory this should work as it is, but it does not. The reason being somewhere in the deep, murky waters of buffered IO. It seems that the output of a subprocess of a subprocess can get lost if not flushed.
You have two workarounds:
One is to use flush() in your print_line.py:
def do_task():
while True:
task = task_q.get()
pid = current_process().pid
print 'pid: {}, task: {}'.format(pid, task)
sys.stdout.flush()
task_q.task_done()
This will fix the issue as you will flush your stdout as soon as you have written something to it.
Another option is to use -u flag to Python in your main.py:
p = subprocess.Popen('python2 -u print_line.py',
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True,
shell=True,
universal_newlines=True)
-u will force stdin and stdout to be completely unbuffered in print_line.py, and children of print_line.py will then inherit this behaviour.
These are workarounds to the problem. If you are interested in the theory why this happens, it definitely has something to do with unflushed stdout being lost if subprocess terminates, but I am not the expert in this.
It's not a multiprocessing issue, but it is a subprocess issue—or more precisely, it has to to with standard I/O and buffering, as in Hannu's answer. The trick is that by default, the output of any process, whether in Python or not, is line buffered if the output device is a "terminal device" as determined by os.isatty(stream.fileno()):
>>> import sys
>>> sys.stdout.fileno()
1
>>> import os
>>> os.isatty(1)
True
There is a shortcut available to you once the stream is open:
>>> sys.stdout.isatty()
True
but the os.isatty() operation is the more fundamental one. That is, internally, Python inspects the file descriptor first using os.isatty(fd), then chooses the stream's buffering based on the result (and/or arguments and/or the function used to open the stream). The sys.stdout stream is opened early on during Python's startup, before you generally have much control.1
When you call open or codecs.open or otherwise do your own operation to open a file, you can specify the buffering via one of the optional arguments. The default for open is the system default, which is line buffering if isatty(), otherwise fully buffered. Curiously, the default for codecs.open is line buffered.
A line buffered stream gets an automatic flush() applied when you write a newline to it.
An unbuffered stream writes each byte to its output immediately. This is very inefficient in general. A fully buffered stream writes its output when the buffer gets sufficiently full—the definition of "sufficient" here tends to be pretty variable, anything from 1024 (1k) to 1048576 (1 MB)—or when explicitly directed.
When you run something as a process, it's the process itself that decides how to do any buffering. Your own Python code, reading from the process, cannot control it. But if you know something—or a lot—about the processes that you will run, you can set up their environment so that they run line-buffered, or even unbuffered. (Or, as in your case, since you write that code, you can write it to do what you want.)
1There are hooks that fire up very early, where you can fuss with this sort of thing. They are tricky to work though.
I am trying to run one python file from python windows application.For that I have used subprocess.For getting live streaming output on app console I have tried the below statements.
With PIPE
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, shell=True)
for line in iter(p.stdout.readline, ''):
print line
(or)
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
Not only above code ,tried so many methods.Getting same results like below:
1.Python windows app taking so much of time to run
2.Then the app window went to "not responding" state for long time
3.Then whole output is printed on the console
I know that the buffer overflow is happening in python app thats why i am not getting live output.
I posted so many queries for this still not getting solution.
Just now found and tried tempfile for this.But i am not sure this will give live streaming output.
Shall I try this way?
import tempfile
import subprocess
w = tempfile.NamedTemporaryFile()
p = subprocess.Popen(cmd, shell=True, stdout=w,
stderr=subprocess.STDOUT, bufsize=0)
with open(w.name, 'r') as r:
for line in r:
print line
w.close()
Or any other best solutions for non blocking,unbuffering live output on windows app.
Any help would be appreciated.
Note:1.The python file which I want to run has more print statements(ie more content)
2.Windows server 2012,python 2.7
I understand you're frustration. It looks like you've almost come to the answer yourself.
I'm building on the answer from this SO post. But that answer doesn't use TemporaryFile and also I used the tail follow method from here which I have found to offer the fastest output to the terminal with very large volumes of output. This eliminates extraneous calls to print.
Side note: If you've got other asynchronous stuff to do then you can wrap up the code below the imports in a function and use the gevent package and import sleep from gevent and Popen, STDOUT from gevent.subprocess. It is what I'm doing and may help you avoid leftover slowdown (only reason I mention it).
import sys
from tempfile import TemporaryFile
from time import sleep
from subprocess import Popen, STDOUT
# the temp file will be automatically cleaned up using context manager
with TemporaryFile() as output:
sub = Popen(cmd, stdout=output, stderr=STDOUT, shell=True)
# sub.poll returns None until the subprocess ends,
# it will then return the exit code, hopefully 0 ;)
while sub.poll() is None:
where = output.tell()
lines = output.read()
if not lines:
# Adjust the sleep interval to your needs
sleep(0.1)
# make sure pointing to the last place we read
output.seek(where)
else:
sys.__stdout__.write(lines)
sys.__stdout__.flush()
# A last write needed after subprocess ends
sys.__stdout__.write(output.read())
sys.__stdout__.flush()
I need to start a process and read the output of that process while the process is running. I want to be able to print the output (optional) and to return the output when the process has finished. Here is what I have so far (merged from other answers in stackoverflow):
def call(command, print_output):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out = ""
while True:
line = process.stdout.readline().rstrip().decode("utf-8")
if line == '':
break
if print_output:
print(line)
out += line + "\n"
process.wait()
return process.returncode, out
This code works great in windows (tested with windows 7, python 3.3) but fails in linux (Ubuntu 12.04, python 3.2). In linux, the script hangs at the line
line = process.stdout.readline().rstrip().decode("utf-8")
when the process has finished.
What's wrong with the code? I've tried to check whether the process has been finished with process.poll() as well, but that returns always None under Linux.
The docs say
Warning Use communicate() rather than
.stdin.write, .stdout.read or .stderr.read to
avoid deadlocks due to any of the other OS pipe buffers filling
up and blocking the child process.
I know I had issues before on Windows.
I presume the command is running in unbuffered mode somehow.
The docs have recipes for using subprocess your sounds like shell-backquote yet your use of subprocess is different.
I am writing a script in which in the external system command may sometimes require user input. I am not able to handle that properly. I have tried using os.popen4 and subprocess module but could not achieve the desired behavior.
Below mentioned example would show this problem using "cp" command. ("cp" command is used to show this problem, i am calling some different exe which may similarly prompt for user response in some scenarios). In this example there are two files present on disk and when user tries to copy file1 to file2, an conformer message comes up.
proc = subprocess.Popen("cp -i a.txt b.txt", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout_val, stderr_val = proc.communicate()
print stdout_val
b.txt?
proc.communicate("y")
Now in this example if i read only stdout/stderr and prints it, later on if i try to write "y" or "n" based on user's input, i got an error that channel is closed.
Can some one please help me on achieving this behavior in python such that i can print stdout first, then should take user input and write stdin later on.
I found another solution (Threading) from Non-blocking read on a subprocess.PIPE in python , not sure whether it would help. But it appears it is printing question from cp command, i have modified code but not sure on how to write in threading code.
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['cp', '-i', 'a.txt', 'b.txt'],stdin=PIPE, stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.start()
try:
line = q.get_nowait()
except Empty:
print('no output yet')
else:
pass
Popen.communicate will run the subprocess to completion, so you can't call it more than once. You could use the stdin and stdout attributes directly, although that's risky as you could deadlock if the process uses block buffering or the buffers fill up:
stdout_val = proc.stdout.readline()
print stdout_val
proc.stdin.write('y\n')
As there is a risk of deadlock and because this may not work if the process uses block buffering, you would do well to consider using the pexpect package instead.
I don't have a technical answer to this question. More of just a solution. It has something to do with the way the process waits for the input, and once you communicate with the process, a None input is enough to close the process.
For your cp example, what you can do is check the return code immediately with proc.poll(). If the return value is None, you might assume it is trying to wait for input and can ask your user a question. You can then pass the response to the process via proc.communicate(response). It will then pass the value and proceed with the process.
Maybe someone else can chime in with a more technical reason why an initial communicate with a None value closes the process.
I want to be able to start a process and then be able to kill it afterwards
Here's a little python script that starts a process, checks if it is running, waits a while, kills it, waits for it to terminate, then checks again. It uses the 'kill' command. Version 2.6 of python subprocess has a kill function. This was written on 2.5.
import subprocess
import time
proc = subprocess.Popen(["sleep", "60"], shell=False)
print 'poll =', proc.poll(), '("None" means process not terminated yet)'
time.sleep(3)
subprocess.call(["kill", "-9", "%d" % proc.pid])
proc.wait()
print 'poll =', proc.poll()
The timed output shows that it was terminated after about 3 seconds, and not 60 as the call to sleep suggests.
$ time python prockill.py
poll = None ("None" means process not terminated yet)
poll = -9
real 0m3.082s
user 0m0.055s
sys 0m0.029s
Have a look at the subprocess module.
You can also use low-level primitives like fork() via the os module.
http://docs.python.org/library/os.html#process-management
A simple function that uses subprocess module:
def CMD(cmd) :
p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=False)
return (p.stdin, p.stdout, p.stderr)
see docs for primitive fork() and modules subprocess, multiprocessing, multithreading
If you need to interact with the sub process at all, I recommend the pexpect module (link text). You can send input to the process, receive (or "expect") output in return, and you can close the process (with force=True to send SIGKILL).