Develop Reference

Python subprocess readline is blocking even after closing stdout

test.py file
#test.py
#!/usr/bin/env python3
while True:
inp = input("input: ")
print("output: " + inp)
subp.py file:
#subp.py
#!/usr/bin/env python3
from subprocess import Popen, PIPE
cmd = Popen(["python3", "test.py"], stdin=PIPE, stdout=PIPE)
while True:
print("Writing to stdin")
cmd.stdin.write("My Input To PIPE")
print("Done Writing to stdin")
print("Reading from stdout")
s = cmd.stdout.readline()
print("Done reading from stdout") # Never executes this line
The output is the following:
Writing to stdin
Done Writing to stdin
Reading from stdout
I understand that the line s = cmd.stdout.readline() is going to block until EOF is found in the stdout file object.
And if I am right, EOF will never be found in stdout unless stdout gets closed ? Can somebody correct me on this?
Going by my understanding, if I modify test.py
import sys
while True:
inp = input("input: ")
print("output: " + inp)
sys.stdout.close() # Added this line hoping to unblock stdout.readline()
Nothing changes, cmd.stdout.readline() still is looking for EOF even though the stdout file is closed ?
What am I missing? I am more concerned about the theory rather done just making it work without understanding. Thank you for the explanations
Also if modify subp.py and add the line cmd.stdout.close() before the line s = cmd.stdout.readline(), it throws an error saying that I tried reading from a closed file object, which makes sense, but how come it did not throw an error when I close the stdout file object in the test.py file by adding the line sys.stdout.close(). Are these two stdout different things?

control stdin and stdout of a ruby program in python

First I should notice: I'm a python programmer with no knowledge about ruby!
Now, I need to feed stdin of a ruby program and capture stdout of the script with
a python program.
I tried this (forth solution) and the code works in python2.7 but not in python3; The python3 code reads input with no output.
Now, I need a way to tie the ruby program to either python 2 or 3.
My try:
This code written with six module to have cross version compatibility.
python code:
from subprocess import Popen, PIPE as pipe, STDOUT as out
import six
print('launching slave')
slave = Popen(['ruby', 'slave.rb'], stdin=pipe, stdout=pipe, stderr=out)
while True:
if six.PY3:
from sys import stderr
line = input('enter command: ') + '\n'
line = line.encode('ascii')
else:
line = raw_input('entercommand: ') + '\n'
slave.stdin.write(line)
res = []
while True:
if slave.poll() is not None:
print('slave rerminated')
exit()
line = slave.stdout.readline().decode().rstrip()
print('line:', line)
if line == '[exit]': break
res.append(line)
print('results:')
print('\n'.join(res))
ruby code:
while cmd = STDIN.gets
cmd.chop!
if cmd == "exit"
break
else
print eval(cmd), "\n"
print "[exit]\n"
STDOUT.flush
end
end
NOTE:
Either another way to do this stuff is welcomed! (like socket programming, etc.)
Also I think it's a better idea to not using pipe as stdout and use a file-like object. (like tempfile or StringIO or etc.)

It's because of bufsize. In Python 2.x, default value was 0 (unbufffered). And in Python 3.x it changed to -1 (using default buffer size of system).
Specifying it explicitly will solve your problem.
slave = Popen(['ruby', 'slave.rb'], stdin=pipe, stdout=pipe, stderr=out, bufsize=0)
DEMO

Below is the code on how I got it working with Ruby & Python3.
Ruby Slave
# read command from standard input:
while cmd = STDIN.gets
# remove whitespaces:
cmd.chop!
# if command is "exit", terminate:
if cmd == "exit"
break
else
# else evaluate command, send result to standard output:
print eval(cmd), "\n"
print "[exit]\n"
# flush stdout to avoid buffering issues:
STDOUT.flush
end
end
Python master
from subprocess import Popen, PIPE as pipe, STDOUT as out
print('Launching slave')
slave = Popen(['ruby', 'slave.rb'], stdin=pipe, stdout=pipe, stderr=out, bufsize=0)
while True:
from sys import stderr
line = input('Enter command: ') + '\n'
line = line.encode('ascii')
slave.stdin.write(line)
res = []
while True:
if slave.poll() is not None:
print('Slave terminated')
exit()
line = slave.stdout.readline().decode().rstrip()
if line == '[exit]': break
res.append(line)
print('results:')
print('\n'.join(res))

Running an interactive command from within Python

I have a script that I want to run from within Python (2.6.5) that follows the logic below:
Prompts the user for a password. It looks like ("Enter password: ") (*Note: Input does not echo to screen)
Output irrelevant information
Prompt the user for a response ("Blah Blah filename.txt blah blah (Y/N)?: ")
The last prompt line contains text which I need to parse (filename.txt). The response provided doesn't matter (the program could actually exit here without providing one, as long as I can parse the line).
My requirements are somewhat similar to Wrapping an interactive command line application in a Python script, but the responses there seem a bit confusing, and mine still hangs even when the OP mentions that it doesn't for him.
Through looking around, I've come to the conclusion that subprocess is the best way of doing this, but I'm having a few issues. Here is my Popen line:
p = subprocess.Popen("cmd", shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
When I call a read() or readline() on stdout, the prompt is printer to the screen and it hangs.
If I call a write("password\n") for stdin, the prompt is written to the screen and it hangs. The text in write() is not written (I don't the cursor move the a new line).
If I call p.communicate("password\n"), same behavior as write()
I was looking for a few ideas here on the best way to input to stdin and possibly how to parse the last line in the output if your feeling generous, though I could probably figure that out eventually.

If you are communicating with a program that subprocess spawns, you should check out A non-blocking read on a subprocess.PIPE in Python. I had a similar problem with my application and found using queues to be the best way to do ongoing communication with a subprocess.
As for getting values from the user, you can always use the raw_input() builtin to get responses, and for passwords, try using the getpass module to get non-echoing passwords from your user. You can then parse those responses and write them to your subprocess' stdin.
I ended up doing something akin to the following:
import sys
import subprocess
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # Python 3.x
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
def getOutput(outQueue):
outStr = ''
try:
while True: # Adds output from the Queue until it is empty
outStr+=outQueue.get_nowait()
except Empty:
return outStr
p = subprocess.Popen("cmd", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, universal_newlines=True)
outQueue = Queue()
errQueue = Queue()
outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = Thread(target=enqueue_output, args=(p.stderr, errQueue))
outThread.daemon = True
errThread.daemon = True
outThread.start()
errThread.start()
try:
someInput = raw_input("Input: ")
except NameError:
someInput = input("Input: ")
p.stdin.write(someInput)
errors = getOutput(errQueue)
output = getOutput(outQueue)
Once you have the queues made and the threads started, you can loop through getting input from the user, getting errors and output from the process, and processing and displaying them to the user.

Using threading it might be slightly overkill for simple tasks.
Instead os.spawnvpe can be used. It will spawn script shell as a process. You will be able to communicate interactively with the script.
In this example I passed password as an argument, obviously that is not a good idea.
import os
import sys
from getpass import unix_getpass
def cmd(cmd):
cmd = cmd.split()
code = os.spawnvpe(os.P_WAIT, cmd[0], cmd, os.environ)
if code == 127:
sys.stderr.write('{0}: command not found\n'.format(cmd[0]))
return code
password = unix_getpass('Password: ')
cmd_run = './run.sh --password {0}'.format(password)
cmd(cmd_run)
pattern = raw_input('Pattern: ')
lines = []
with open('filename.txt', 'r') as fd:
for line in fd:
if pattern in line:
lines.append(line)
# manipulate lines

If you just want a user to enter a password without it being echoed to the screen just use the standard library's getpass module:
import getpass
print("You entered:", getpass.getpass())
NOTE:The prompt for this function defaults to "Password: " also this will only work on command lines where echoing can be controlled. So if it doesn't work try running it from terminal.

How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out

I want output from execute Test_Pipe.py, I tried following code on Linux but it did not work.
Test_Pipe.py
import time
while True :
print "Someting ..."
time.sleep(.1)
Caller.py
import subprocess as subp
import time
proc = subp.Popen(["python", "Test_Pipe.py"], stdout=subp.PIPE, stdin=subp.PIPE)
while True :
data = proc.stdout.readline() #block / wait
print data
time.sleep(.1)
The line proc.stdout.readline() was blocked, so no data prints out.

You obviously can use subprocess.communicate but I think you are looking for real time input and output.
readline was blocked because the process is probably waiting on your input. You can read character by character to overcome this like the following:
import subprocess
import sys
process = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()

Nadia's snippet does work but calling read with a 1 byte buffer is highly unrecommended. The better way to do this would be to set the stdout file descriptor to nonblocking using fcntl
fcntl.fcntl(
proc.stdout.fileno(),
fcntl.F_SETFL,
fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
)
and then using select to test if the data is ready
while proc.poll() == None:
readx = select.select([proc.stdout.fileno()], [], [])[0]
if readx:
chunk = proc.stdout.read()
print chunk
She was correct in that your problem must be different from what you posted as Caller.py and Test_Pipe.py do work as provided.
https://derrickpetzold.com/p/capturing-output-from-ffmpeg-python/

Test_Pipe.py buffers its stdout by default so proc in Caller.py doesn't see any output until the child's buffer is full (if the buffer size is 8KB then it takes around a minute to fill Test_Pipe.py's stdout buffer).
To make the output unbuffered (line-buffered for text streams) you could pass -u flag to the child Python script. It allows to read subprocess' output line by line in "real-time":
import sys
from subprocess import Popen, PIPE
proc = Popen([sys.executable, "-u", "Test_Pipe.py"], stdout=PIPE, bufsize=1)
for line in iter(proc.stdout.readline, b''):
print line,
proc.communicate()
See links in Python: read streaming input from subprocess.communicate() on how to solve the block-buffering issue for non-Python child processes.

To avoid the many problems that can always arise with buffering for tasks such as "getting the subprocess's output to the main process in real time", I always recommend using pexpect for all non-Windows platform, wexpect on Windows, instead of subprocess, when such tasks are desired.

How do I get 'real-time' information back from a subprocess.Popen in python (2.5)

I'd like to use the subprocess module in the following way:
create a new process that potentially takes a long time to execute.
capture stdout (or stderr, or potentially both, either together or separately)
Process data from the subprocess as it comes in, perhaps firing events on every line received (in wxPython say) or simply printing them out for now.
I've created processes with Popen, but if I use communicate() the data comes at me all at once, once the process has terminated.
If I create a separate thread that does a blocking readline() of myprocess.stdout (using stdout = subprocess.PIPE) I don't get any lines with this method either, until the process terminates. (no matter what I set as bufsize)
Is there a way to deal with this that isn't horrendous, and works well on multiple platforms?

Update with code that appears not to work (on windows anyway)
class ThreadWorker(threading.Thread):
def __init__(self, callable, *args, **kwargs):
super(ThreadWorker, self).__init__()
self.callable = callable
self.args = args
self.kwargs = kwargs
self.setDaemon(True)
def run(self):
try:
self.callable(*self.args, **self.kwargs)
except wx.PyDeadObjectError:
pass
except Exception, e:
print e
if __name__ == "__main__":
import os
from subprocess import Popen, PIPE
def worker(pipe):
while True:
line = pipe.readline()
if line == '': break
else: print line
proc = Popen("python subprocess_test.py", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
stdout_worker = ThreadWorker(worker, proc.stdout)
stderr_worker = ThreadWorker(worker, proc.stderr)
stdout_worker.start()
stderr_worker.start()
while True: pass

stdout will be buffered - so you won't get anything till that buffer is filled, or the subprocess exits.
You can try flushing stdout from the sub-process, or using stderr, or changing stdout on non-buffered mode.

It sounds like the issue might be the use of buffered output by the subprocess - if a relatively small amount of output is created, it could be buffered until the subprocess exits. Some background can be found here:

Here's what worked for me:
cmd = ["./tester_script.bash"]
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
while p.poll() is None:
out = p.stdout.readline()
do_something_with( out, err )
In your case you could try to pass a reference to the sub-process to your Worker Thread, and do the polling inside the thread. I don't know how it will behave when two threads poll (and interact with) the same subprocess, but it may work.
Also note thate the while p.poll() is None: is intended as is. Do not replace it with while not p.poll() as in python 0 (the returncode for successful termination) is also considered False.

I've been running into this problem as well. The problem occurs because you are trying to read stderr as well. If there are no errors, then trying to read from stderr would block.
On Windows, there is no easy way to poll() file descriptors (only Winsock sockets).
So a solution is not to try and read from stderr.

Using pexpect [http://www.noah.org/wiki/Pexpect] with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app's output is getting buffered by the pipe, therefore you can't get to that output until the buffer fills or the process dies.

This seems to be a well-known Python limitation, see
PEP 3145 and maybe others.

Read one character at a time: http://blog.thelinuxkid.com/2013/06/get-python-subprocess-output-without.html
import contextlib
import subprocess
# Unix, Windows and old Macintosh end-of-line
newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
stream = getattr(proc, stream)
with contextlib.closing(stream):
while True:
out = []
last = stream.read(1)
# Don't loop forever
if last == '' and proc.poll() is not None:
break
while last not in newlines:
# Don't loop forever
if last == '' and proc.poll() is not None:
break
out.append(last)
last = stream.read(1)
out = ''.join(out)
yield out
def example():
cmd = ['ls', '-l', '/']
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
# Make all end-of-lines '\n'
universal_newlines=True,
)
for line in unbuffered(proc):
print line
example()

Using subprocess.Popen, I can run the .exe of one of my C# projects and redirect the output to my Python file. I am able now to print() all the information being output to the C# console (using Console.WriteLine()) to the Python console.
Python code:
from subprocess import Popen, PIPE, STDOUT
p = Popen('ConsoleDataImporter.exe', stdout = PIPE, stderr = STDOUT, shell = True)
while True:
line = p.stdout.readline()
print(line)
if not line:
break
This gets the console output of my .NET project line by line as it is created and breaks out of the enclosing while loop upon the project's termination. I'd imagine this would work for two python files as well.

I've used the pexpect module for this, it seems to work ok. http://sourceforge.net/projects/pexpect/