Develop Reference

Python subprocess get stuck at communicate() call

Context:
I am using python 2.7.5.
I need to run a subprocess from a python script, wait for its termination and get the output.
The subprocess is run around 1000 times.
In order to run my subprocess, I have defined a function:
def run(cmd):
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate()
return (p.returncode, stdout, stderr)
The subprocess to be executed is a bash script and is passed as the cmd parameter of the run() function.
The command and its arguments are given through a list (as expected by Popen()).
Issue:
In the past, it has always worked without any error.
But recently, the python script get always stuck on a subprocess call after having successfully executed a lot of calls. The subprocess in question is not executed at all (the bash script is not even started) and the python script blocks.
After stopping the execution with Ctrl+C, I get the point where it was stuck:
[...]
File "import_debug.py", line 20, in run
(stdout, stderr) = p.communicate()
File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
return self._communicate(input)
File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
ready = poller.poll()
KeyboardInterrupt
I don't understand why I have this issue nor how to solve it.
I have found this SO thread that seems to tackle the same issue or something equivalent (since the output after the keyboard interruption is the same) but there is no answer.
Question: What is happening here ? What am I missing ? How to solve this issue ?
EDIT:
The call is under the form:
(code, out, err) = run(["/path/to/bash_script.sh", "arg1", "arg2", "arg3"])
print out
if code:
print "Failed: " + str(err)
The bash script is doing some basic processing with the data (unzip archives and do something with the extracted data).
When the error occurs, none of the bash script instructions are executed.
I cannot provide the exact command, arguments and contents for company privacy concerns.

The author of the original thread you're referring to says: "If I set stderr=None instead of stderr=subprocess.PIPE I never see this issue." -- I'd recommend to do exactly that and get your script working.
Added after reading the comment section:
There are a few useful options, you may want or not to use:
-f freshen existing files, create none
-n never overwrite existing files
-o overwrite files WITHOUT prompting

Python subprocess package returns broken pipe

I am trying to do a very simple example of using subprocess package. The python script should open a new process and run read command. read command should receive input from stdin via PIPE. Every time when I try to use write() and flush() it says:
Traceback (most recent call last):
File "recorder.py", line 68, in <module>
p.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe
My python code looks like:
import subprocess
import time
p = subprocess.Popen(
[
"read",
],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
bufsize=1
)
for character in "This is the message!\n":
p.stdin.write(character.encode("utf-8"))
time.sleep(0.25)
p.stdin.flush()
assert p.returncode == 0
Note: it's very important to send character after character (with sleeping timeout).

I actually could not replicate your result*, in my case your loop runs through and it'd fail on the assert as p has not finished yet and has no returncode (or rather its value is still None at that time). Inserting p.wait() after the loop and before the assert would force we only check for result after p has terminated.
Now for the exception you're seeing, it most likely indicates the pipe you're trying to perform flush() on is closed. Most likely due to the process having already terminated. Perhaps in your case at that point it already has a (non-zero) returncode too which could further help understand the problem?**
* On my system /bin/sh used by subprocess.Popen() with shell=True is actually bash. Running ["/bin/dash", "-c", "read"] which presumably happens to be shell called for /bin/sh on your system, I got broken pipe as well.
** Running dash like this seems to fail with:
/bin/dash: 1: read: arg count
And return 2.
Which sort of makes it more of a dash question: why calling /bin/dash -c "read" (from python) fails. It appears that dash read (unlike its bash counterpart) always expect at least one variable name to read into as an argument (replace read with read foo).
I guess this python question just became a lesson about assumptions and shell scripts portability. :)

Communication with process using pipes in Python

I have a process with which I can communicate on the command line like this:
% process -
input
^D^D
output
So: I start the process, type some input and after hitting Ctrl-D twice, I get the output.
I want to make a Python wrapper around this process. I created this:
from subprocess import Popen, PIPE
p = Popen('process -', stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
while True:
input = raw_input('Enter input: ')
p.stdin.write(input)
p.stdin.close()
p.wait()
output = p.stdout.read()
print output
This works the first time, but after that I get:
Traceback (most recent call last):
File "test.py", line 7, in <module>
p.stdin.write(input)
ValueError: I/O operation on closed file
Is there another way to interact with this process without closing the file?

p.wait() will wait until the subprocess has exited prior to returning, so on the second iteration in your script, p has exited already (and has therefore closed p.stdin).

If the proccess you're wrapping ends after the first ouput, the comunication will fail to the second. Due all pipes (stdin and stdout) will be closed. Hence the error:
ValueError: I/O operation on closed file.
Each time you try to send input to the wrapped process, this must be expecting that input and pipes must be opened.
On the other hand is what Thomas said in his answer, p.wait() is not the way to go for repetitive input/output strategy.
You can't use subprocess.Popen.communicate() neither, due it calls subprocess.Popen.wait() internally.
You can try use p.stdin.write and p.stdout.read here you have a good article about the subject: Writing to a python subprocess pipe

To emulate the shell session:
$ process -
input
^D^D
output
In Python, using check_output():
#!/usr/bin/env python3
from subprocess import check_output
out = check_output(['process', '-'], input='input\n', universal_newlines=True)
print(out, end='')
Ctrl+D is recognized as EOF (terminate the input) by a Unix terminal ($ stty -a -- look for eof = ^D and icanon in the output). If you need to type Ctrl+D twice (at the beginning of a line); it might indicate a bug in the process program such as "for line in sys.stdin: doesn't notice EOF the first time" Python bug.

How do you execute a terminal command and then repeatedly check and store the response in Python?

I have a terminal command that is to be run once. Basically, when you execute the command, the program runs continuously until told otherwise. While the program is running, it will keep outputting strings of text into the terminal. I would like to somehow store these strings as a variable so I can check to see if it contains specific keywords.
I tried using os.system, but alas, that does not store the response from the program.
Thanks in advance, and I'm sorry for my naivety

To spawn a subprocess running a command, use subprocess.Popen:
proc = subprocess.Popen(['cat', 'myfile'], stdout = subprocess.PIPE)
By setting stdout to PIPE, you'll make cat output text through a channel that you control.
When Popen returns, proc.stdout will be a file-like object that you can read(). It also exposes readline(), for convenience.
readline() returns the next line of the stream, \n at the end included. When it returns '', there's nothing left to read.
line = proc.stdout.readline()
while line != '':
print line
You can also create a line iterator to walk the output line by line. You can give this iterator to other functions, that can then consume the output of your subprocess without interacting with the process object directly:
iterator = iter(proc.stdout.readline, '')
# Somewhere else in your program...
for line in iterator:
print line

Communicate multiple times with a process without breaking the pipe?

It's not the first time I'm having this problem, and it's really bugging me.
Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached
proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;\n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
The problem here is that the second time, Python isn't happy. Indeed, he decided to close the file after the first communicate:
Traceback (most recent call last):
File "a.py", line 30, in <module>
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
Are multiple communications allowed?

I think you misunderstand communicate...
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
communicate sends a string to the other process and then waits on it to finish... (Like you said waits for the EOF listening to the stdout & stderror)
What you should do instead is:
proc.stdin.write('message')
# ...figure out how long or why you need to wait...
proc.stdin.write('message2')
(and if you need to get the stdout or stderr you'd use proc.stdout or proc.stderr)

I've had this problem before, and as far as I could ever figure, you couldn't do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).

You can use:
proc.stdin.write('input')
if proc.stdout.closed:
print(proc.stdout)

You can do this simply with single call of communicate():
query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}\n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])
The key-point here is that you only write once to stdin, and \n serve as EOL.
your psql subprocess reads from stdin until \n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.