I have a process with which I can communicate on the command line like this:
% process -
input
^D^D
output
So: I start the process, type some input and after hitting Ctrl-D twice, I get the output.
I want to make a Python wrapper around this process. I created this:
from subprocess import Popen, PIPE
p = Popen('process -', stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
while True:
input = raw_input('Enter input: ')
p.stdin.write(input)
p.stdin.close()
p.wait()
output = p.stdout.read()
print output
This works the first time, but after that I get:
Traceback (most recent call last):
File "test.py", line 7, in <module>
p.stdin.write(input)
ValueError: I/O operation on closed file
Is there another way to interact with this process without closing the file?
p.wait() will wait until the subprocess has exited prior to returning, so on the second iteration in your script, p has exited already (and has therefore closed p.stdin).
If the proccess you're wrapping ends after the first ouput, the comunication will fail to the second. Due all pipes (stdin and stdout) will be closed. Hence the error:
ValueError: I/O operation on closed file.
Each time you try to send input to the wrapped process, this must be expecting that input and pipes must be opened.
On the other hand is what Thomas said in his answer, p.wait() is not the way to go for repetitive input/output strategy.
You can't use subprocess.Popen.communicate() neither, due it calls subprocess.Popen.wait() internally.
You can try use p.stdin.write and p.stdout.read here you have a good article about the subject: Writing to a python subprocess pipe
To emulate the shell session:
$ process -
input
^D^D
output
In Python, using check_output():
#!/usr/bin/env python3
from subprocess import check_output
out = check_output(['process', '-'], input='input\n', universal_newlines=True)
print(out, end='')
Ctrl+D is recognized as EOF (terminate the input) by a Unix terminal ($ stty -a -- look for eof = ^D and icanon in the output). If you need to type Ctrl+D twice (at the beginning of a line); it might indicate a bug in the process program such as "for line in sys.stdin: doesn't notice EOF the first time" Python bug.
Related
I am trying to do a very simple example of using subprocess package. The python script should open a new process and run read command. read command should receive input from stdin via PIPE. Every time when I try to use write() and flush() it says:
Traceback (most recent call last):
File "recorder.py", line 68, in <module>
p.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe
My python code looks like:
import subprocess
import time
p = subprocess.Popen(
[
"read",
],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
bufsize=1
)
for character in "This is the message!\n":
p.stdin.write(character.encode("utf-8"))
time.sleep(0.25)
p.stdin.flush()
assert p.returncode == 0
Note: it's very important to send character after character (with sleeping timeout).
I actually could not replicate your result*, in my case your loop runs through and it'd fail on the assert as p has not finished yet and has no returncode (or rather its value is still None at that time). Inserting p.wait() after the loop and before the assert would force we only check for result after p has terminated.
Now for the exception you're seeing, it most likely indicates the pipe you're trying to perform flush() on is closed. Most likely due to the process having already terminated. Perhaps in your case at that point it already has a (non-zero) returncode too which could further help understand the problem?**
* On my system /bin/sh used by subprocess.Popen() with shell=True is actually bash. Running ["/bin/dash", "-c", "read"] which presumably happens to be shell called for /bin/sh on your system, I got broken pipe as well.
** Running dash like this seems to fail with:
/bin/dash: 1: read: arg count
And return 2.
Which sort of makes it more of a dash question: why calling /bin/dash -c "read" (from python) fails. It appears that dash read (unlike its bash counterpart) always expect at least one variable name to read into as an argument (replace read with read foo).
I guess this python question just became a lesson about assumptions and shell scripts portability. :)
I'm trying to send the cmd many commands according to the answers he sends me.
I'm getting a run time error message:
ValueError: I/O operation on closed file
When I'm running something like this:
import subprocess
process = subprocess.Popen("cmd.exe", stdout=subprocess.PIPE,stdin=subprocess.PIPE)
answer = process.communicate(input="some command\n" + '\n')[0]
"""
choosing another command according to answer
"""
print process.communicate(input=another_command + '\n')[0]
process.kill()
Any idea on how to solve the problem?
Thank you for your help!
Do not send your commands to cmd.exe. Call your commands directly like:
subprocess.Popen("dir", shell=True, stdout=subprocess.PIPE,stdin=subprocess.PIPE)
Perhaps you will not need the pipe for stdin if you use it this way.
The error is normal. communicate closes the standard input of the subprocess to indicate that no more input is pending so that the subprocess can flush its output. So you cannot chain multiple communicate calls on one single subprocess.
But if you commands are simple enough (not many kbytes of input data), and if you do not need to collect and process the output of one command before sending the next one, you should be able to write all the commands in sequence, reading as much output as possible between two of them. After last command, you could then close the subprocess standard input and wait for it to terminate, still collating the output:
process = subprocess.Popen("cmd.exe", stdout=subprocess.PIPE, stdin=subprocess.PIPE)
process.stdin.write("some command\n\n")
partial_answer = process.stdout.read() # all or part of the answer can still be buffered subprocess side
...
process.stdin.write("some other command\n\n")
...
# after last command, time to close subprocess
process.stdin.close()
retcode = None
while True:
end_of_answer += process.stdout.read()
if retcode is not None: break
Off the bat, here is what I am importing:
import os, shutil
from subprocess import call, PIPE, STDOUT
I have a line of code that calls bjam to compile a library:
call(['./bjam',
'-j8',
'--prefix="' + tools_dir + '"'],
stdout=PIPE)
I want it to print out text as the compilation occurs. Instead, it prints everything out at the end.
It does not print anything when I run it like this. I have tried running the command outside of Python and determined that all of the output is to stdout (when I did ./bjam -j8 > /dev/null I got no output, and when I ran ./bjam -j8 2> /dev/null I got output).
What am I doing wrong here? I want to print the output from call live.
As a sidenote, I also noticed something when I was outputting the results of a git clone operation:
call(['git',
'clone', 'https://github.com/moses-smt/mosesdecoder.git'],
stdout=PIPE)
prints the stdout text live as the call process is run.
call(['git',
'clone', 'https://github.com/moses-smt/mosesdecoder.git'],
stdout=PIPE, stderr=STDOUT)
does not print out any text. What is going on here?
stdout=PIPE redirects subprocess' stdout to a pipe. Don't do it unless you want to read from the subprocesses stdout in your code using proc.communicate() method or using proc.stdout attribute directly.
If you remove it then subprocess should print to stdout like it does in the shell:
from subprocess import check_call
check_call(['./bjam', '-j8', '--prefix', tools_dir])
I've used check_call() to raise an exception if the child process fails.
See Python: read streaming input from subprocess.communicate() if you want to read subprocess' output line by line (making the line available as a variable in Python) as soon as it is avaiable.
Try:
def run(command):
proc = subprocess.Popen(command, stdout=subprocess.PIPE)
for lineno, line in enumerate(proc.stdout):
try:
print(line.decode('utf-8').replace('\n', ''))
except UnicodeDecodeError:
print('error(%d): cannot decode %s' % (lineno, line))
The try...except logic is for python 3 (maybe 3.2/3.3, I'm not sure), as there line is a byte array not a string. For earlier versions of python, you should be able to do:
def run(command):
proc = subprocess.Popen(command, stdout=subprocess.PIPE)
for line in proc.stdout:
print(line.replace('\n', ''))
Now, you can do:
run(['./bjam', '-j8', '--prefix="' + tools_dir + '"'])
call will not print anything it captures. As documentation says "Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer."
Consider using check_output and print its return value.
In the first case with git call you are not capturing stderr and therefor it normally flows onto your terminal.
I am trying to read from both stdout and stderr from a Popen and print them out. The command I am running with Popen is the following
#!/bin/bash
i=10
while (( i > 0 )); do
sleep 1s
echo heyo-$i
i="$((i-1))"
done
echo 'to error' >&2
When I run this in the shell, I get one line of output and then a second break and then one line again, etc. However, I am unable to recreate this using python. I am starting two threads, one each to read from stdout and stderr, put the lines read into a Queue and another thread that takes items from this queue and prints them out. But with this, I see that all the output gets printed out at once, after the subprocess ends. I want the lines to be printed as and when they are echo'ed.
Here's my python code:
# The `randoms` is in the $PATH
proc = sp.Popen(['randoms'], stdout=sp.PIPE, stderr=sp.PIPE, bufsize=0)
q = Queue()
def stream_watcher(stream, name=None):
"""Take lines from the stream and put them in the q"""
for line in stream:
q.put((name, line))
if not stream.closed:
stream.close()
Thread(target=stream_watcher, args=(proc.stdout, 'out')).start()
Thread(target=stream_watcher, args=(proc.stderr, 'err')).start()
def displayer():
"""Take lines from the q and add them to the display"""
while True:
try:
name, line = q.get(True, 1)
except Empty:
if proc.poll() is not None:
break
else:
# Print line with the trailing newline character
print(name.upper(), '->', line[:-1])
q.task_done()
print('-*- FINISHED -*-')
Thread(target=displayer).start()
Any ideas? What am I missing here?
Only stderr is unbuffered, not stdout. What you want cannot be done using the shell built-ins alone. The buffering behavior is defined in the stdio(3) C library, which applies line buffering only when the output is to a terminal. When the output is to a pipe, it is pipe-buffered, not line-buffered, and so the data is not transferred to the kernel and thence to the other end of the pipe until the pipe buffer fills.
Moreover, the shell has no access to libc’s buffer-controlling functions, such as setbuf(3) and friends. The only possible solution within the shell is to launch your co-process on a pseudo-tty, and pty management is a complex topic. It is much easier to rewrite the equivalent shell script in a language that does grant access to low-level buffering features for output streams than to arrange to run something over a pty.
However, if you call /bin/echo instead of the shell built-in echo, you may find it more to your liking. This works because now the whole line is flushed when the newly launched /bin/echo process terminates each time. This is hardly an efficient use of system resources, but may be an efficient use of your own.
IIRC, setting shell=True on Popen should do it.
It's not the first time I'm having this problem, and it's really bugging me.
Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached
proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;\n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
The problem here is that the second time, Python isn't happy. Indeed, he decided to close the file after the first communicate:
Traceback (most recent call last):
File "a.py", line 30, in <module>
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
Are multiple communications allowed?
I think you misunderstand communicate...
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
communicate sends a string to the other process and then waits on it to finish... (Like you said waits for the EOF listening to the stdout & stderror)
What you should do instead is:
proc.stdin.write('message')
# ...figure out how long or why you need to wait...
proc.stdin.write('message2')
(and if you need to get the stdout or stderr you'd use proc.stdout or proc.stderr)
I've had this problem before, and as far as I could ever figure, you couldn't do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).
You can use:
proc.stdin.write('input')
if proc.stdout.closed:
print(proc.stdout)
You can do this simply with single call of communicate():
query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}\n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])
The key-point here is that you only write once to stdin, and \n serve as EOL.
your psql subprocess reads from stdin until \n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.