I have a simulation program which is piloted though stdin and provides output to stdout
Doing a C++/Qt program for running it in a QProcess works well.
Doing a Python program for running it under linux works well, using:
p = subprocess.Popen(cmd,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
And using p.stdin.write, p.stdout.readline, and p.wait
However, under windows, the program runs and gets the commands through stdin as it should(this has been verified by debugging the subprocess), but the python program deadlocks at any p.stdout.readline, and p.wait. If the stdout=subprocess.PIPE parameter is removed, the program works, the output is displayed on the console and no deadlock occurs.
This sounds familiar with a warning from the Python documentation:
Warning : This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe
such that it blocks waiting for the OS pipe buffer to accept more
data. Use communicate() to avoid that.
However, I can't use communicate(), as the program protocol is not a single command and a single output, rather several commands and replies are required.
Is there any solution?
Unsure of it, but it looks like a buffering problem. On Linux (as on most Unix or Unix-like), output to a file or a pipe is internally buffered at the OS level. That means that after a write call, all the data is buffered but nothing is available at the other end of the pipe until either the internal buffer is full, the data is flushed or the pipe is closed. That's one of the reasons why ptys were invented and are not implemented with a pipe pair.
Said differently, it is not possible to drive a program where you need to use previous output to know what you should give as input with pipes, unless the program has been specially tailored for it by consistently flushing its output before reading anything. It works on a true terminal (tty or pty) because the driver automatically forces a flush of the output before any read on the same device.
But it is not the same dealock that is described in the documentation that you have cited in your question.
Related
I'm using subprocess.Popen to run different scripts from another python script and some of them just hang in some point until they are killed by the scheduler script. I tried setting bufsize to 100 * 1024 * 1024 but didn't succeeded solving the problem. If I ignore all the output setting stdoutand stderr to subprocess.DEVNULL it doesn't get stuck.
Example:
# For most scripts this works, but for long running and very verbose scripts, it gets stuck
subprocess.Popen(args=args_list, bufsize=100*1024*1024, stderr=subprocess.PIPE, stdout=subprocess.DEVNULL)
# This works fine, but I don't get the stderr content
subprocess.Popen(args=args_list, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
Thanks for your help!
Large buffers should only be configured for performance, and never relied on as a correctness measure.
That means that using a large bufsize is not a suitable replacement for ensuring that you can handle reading from a subprocess's stdout and stderr in whichever order contents become available. An operating system isn't guaranteed to support arbitrarily-sized I/O buffers; you should never assume that just because you request a buffer of a given size, you're going to actually receive it.
On Linux, for example, some kernel versions support the F_SETPIPE_SZ ioctl to request a given pipe buffer size. However, unprivileged processes can't actually set to this to any value larger than the fs.pipe-max-size sysctl, so this request may fail; your program needs to be prepared for such a failure.
Thus, use the general best-practices for reading from a subprocess, no matter what your buffer size is.
That means, in something like a rough order of preference:
Using subprocess.communicate() wherever you can.
Combining stdout and stderr into a single file descriptor, if you don't actually need to care about the distinction between them.
Reading from both descriptors, either asynchronously or simultaneously through threading, to ensure that writes that happen will be handled in the order they came in.
As some existing Q&A entries that will help, see:
Python read from subprocess stdout and stderr separately while preserving order
subprocess popen.communicate() vs. stdin.write() and stdout.read()
I've created a subprocess using subprocess.Popen(['myapp'], stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=0) that executes a C-program that writes to stdout using e.g. puts().
The problem is that the Python program blocks in p.stdout.read(1024), although the subprocess starts by puts("HelloWorld"). Only after the subprocess terminates, is output available on p.stdout. I thought that bufsize=0 would mean that pipes become unbuffered, so that output is immediately available on the pipe.
I have read the below question, which states that newlines should cause output to be flushed. However, puts() prints a newline, so are the pipes not recognized as an interactive device?
Difference between puts() and printf() in C while using sleep()
It's because puts is also outputting a newline character which, on
devices that can be determined to be interactive, causes flushing by
default (for standard output) (a).
Any ideas?
This is application behavior. Even if the pipe is unbuffered, applications normally buffer information that they are going to write to a file (any type of file) for some time before actually writing it. As Jon's comment above indicates, a system-call like fflush() can be used by programs to ensure that they actually have posted the data, and, if applicable, that a physical I/O operation has actually completed.
I want to call an external process from python. The process I'm calling reads an input string and gives tokenized result, and waits for another input (binary is MeCab tokenizer if that helps).
I need to tokenize thousands of lines of string by calling this process.
Problem is Popen.communicate() works but waits for the process to die before giving out the STDOUT result. I don't want to keep closing and opening new subprocesses for thousands of times. (And I don't want to send the whole text, it may easily grow over tens of thousands of -long- lines in future.)
from subprocess import PIPE, Popen
with Popen("mecab -O wakati".split(), stdin=PIPE,
stdout=PIPE, stderr=PIPE, close_fds=False,
universal_newlines=True, bufsize=1) as proc:
output, errors = proc.communicate("foobarbaz")
print(output)
I've tried reading proc.stdout.read() instead of using communicate but it is blocked by stdin and doesn't return any results before proc.stdin.close() is called. Which, again means I need to create a new process everytime.
I've tried to implement queues and threads from a similar question as below, but it either doesn't return anything so it's stuck on While True, or when I force stdin buffer to fill by repeteadly sending strings, it outputs all the results at once.
from subprocess import PIPE, Popen
from threading import Thread
from queue import Queue, Empty
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen('mecab -O wakati'.split(), stdout=PIPE, stdin=PIPE,
universal_newlines=True, bufsize=1, close_fds=False)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True
t.start()
p.stdin.write("foobarbaz")
while True:
try:
line = q.get_nowait()
except Empty:
pass
else:
print(line)
break
Also looked at the Pexpect route, but it's windows port doesn't support some important modules (pty based ones), so I couldn't apply that as well.
I know there are a lot of similar answers, and I've tried most of them. But nothing I've tried seems to work on Windows.
EDIT: some info on the binary I'm using, when I use it via command line. It runs and tokenizes sentences I give, until I'm done and forcibly close the program.
(...waits_for_input -> input_recieved -> output -> waits_for_input...)
Thanks.
If mecab uses C FILE streams with default buffering, then piped stdout has a 4 KiB buffer. The idea here is that a program can efficiently use small, arbitrary-sized reads and writes to the buffers, and the underlying standard I/O implementation handles automatically filling and flushing the much-larger buffers. This minimizes the number of required system calls and maximizes throughput. Obviously you don't want this behavior for interactive console or terminal I/O or writing to stderr. In these cases the C runtime uses line-buffering or no buffering.
A program can override this behavior, and some do have command-line options to set the buffer size. For example, Python has the "-u" (unbuffered) option and PYTHONUNBUFFERED environment variable. If mecab doesn't have a similar option, then there isn't a generic workaround on Windows. The C runtime situation is too complicated. A Windows process can link statically or dynamically to one or several CRTs. The situation on Linux is different since a Linux process generally loads a single system CRT (e.g. GNU libc.so.6) into the global symbol table, which allows an LD_PRELOAD library to configure the C FILE streams. Linux stdbuf uses this trick, e.g. stdbuf -o0 mecab -O wakati.
One option to experiment with is to call CreateConsoleScreenBuffer and get a file descriptor for the handle from msvcrt.open_osfhandle. Then pass this as stdout instead of using a pipe. The child process will see this as a TTY and use line buffering instead of full buffering. However managing this is non-trivial. It would involve reading (i.e. ReadConsoleOutputCharacter) a sliding buffer (call GetConsoleScreenBufferInfo to track the cursor position) that's actively written to by another process. This kind of interaction isn't something that I've ever needed or even experimented with. But I have used a console screen buffer non-interactively, i.e. reading the buffer after the child has exited. This allows reading up to 9,999 lines of output from programs that write directly to the console instead of stdout, e.g. programs that call WriteConsole or open "CON" or "CONOUT$".
Here is a workaround for Windows. This should also be adaptable to other operating systems.
Download a console emulator like ConEmu (https://conemu.github.io/)
Start it instead of mecab as your subprocess.
p = Popen(['conemu'] , stdout=PIPE, stdin=PIPE,
universal_newlines=True, bufsize=1, close_fds=False)
Then send the following as the first input:
mecab -O wakafi & exit
You are letting the emulator handle the file output issues for you; the way it normally does when you manually interact with it.
I am still looking into this; but already looks promising...
Only problem is conemu is a gui application; so if no other way to hook into its input and output, one might have to tweak and rebuild from sources (it's open source). I haven't found any other way; but this should work.
I have asked the question about running in some sort of console mode here; so you can check that thread also for something. The author Maximus is on SO...
The code
while True:
try:
line = q.get_nowait()
except Empty:
pass
else:
print(line)
break
is essentially the same as
print(q.get())
except less efficient because it burns CPU time while waiting. The explicit loop won't make data from the subprocess arrive sooner; it arrives when it arrives.
For dealing with uncooperative binaries I have a few suggestions, from best to worst:
Find a Python library and use that instead. It appears that there's an official Python binding in the MeCab source tree and I see some prebuilt packages on PyPI. You can also look for a DLL build that you can call with ctypes or another Python FFI. If that doesn't work...
Find a binary that flushes after each line of output. The most recent Win32 build I found online, v0.98, does flush after each line. Failing that...
Build your own binary that flushes after each line. It should be easy enough to find the main loop and insert a flush call in it. But MeCab seems to explicitly flush already, and git blame says that the flush statement was last changed in 2011, so I'm surprised you ever had this problem and I suspect that there may have just been a bug in your Python code. Failing that...
Process the output asynchronously. If your concern is that you want to deal with the output in parallel with the tokenization for performance reasons, you can mostly do that, after the first 4K. Just do the processing in the second thread instead of stuffing the lines in a queue. If you can't do that...
This is a terrible hack but it may work in some cases: intersperse your inputs with dummy inputs that produce at least 4K of output. For example, you could output 2047 blank lines after every real input line (2047 CRLFs plus the CRLF from the real output = 4K), or a single line of b'A' * 4092 + b'\r\n', whichever is faster.
Not on this list at all is an approach suggested by the two previous answers: directing the output to a Win32 console and scraping the console. This is a terrible idea because scraping gets you cooked output as a rectangular array of characters. The scraper has no way to know whether two lines were originally one overlong line that wrapped. If it guesses wrong, your outputs will get out of sync with your inputs. It's impossible to work around output buffering in this way if you care at all about the integrity of the output.
I guess the answer, if not the solution, can be found here
https://github.com/ikriv/ConsoleProxy/blob/master/src/Tools/Exec/readme.md
I guess, because I had a similar problem, which I worked around, and could not try this route because this tool is not available for Windows 2003, which is the OS I had to use (in a VM for a legacy application).
I'd like to know if I guessed right.
I have been trying to troubleshoot subprocess.PIPE with subprocesses with no luck.
I'm trying to pass commands to an always running process and receive the results without having to close/open the process each time.
Here is the main launching code:
launcher.py:
import subprocess
import time
command = ['python', 'listener.py']
process = subprocess.Popen(
command, bufsize=0,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
# simulates sending a new command every 10 seconds
for x in range(1,10):
process.stdin.write(b'print\r\n')
process.stdin.flush()
time.sleep(10)
listener.py:
import sys
file = open('log.txt', 'w+')
while True:
file.write(sys.stdin.read(1))
file.close()
This is simplified to show relevent pieces. In the end I'll have threads listening on the stdout and stderr but for now I'm trying to troubleshoot the basics.
What I expect to happen: for each loop in launcher.py, the file.write() in listener.py would write.
What happens instead: everything writes when the loop closes and the program terminates, or I SIGTERM / CTRL-C the script.
I'm running this in Windows 8 Python 3.4.
It's almost as if stdin buffers until the process closes and then it passes through. I have buffsize=0 set, and I'm flushing, so that doesn't make sense to me. I thought either one or the other would be sufficient.
The subprocess is running in a different process, so the sleep in launcher should have no impact on the subprocess.
Does anyone have any ideas why this is blocking?
Update 1:
The same behaviour is also seen with the following code run from the console (python.exe stdinreader.py)
That is, when you type into the console while the program is running, nothing is written to the file.
stdinreader.py:
import sys
import os
file = open('log.txt', 'w+b')
while True:
file.write(sys.stdin.read(1))
file.close()
Adding a file.flush() just before file.write() solves this, but that doesn't help me with the subprocess because I don't have control of how subprocess flushes (which would be my return subprocess.PIPE). Maybe if I reinitialize that PIPE with open('wb') it will not buffer. I will try.
Update 2:
I seem to have isolated this problem to the subprocess being called which is not flushing after it's writes to stdout.
Is there anything I can do to force a flush on the stdout PIPE between parent and child without modifying the subprocess? The subprocess is magick.exe (imagemagick 7) running with args ['-script, '-']. From the point of view of the subprocess it has a stdout object of <_io.TextIOWrapper name='' mode='w' encoding='cp1252'>. I guess the subprocess will just open the default stdout objects on initialization and we can't really control whether it buffers or not.
The strange thing is that passing the child the normal sys.stdout object instead of subprocess.PIPE does not require the subprocess to .flush() after write.
Programs run differently depending on whether they are run from the console or through a pipe. If the console (a python process can check with os.stdin.isatty()), stdout data is line buffered and you see data promptly. If a pipe, stdout data is block buffered and you only see data when quite a bit has piled up or the program flushes the pipe.
When you want to grab program output, you have to use a pipe and the program runs in buffered mode. On linux, you can trick programs by creating a fake console (pseudo tty, pty, ...). The pty module, pexpect and others do that.
On windows, I don't know of any way to get it to work. If you control the program being run, have it flush often. Otherwise, glare futilely at the Windows logo. You can even mention the problem on your next blind date if you want it to end early. But I can't think of anything more.
(if somebody knows of a fix, I'd like to hear it. I've seen some code that tries to open a Windows console and screen scrape it, but those solutions keep losing data. It should work if there is a loopback char device out there somewhere).
The problem was that the subprocess being called was not flushing after writing to stdout. Thanks to J.F. and tdelaney for pointing me in the right direction. I have raised this with the developer here: http://www.imagemagick.org/discourse-server/viewtopic.php?f=2&t=26276&p=115545#p115545
There doesn't appear to be a work-around for this in Windows other than to alter the subprocess source. Perhaps if you redirected the output of the subprocess to a NamedTemporaryFile that might work, but I have not tested it and I think it would be locked in Windows so only one of the parent and child could open it at once. Not insurmountable but annoying. There might also be a way to exec the application through unixutils port of stdbuf or something similar as J.F. suggested here: Python C program subprocess hangs at "for line in iter"
If you have access to the source code of the subprocess you're calling you can always recompile it with buffering disabled. It's simple to disable buffering on stdout in C:
setbuf(stdout, NULL)
or set per-line buffering instead of block buffering:
setvbuf(stdout, (char *) NULL, _IOLBF, 0);
See also: Python C program subprocess hangs at "for line in iter"
Hope this helps someone else down the road.
can you try to close the pipe at the end of listener.py? i think that is the issue
I am trying to employ a Subprocess in Python for keeping an external script open in a Server-like fashion. The external script first loads a model. Once this is done, it accepts requests via STDIN and returns processed strings to STDOUT.
So far, I've tried
tokenizer = subprocess.Popen([tokenizer_path, '-l', lang_prefix], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
However, I cannot use
tokenizer.stdin.write(input_string+'\n')
out = self._tokenizer.stdout.readline()
in order to repeatedly process input_strings by means of the subprocess – out will just be empty, no matter if I use stdout.read() or stdout.readline(). However, it works when I close the stdin with tokenizer.stdin.close() before reading STDOUT, but this closes the subprocess, which is not what I want as I would have to reload the whole external script again before sending another request.
Is there any way to use a subprocess in a server-like fashion in python without closing and re-opening it?
Thanks to this Answer, I found out that a slave handle must be used in order to properly communicate with the subprocess:
master, slave = pty.openpty()
tokenizer = subprocess.Popen(script, shell=True stdin=subprocess.PIPE, stdout=slave)
stdin_handle = process.stdin
stdout_handle = os.fdopen(master)
Now, I can communicate to the subprocess without closing it via
stdin_handle.write(input)
stdout_handle.readline() #gets the processed input
Your external script probably buffers its output, so you only can read it in the father when the buffer in the child is flushed (which the child must do itself). One way to make it flush its buffers is probably closing the input because then it terminates in a proper fashion and flushes its buffers in the process.
If you have control over the external program (i. e. if you can patch it), insert a flushing after the output is produced.
Otherwise programs sometimes can be made to not buffer their output by attaching them to a pseudo-TTY (many programs, including the stdlib, assume that when their output is going to a TTY, no buffering is wished). But this is a bit tricky.