I have a very simple Python 3 script:
f1 = open('a.txt', 'r')
print(f1.readlines())
f2 = open('b.txt', 'r')
print(f2.readlines())
f3 = open('c.txt', 'r')
print(f3.readlines())
f4 = open('d.txt', 'r')
print(f4.readlines())
f1.close()
f2.close()
f3.close()
f4.close()
But it always says:
IOError: [Errno 32] Broken pipe
I saw on the internet all the complicated ways to fix this, but I copied this code directly, so I think that there is something wrong with the code and not Python's SIGPIPE.
I am redirecting the output, so if the above script was named "open.py", then my command to run would be:
open.py | othercommand
The problem is due to SIGPIPE handling. You can solve this problem using the following code:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
Update: As pointed out in the comments, python docs already have a good answer.
See here for background on this solution. Better answer here.
To bring information from the many helpful answers together, with some additional information:
Standard Unix signal SIGPIPE is sent to a process writing to a pipe when there's no process reading from the pipe (anymore).
This is not necessarily an error condition; some Unix utilities such as head by design stop reading prematurely from a pipe, once they've received enough data.
Therefore, an easy way to provoke this error is to pipe to head[1]; e.g.:
python -c 'for x in range(10000): print(x)' | head -n 1
By default - i.e., if the writing process does not explicitly trap SIGPIPE - the writing process is simply terminated, and its exit code is set to 141, which is calculated as 128 (to signal termination by signal in general) + 13 (SIGPIPE's specific signal number).
However, by design Python itself traps SIGPIPE and translates it into a Python BrokenPipeError (Python 3) / IOError (Python 2) instance with errno value errno.EPIPE.
Note: If you use a Unix emulation environment on Windows, the error may surface differently - see this answer.
If a Python script does not catch the exception, Python outputs error message BrokenPipeError: [Errno 32] Broken pipe (Python 3, possibly twice, with Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> sandwiched in between) / IOError: [Errno 32] Broken pipe (Python 2) and terminates the script with exit code 1[2] - this is the symptom Johannes (the OP) saw.
Windows considerations (SIGPIPE is a Unix-only signal)
If your script needs to run directly on Windows too, you may have to conditionally bypass code that references SIGPIPE, as shown in this answer.
If your script runs in a Unix subsystem on Windows, the SIGPIPE signal may surface differently than on Unix - see this answer.
There are two ways to solve this problem:
Generally, it is not advisable to silence this exception, as it may signal a severe error condition, depending on your script's purpose, such as the receiving end of a network socket unexpectedly closing.
However, if your script is a command-line utility, where quiet termination may not only be acceptable but preferred so as to play nicely with the standard head utility, for instance, you can abort quietly as follows, using signal.signal() to install the platform's default signal handler (which behaves as described above), as also shown in akhan's answer (works in both Python 3 and 2):
# ONLY SUITABLE FOR COMMAND-LINE UTILITIES
# Install the default signal handler.
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE, SIG_DFL)
# Start printing many lines.
# If this gets interrupted with SIGPIPE,
# the script aborts quietly, and the process exit code is set to
# 141 (128 + SIGPIPE)
for x in range(10000): print(x)
Otherwise, if you want to handle the SIGPIPE-triggered exception yourself (works in both Python 3 and 2, adapted from the docs):
import sys, os, errno
try:
# Start printing many lines.
for x in range(10000): print(x)
# IMPORTANT: Flush stdout here, to ensure that the
# SIGPIPE-triggered exception can be caught.
sys.stdout.flush()
except IOError as e:
# Note: Python 3 has the more specific BrokenPipeError,
# but this way the code works in Python 2 too.
if e.errno != errno.EPIPE: raise e # Unrelated error, re-throw.
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
# ... perform other handling.
# Note: You can't write to stdout here.
# (print() and sys.stdout.write won't work)
# However, sys.stderr.write() can be used.
sys.stderr.write("SIGPIPE received, terminating.\n")
# Finally, exit with an exit code of choice.
sys.exit(141)
[1] Note that in bash you will by default only see head's exit code - which is 0 - reflected in $? afterwards. Use echo ${PIPESTATUS[0]} to see Python's exit code.
[2] Curiously, on macOS 10.15.7 (Catalina), with Python 3.9.2 (but not 2.x), I see exit code 120, but the docs say 1, and that's what I also see on Linux.
I haven't reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)
import sys
with open('a.txt', 'r') as f1:
for line in f1:
sys.stdout.write(line)
You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.
import sys, errno
try:
with open('a.txt', 'r') as f1:
for line in f1:
sys.stdout.write(line)
except IOError as e:
if e.errno == errno.EPIPE:
# Handle error
You also need to make sure that othercommand is reading from the pipe before it gets too big - https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer
A "Broken Pipe" error occurs when you try to write to a pipe that has been closed on the other end. Since the code you've shown doesn't involve any pipes directly, I suspect you're doing something outside of Python to redirect the standard output of the Python interpreter to somewhere else. This could happen if you're running a script like this:
python foo.py | someothercommand
The issue you have is that someothercommand is exiting without reading everything available on its standard input. This causes your write (via print) to fail at some point.
I was able to reproduce the error with the following command on a Linux system:
python -c 'for i in range(1000): print i' | less
If I close the less pager without scrolling through all of its input (1000 lines), Python exits with the same IOError you have reported.
I feel obliged to point out that the method using
signal(SIGPIPE, SIG_DFL)
is indeed dangerous (as already suggested by David Bennet in the comments) and in my case led to platform-dependent funny business when combined with multiprocessing.Manager (because the standard library relies on BrokenPipeError being raised in several places). To make a long and painful story short, this is how I fixed it:
First, you need to catch the IOError (Python 2) or BrokenPipeError (Python 3). Depending on your program you can try to exit early at that point or just ignore the exception:
from errno import EPIPE
try:
broken_pipe_exception = BrokenPipeError
except NameError: # Python 2
broken_pipe_exception = IOError
try:
YOUR CODE GOES HERE
except broken_pipe_exception as exc:
if broken_pipe_exception == IOError:
if exc.errno != EPIPE:
raise
However, this isn't enough. Python 3 may still print a message like this:
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
Unfortunately getting rid of that message is not straightforward, but I finally found http://bugs.python.org/issue11380 where Robert Collins suggests this workaround that I turned into a decorator you can wrap your main function with (yes, that's some crazy indentation):
from functools import wraps
from sys import exit, stderr, stdout
from traceback import print_exc
def suppress_broken_pipe_msg(f):
#wraps(f)
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except SystemExit:
raise
except:
print_exc()
exit(1)
finally:
try:
stdout.flush()
finally:
try:
stdout.close()
finally:
try:
stderr.flush()
finally:
stderr.close()
return wrapper
#suppress_broken_pipe_msg
def main():
YOUR CODE GOES HERE
I know this is not the "proper" way to do it, but if you are simply interested in getting rid of the error message, you could try this workaround:
python your_python_code.py 2> /dev/null | other_command
The top answer (if e.errno == errno.EPIPE:) here didn't really work for me. I got:
AttributeError: 'BrokenPipeError' object has no attribute 'EPIPE'
However, this ought to work if all you care about is ignoring broken pipes on specific writes. I think it's safer than trapping SIGPIPE:
try:
# writing, flushing, whatever goes here
except BrokenPipeError:
exit( 0 )
You obviously have to make a decision as to whether your code is really, truly done if you hit the broken pipe, but for most purposes I think that's usually going to be true. (Don't forget to close file handles, etc.)
Depending on the exact cause of the issue, it might help to set an environment variable PYTHONUNBUFFERED=1, which forces the stdout and stderr streams to be unbuffered. See: https://docs.python.org/3/using/cmdline.html#cmdoption-u
So, your command
open.py | othercommand
becomes:
PYTHONUNBUFFERED=1 open.py | othercommand
Example:
$ python3 -m http.server | tee -a access.log
^CException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
$ PYTHONUNBUFFERED=1 python3 -m http.server | tee -a access.log
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
^C
$
This can also occur if the read end of the output from your script dies prematurely
ie open.py | otherCommand
if otherCommand exits and open.py tries to write to stdout
I had a bad gawk script that did this lovely to me.
Closes should be done in reverse order of the opens.
Related
I want to use logging, but with stderr redirection to stdout, as in:
import logging
import sys
logging.basicConfig(stream=sys.stdout)
for i in range(1, 100):
logging.warning("foo") # this should go to stdout
However, this setup is insufficient: if I run this script with grep -q foo, for instance, it will fail with BrokenPipeError: [Errno 32] Broken pipe.
Even if I wrap the whole for block around a try ... except, the error still happens.
As mentioned in this question, solutions such as sys.stderr.close() are not ideal, since they mask useful errors.
The solution from the question above (wrap a try ... except and then do sys.stdout = None) does not work in the case of the logging setup above. Neither does calling logging.shutdown() in the except block.
This logging-related question about redirecting stdout and stderr to a logger seems to indicate that it is necessary to write a class and several methods. Is it necessary even in my case? One of the answers suggests that using contextlib.redirect_stderr might help, but I tried and the error still happens (it happens inside the TextIOWrapper used by the logger, so it seems I cannot catch it).
Finally, when googling the "exception ignored message" sent by the logger, I find this SO question, but its solution is specific to the az command mentioned in the question.
So, I still couldn't find a workable solution: what's the simplest, correct way to setup a logger which sends warnings and errors to stdout?
Edit: on Windows, it's even worse: the broken pipe error may become an EINVAL (OSError: [Errno 22] Invalid argument). Apparently the only way to prevent it would be to code a custom stream (via TextIOBase) and then use that stream in a StreamHandler. So, replacing sys.stdout with something very similar, but which allows me to ignore the broken pipe error (possibly quitting execution if needed).
I have the following Python(2.7) code:
try:
FNULL = open(os.devnull,'w')
subprocess.check_call(["tar", "-czvf", '/folder/archive.tar.gz', '/folder/some_other_folder'], stdout=FNULL, stderr=subprocess.STDOUT)
except Exception as e:
print str(e)
The problem which I face is that, when there is no more space for the archive, print str(e) prints Command '['tar', '-czvf', '/folder/archive.tar.gz', '/folder/some_other_folder']' returned non-zero exit status 1, which is true, but I want to catch the real error here, that is gzip: write error: No space left on device (I got the this error when I ran the same tar comand manually). Is that possible somehow? I assume that gzip is another process within tar. Am I wrong? Please keep in mind that upgrading to Python 3 is not possible.
EDIT: I also tried to use subprocess.check_output() and print the contents of e.output but that also didn't work
Python 3 solution for sane people
On Python 3, the solution is simple, and you should be using Python 3 for new code anyway (Python 2.7 ended all support nearly a year ago):
The problem is that the program is echoing the error to stderr, so check_output doesn't capture it (either normally, or in the CalledProcessError). The best solution is to use subprocess.run (which check_call/check_output are just a thin wrapper over) and ensure you capture both stdout and stderr. The simplest approach is:
try:
subprocess.run(["tar", "-czvf", '/folder/archive.tar.gz', '/folder/some_other_folder'],
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
# ^ Ignores stdout ^ Captures stderr so e.stderr is populated if needed
except CalledProcessError as e:
print("tar exited with exit status {}:".format(e.returncode), e.stderr, file=sys.stderr)
Python 2 solution for people who like unsupported software
If you must do this on Python 2, you have to handle it all yourself by manually invoking Popen, as none of the high level functions available there will cover you (CalledProcessError didn't spawn a stderr attribute until 3.5, because no high-level API that raised it was designed to handle stderr at all):
with open(os.devnull, 'wb') as f:
proc = subprocess.Popen(["tar", "-czvf", '/folder/archive.tar.gz', '/folder/some_other_folder'],
stdout=f, stderr=subprocess.PIPE)
_, stderr = proc.communicate()
if proc.returncode != 0:
# Assumes from __future__ import print_function at top of file
# because Python 2 print statements are terrible
print("tar exited with exit status {}:".format(proc.returncode), stderr, file=sys.stderr)
I am trying to do a very simple example of using subprocess package. The python script should open a new process and run read command. read command should receive input from stdin via PIPE. Every time when I try to use write() and flush() it says:
Traceback (most recent call last):
File "recorder.py", line 68, in <module>
p.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe
My python code looks like:
import subprocess
import time
p = subprocess.Popen(
[
"read",
],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
bufsize=1
)
for character in "This is the message!\n":
p.stdin.write(character.encode("utf-8"))
time.sleep(0.25)
p.stdin.flush()
assert p.returncode == 0
Note: it's very important to send character after character (with sleeping timeout).
I actually could not replicate your result*, in my case your loop runs through and it'd fail on the assert as p has not finished yet and has no returncode (or rather its value is still None at that time). Inserting p.wait() after the loop and before the assert would force we only check for result after p has terminated.
Now for the exception you're seeing, it most likely indicates the pipe you're trying to perform flush() on is closed. Most likely due to the process having already terminated. Perhaps in your case at that point it already has a (non-zero) returncode too which could further help understand the problem?**
* On my system /bin/sh used by subprocess.Popen() with shell=True is actually bash. Running ["/bin/dash", "-c", "read"] which presumably happens to be shell called for /bin/sh on your system, I got broken pipe as well.
** Running dash like this seems to fail with:
/bin/dash: 1: read: arg count
And return 2.
Which sort of makes it more of a dash question: why calling /bin/dash -c "read" (from python) fails. It appears that dash read (unlike its bash counterpart) always expect at least one variable name to read into as an argument (replace read with read foo).
I guess this python question just became a lesson about assumptions and shell scripts portability. :)
I have a script that runs automatically on server through cronjob and it import and run several other scripts.
Some of them use prints, which naturally creates IOError: [Errno 5] Input/output error because the script runs without any SSH / terminal connected, so there's no proper stdout setup.
There are lots of questions about this subject but I couldn't find anyone that actually solve it, assuming I can't remove the print or change the executed scripts.
I tried several things, including:
class StdOut(object):
def __init__(self):
pass
def write(self, string):
pass
sys.stdout = StdOut()
sys.stderr = StdOut()
and
from __future__ import print_function
import __builtin__
def print(*args, **kwargs):
pass
__builtin__.print = print
But none of it works. I assume it only affect the module itself and not the modules I import / run later.
So how can I create a stub stdout that will affect all modules in the process? Like I said, I don't want to change the scripts that are executed from the main module, but I can change everything inside the importing module. And just to clearify - everything is imported, no new processes are spawned etc.
Thanks,
Modifying the builtin or changing sys.stdout should work (except for subprocesses—but you ruled those out) as long as you do it early enough. If not, though, there's a lower level trick that's much easier:
run your python scripts with I/O redirection that discards output:
python foo.py >/dev/null 2>&1
(assuming Unix-y scripts, as implied by "cron" in the question)
or, redirect file descriptors 1 and 2 (same idea as above, but done within your Python startup rather than as part of the cron-invoked command):
import os
fd = os.open(os.devnull, os.O_RDWR)
# NB: even if stdin is closed, fd >= 0
os.dup2(fd, 1)
os.dup2(fd, 2)
if fd > 2:
os.close(fd)
(this particular bit of code has the side effect of making /dev/null act as stdin, if all descriptors were closed). [Edit: I started with with open(...) and then switched to os.open and did not test the final version. Fixed now.]
All that said, a good cron really should have stdout and stderr connected somewhere, and should email the output/error-output to you. Not all cron versions are this nice though.
I have inherited some code which is periodically (randomly) failing due to an Input/Output error being raised during a call to print. I am trying to determine the cause of the exception being raised (or at least, better understand it) and how to handle it correctly.
When executing the following line of Python (in a 2.6.6 interpreter, running on CentOS 5.5):
print >> sys.stderr, 'Unable to do something: %s' % command
The exception is raised (traceback omitted):
IOError: [Errno 5] Input/output error
For context, this is generally what the larger function is trying to do at the time:
from subprocess import Popen, PIPE
import sys
def run_commands(commands):
for command in commands:
try:
out, err = Popen(command, shell=True, stdout=PIPE, stderr=PIPE).communicate()
print >> sys.stdout, out
if err:
raise Exception('ERROR -- an error occurred when executing this command: %s --- err: %s' % (command, err))
except:
print >> sys.stderr, 'Unable to do something: %s' % command
run_commands(["ls", "echo foo"])
The >> syntax is not particularly familiar to me, it's not something I use often, and I understand that it is perhaps the least preferred way of writing to stderr. However I don't believe the alternatives would fix the underlying problem.
From the documentation I have read, IOError 5 is often misused, and somewhat loosely defined, with different operating systems using it to cover different problems. The best I can see in my case is that the python process is no longer attached to the terminal/pty.
As best I can tell nothing is disconnecting the process from the stdout/stderr streams - the terminal is still open for example, and everything 'appears' to be fine. Could it be caused by the child process terminating in an unclean fashion? What else might be a cause of this problem - or what other steps could I introduce to debug it further?
In terms of handling the exception, I can obviously catch it, but I'm assuming this means I wont be able to print to stdout/stderr for the remainder of execution? Can I reattach to these streams somehow - perhaps by resetting sys.stdout to sys.__stdout__ etc? In this case not being able to write to stdout/stderr is not considered fatal but if it is an indication of something starting to go wrong I'd rather bail early.
I guess ultimately I'm at a bit of a loss as to where to start debugging this one...
I think it has to do with the terminal the process is attached to. I got this error when I run a python process in the background and closed the terminal in which I started it:
$ myprogram.py
Ctrl-Z
$ bg
$ exit
The problem was that I started a not daemonized process in a remote server and logged out (closing the terminal session). A solution was to start a screen/tmux session on the remote server and start the process within this session. Then detaching the session+log out keeps the terminal associated with the process. This works at least in the *nix world.
I had a very similar problem. I had a program that was launching several other programs using the subprocess module. Those subprocesses would then print output to the terminal. What I found was that when I closed the main program, it did not terminate the subprocesses automatically (as I had assumed), rather they kept running. So if I terminated both the main program and then the terminal it had been launched from*, the subprocesses no longer had a terminal attached to their stdout, and would throw an IOError. Hope this helps you.
*NB: it must be done in this order. If you just kill the terminal, (for some reason) that would kill both the main program and the subprocesses.
I just got this error because the directory where I was writing files to ran out of memory. Not sure if this is at all applicable to your situation.
I'm new here, so please forgive if I slip up a bit when it comes to the code detail.
Recently I was able to figure out what cause the I/O error of the print statement when the terminal associated with the run of the python script is closed.
It is because the string to be printed to stdout/stderr is too long. In this case, the "out" string is the culprit.
To fix this problem (without having to keep the terminal open while running the python script), simply read the "out" string line by line, and print line by line, until we reach the end of the "out" string. Something like:
while true:
ln=out.readline()
if not ln: break
print ln.strip("\n") # print without new line
The same problem occurs if you print the entire list of strings out to the screen. Simply print the list one item by one item.
Hope that helps!
The problem is you've closed the stdout pipe which python is attempting to write to when print() is called
This can be caused by running a script in the background using & and then closing the terminal session (ie. closing stdout)
$ python myscript.py &
$ exit
One solution is to set stdout to a file when running in the background
Example
$ python myscript.py > /var/log/myscript.log 2>&1 &
$ exit
No errors on print()
It could happen when your shell crashes while the print was trying to write the data into it.
For my case, I just restart the service, then this issue disappear. don't now why.
My issue was the same OSError Input/Output error, for Odoo.
After I restart the service, it disappeared.