How do I force Python's print function to flush the buffered output to the screen?
See also: Disable output buffering if the goal is to change the buffering behaviour generally. This question is about explicitly flushing output after a specific print call, even though output is still being buffered.
For duplicate closers: if a beginner is asking a question about trying to make output appear immediately while not using a newline at the end, please instead use Why doesn't print output show up immediately in the terminal when there is no newline at the end? to close the question. The current question isn't good enough because the person asking will likely not have a concept of buffering or flushing; the other question is intended to explain those concepts first, whereas this question is about the technical details.
In Python 3, print can take an optional flush argument:
print("Hello, World!", flush=True)
In Python 2, after calling print, do:
import sys
sys.stdout.flush()
By default, print prints to sys.stdout (see the documentation for more about file objects).
Running python -h, I see a command line option:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
Here is the relevant documentation.
Since Python 3.3, you can force the normal print() function to flush without the need to use sys.stdout.flush(); just set the "flush" keyword argument to true. From the documentation:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
Print objects to the stream file, separated by sep and followed by end. sep, end and file, if present, must be given as keyword arguments.
All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no objects are given, print() will just write end.
The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.
How to flush output of Python print?
I suggest five ways of doing this:
In Python 3, call print(..., flush=True) (the flush argument is not available in Python 2's print function, and there is no analogue for the print statement).
Call file.flush() on the output file (we can wrap python 2's print function to do this), for example, sys.stdout
apply this to every print function call in the module with a partial function,
print = partial(print, flush=True) applied to the module global.
apply this to the process with a flag (-u) passed to the interpreter command
apply this to every python process in your environment with PYTHONUNBUFFERED=TRUE (and unset the variable to undo this).
Python 3.3+
Using Python 3.3 or higher, you can just provide flush=True as a keyword argument to the print function:
print('foo', flush=True)
Python 2 (or < 3.3)
They did not backport the flush argument to Python 2.7 So if you're using Python 2 (or less than 3.3), and want code that's compatible with both 2 and 3, may I suggest the following compatibility code. (Note the __future__ import must be at/very "near the top of your module"):
from __future__ import print_function
import sys
if sys.version_info[:2] < (3, 3):
old_print = print
def print(*args, **kwargs):
flush = kwargs.pop('flush', False)
old_print(*args, **kwargs)
if flush:
file = kwargs.get('file', sys.stdout)
# Why might file=None? IDK, but it works for print(i, file=None)
file.flush() if file is not None else sys.stdout.flush()
The above compatibility code will cover most uses, but for a much more thorough treatment, see the six module.
Alternatively, you can just call file.flush() after printing, for example, with the print statement in Python 2:
import sys
print 'delayed output'
sys.stdout.flush()
Changing the default in one module to flush=True
You can change the default for the print function by using functools.partial on the global scope of a module:
import functools
print = functools.partial(print, flush=True)
if you look at our new partial function, at least in Python 3:
>>> print = functools.partial(print, flush=True)
>>> print
functools.partial(<built-in function print>, flush=True)
We can see it works just like normal:
>>> print('foo')
foo
And we can actually override the new default:
>>> print('foo', flush=False)
foo
Note again, this only changes the current global scope, because the print name on the current global scope will overshadow the builtin print function (or unreference the compatibility function, if using one in Python 2, in that current global scope).
If you want to do this inside a function instead of on a module's global scope, you should give it a different name, e.g.:
def foo():
printf = functools.partial(print, flush=True)
printf('print stuff like this')
If you declare it a global in a function, you're changing it on the module's global namespace, so you should just put it in the global namespace, unless that specific behavior is exactly what you want.
Changing the default for the process
I think the best option here is to use the -u flag to get unbuffered output.
$ python -u script.py
or
$ python -um package.module
From the docs:
Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.
Changing the default for the shell operating environment
You can get this behavior for all python processes in the environment or environments that inherit from the environment if you set the environment variable to a nonempty string:
e.g., in Linux or OSX:
$ export PYTHONUNBUFFERED=TRUE
or Windows:
C:\SET PYTHONUNBUFFERED=TRUE
from the docs:
PYTHONUNBUFFERED
If this is set to a non-empty string it is equivalent to specifying the -u option.
Addendum
Here's the help on the print function from Python 2.7.12 - note that there is no flush argument:
>>> from __future__ import print_function
>>> help(print)
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
Also, as suggested in this blog post, one can reopen sys.stdout in unbuffered mode:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Each stdout.write and print operation will be automatically flushed afterwards.
With Python 3.x the print() function has been extended:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
So, you can just do:
print("Visiting toilet", flush=True)
Python Docs Entry
Using the -u command-line switch works, but it is a little bit clumsy. It would mean that the program would potentially behave incorrectly if the user invoked the script without the -u option. I usually use a custom stdout, like this:
class flushfile:
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
... Now all your print calls (which use sys.stdout implicitly), will be automatically flushed.
Use an unbuffered file:
f = open('xyz.log', 'a', 0)
Or
sys.stdout = open('out.log', 'a', 0)
In Python 3 you can overwrite the print function with the default set to flush = True
def print(*objects, sep=' ', end='\n', file=sys.stdout, flush=True):
__builtins__.print(*objects, sep=sep, end=end, file=file, flush=flush)
Dan's idea doesn't quite work:
#!/usr/bin/env python
class flushfile(file):
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
print "foo"
The result:
Traceback (most recent call last):
File "./passpersist.py", line 12, in <module>
print "foo"
ValueError: I/O operation on closed file
I believe the problem is that it inherits from the file class, which actually isn't necessary. According to the documentation for sys.stdout:
stdout and stderr needn’t be built-in
file objects: any object is acceptable
as long as it has a write() method
that takes a string argument.
so changing
class flushfile(file):
to
class flushfile(object):
makes it work just fine.
Here is my version, which provides writelines() and fileno(), too:
class FlushFile(object):
def __init__(self, fd):
self.fd = fd
def write(self, x):
ret = self.fd.write(x)
self.fd.flush()
return ret
def writelines(self, lines):
ret = self.writelines(lines)
self.fd.flush()
return ret
def flush(self):
return self.fd.flush
def close(self):
return self.fd.close()
def fileno(self):
return self.fd.fileno()
I did it like this in Python 3.4:
'''To write to screen in real-time'''
message = lambda x: print(x, flush=True, end="")
message('I am flushing out now...')
I first struggled to understand how the flush option was working. I wanted to do a 'loading display' and here is the solution I found:
for i in range(100000):
print('{:s}\r'.format(''), end='', flush=True)
print('Loading index: {:d}/100000'.format(i+1), end='')
The first line flushes the previous print and the second line prints a new updated message. I don't know if an one-line syntax exists here.
Related
Is output buffering enabled by default in Python's interpreter for sys.stdout?
If the answer is positive, what are all the ways to disable it?
Suggestions so far:
Use the -u command line switch
Wrap sys.stdout in an object that flushes after every write
Set PYTHONUNBUFFERED env var
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Is there any other way to set some global flag in sys/sys.stdout programmatically during execution?
If you just want to flush after a specific write using print, see How can I flush the output of the print function?.
From Magnus Lycka answer on a mailing list:
You can skip buffering for a whole
python process using python -u
or by
setting the environment variable
PYTHONUNBUFFERED.
You could also replace sys.stdout with
some other stream like wrapper which
does a flush after every call.
class Unbuffered(object):
def __init__(self, stream):
self.stream = stream
def write(self, data):
self.stream.write(data)
self.stream.flush()
def writelines(self, datas):
self.stream.writelines(datas)
self.stream.flush()
def __getattr__(self, attr):
return getattr(self.stream, attr)
import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Hello'
I would rather put my answer in How to flush output of print function? or in Python's print function that flushes the buffer when it's called?, but since they were marked as duplicates of this one (what I do not agree), I'll answer it here.
Since Python 3.3, print() supports the keyword argument "flush" (see documentation):
print('Hello World!', flush=True)
# reopen stdout file descriptor with write mode
# and 0 as the buffer size (unbuffered)
import io, os, sys
try:
# Python 3, open as binary, then wrap in a TextIOWrapper with write-through.
sys.stdout = io.TextIOWrapper(open(sys.stdout.fileno(), 'wb', 0), write_through=True)
# If flushing on newlines is sufficient, as of 3.7 you can instead just call:
# sys.stdout.reconfigure(line_buffering=True)
except TypeError:
# Python 2
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Credits: "Sebastian", somewhere on the Python mailing list.
Yes, it is.
You can disable it on the commandline with the "-u" switch.
Alternatively, you could call .flush() on sys.stdout on every write (or wrap it with an object that does this automatically)
This relates to Cristóvão D. Sousa's answer, but I couldn't comment yet.
A straight-forward way of using the flush keyword argument of Python 3 in order to always have unbuffered output is:
import functools
print = functools.partial(print, flush=True)
afterwards, print will always flush the output directly (except flush=False is given).
Note, (a) that this answers the question only partially as it doesn't redirect all the output. But I guess print is the most common way for creating output to stdout/stderr in python, so these 2 lines cover probably most of the use cases.
Note (b) that it only works in the module/script where you defined it. This can be good when writing a module as it doesn't mess with the sys.stdout.
Python 2 doesn't provide the flush argument, but you could emulate a Python 3-type print function as described here https://stackoverflow.com/a/27991478/3734258 .
def disable_stdout_buffering():
# Appending to gc.garbage is a way to stop an object from being
# destroyed. If the old sys.stdout is ever collected, it will
# close() stdout, which is not good.
gc.garbage.append(sys.stdout)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# Then this will give output in the correct order:
disable_stdout_buffering()
print "hello"
subprocess.call(["echo", "bye"])
Without saving the old sys.stdout, disable_stdout_buffering() isn't idempotent, and multiple calls will result in an error like this:
Traceback (most recent call last):
File "test/buffering.py", line 17, in <module>
print "hello"
IOError: [Errno 9] Bad file descriptor
close failed: [Errno 9] Bad file descriptor
Another possibility is:
def disable_stdout_buffering():
fileno = sys.stdout.fileno()
temp_fd = os.dup(fileno)
sys.stdout.close()
os.dup2(temp_fd, fileno)
os.close(temp_fd)
sys.stdout = os.fdopen(fileno, "w", 0)
(Appending to gc.garbage is not such a good idea because it's where unfreeable cycles get put, and you might want to check for those.)
The following works in Python 2.6, 2.7, and 3.2:
import os
import sys
buf_arg = 0
if sys.version_info[0] == 3:
os.environ['PYTHONUNBUFFERED'] = '1'
buf_arg = 1
sys.stdout = os.fdopen(sys.stdout.fileno(), 'a+', buf_arg)
sys.stderr = os.fdopen(sys.stderr.fileno(), 'a+', buf_arg)
Yes, it is enabled by default. You can disable it by using the -u option on the command line when calling python.
In Python 3, you can monkey-patch the print function, to always send flush=True:
_orig_print = print
def print(*args, **kwargs):
_orig_print(*args, flush=True, **kwargs)
As pointed out in a comment, you can simplify this by binding the flush parameter to a value, via functools.partial:
print = functools.partial(print, flush=True)
You can also run Python with stdbuf utility:
stdbuf -oL python <script>
You can create an unbuffered file and assign this file to sys.stdout.
import sys
myFile= open( "a.log", "w", 0 )
sys.stdout= myFile
You can't magically change the system-supplied stdout; since it's supplied to your python program by the OS.
You can also use fcntl to change the file flags in-fly.
fl = fcntl.fcntl(fd.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC # or os.O_DSYNC (if you don't care the file timestamp updates)
fcntl.fcntl(fd.fileno(), fcntl.F_SETFL, fl)
One way to get unbuffered output would be to use sys.stderr instead of sys.stdout or to simply call sys.stdout.flush() to explicitly force a write to occur.
You could easily redirect everything printed by doing:
import sys; sys.stdout = sys.stderr
print "Hello World!"
Or to redirect just for a particular print statement:
print >>sys.stderr, "Hello World!"
To reset stdout you can just do:
sys.stdout = sys.__stdout__
It is possible to override only write method of sys.stdout with one that calls flush. Suggested method implementation is below.
def write_flush(args, w=stdout.write):
w(args)
stdout.flush()
Default value of w argument will keep original write method reference. After write_flush is defined, the original write might be overridden.
stdout.write = write_flush
The code assumes that stdout is imported this way from sys import stdout.
Variant that works without crashing (at least on win32; python 2.7, ipython 0.12) then called subsequently (multiple times):
def DisOutBuffering():
if sys.stdout.name == '<stdout>':
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
if sys.stderr.name == '<stderr>':
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)
(I've posted a comment, but it got lost somehow. So, again:)
As I noticed, CPython (at least on Linux) behaves differently depending on where the output goes. If it goes to a tty, then the output is flushed after each '\n'
If it goes to a pipe/process, then it is buffered and you can use the flush() based solutions or the -u option recommended above.
Slightly related to output buffering:
If you iterate over the lines in the input with
for line in sys.stdin:
...
then the for implementation in CPython will collect the input for a while and then execute the loop body for a bunch of input lines. If your script is about to write output for each input line, this might look like output buffering but it's actually batching, and therefore, none of the flush(), etc. techniques will help that.
Interestingly, you don't have this behaviour in pypy.
To avoid this, you can use
while True:
line=sys.stdin.readline()
...
Is output buffering enabled by default in Python's interpreter for sys.stdout?
If the answer is positive, what are all the ways to disable it?
Suggestions so far:
Use the -u command line switch
Wrap sys.stdout in an object that flushes after every write
Set PYTHONUNBUFFERED env var
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Is there any other way to set some global flag in sys/sys.stdout programmatically during execution?
If you just want to flush after a specific write using print, see How can I flush the output of the print function?.
From Magnus Lycka answer on a mailing list:
You can skip buffering for a whole
python process using python -u
or by
setting the environment variable
PYTHONUNBUFFERED.
You could also replace sys.stdout with
some other stream like wrapper which
does a flush after every call.
class Unbuffered(object):
def __init__(self, stream):
self.stream = stream
def write(self, data):
self.stream.write(data)
self.stream.flush()
def writelines(self, datas):
self.stream.writelines(datas)
self.stream.flush()
def __getattr__(self, attr):
return getattr(self.stream, attr)
import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Hello'
I would rather put my answer in How to flush output of print function? or in Python's print function that flushes the buffer when it's called?, but since they were marked as duplicates of this one (what I do not agree), I'll answer it here.
Since Python 3.3, print() supports the keyword argument "flush" (see documentation):
print('Hello World!', flush=True)
# reopen stdout file descriptor with write mode
# and 0 as the buffer size (unbuffered)
import io, os, sys
try:
# Python 3, open as binary, then wrap in a TextIOWrapper with write-through.
sys.stdout = io.TextIOWrapper(open(sys.stdout.fileno(), 'wb', 0), write_through=True)
# If flushing on newlines is sufficient, as of 3.7 you can instead just call:
# sys.stdout.reconfigure(line_buffering=True)
except TypeError:
# Python 2
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Credits: "Sebastian", somewhere on the Python mailing list.
Yes, it is.
You can disable it on the commandline with the "-u" switch.
Alternatively, you could call .flush() on sys.stdout on every write (or wrap it with an object that does this automatically)
This relates to Cristóvão D. Sousa's answer, but I couldn't comment yet.
A straight-forward way of using the flush keyword argument of Python 3 in order to always have unbuffered output is:
import functools
print = functools.partial(print, flush=True)
afterwards, print will always flush the output directly (except flush=False is given).
Note, (a) that this answers the question only partially as it doesn't redirect all the output. But I guess print is the most common way for creating output to stdout/stderr in python, so these 2 lines cover probably most of the use cases.
Note (b) that it only works in the module/script where you defined it. This can be good when writing a module as it doesn't mess with the sys.stdout.
Python 2 doesn't provide the flush argument, but you could emulate a Python 3-type print function as described here https://stackoverflow.com/a/27991478/3734258 .
def disable_stdout_buffering():
# Appending to gc.garbage is a way to stop an object from being
# destroyed. If the old sys.stdout is ever collected, it will
# close() stdout, which is not good.
gc.garbage.append(sys.stdout)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# Then this will give output in the correct order:
disable_stdout_buffering()
print "hello"
subprocess.call(["echo", "bye"])
Without saving the old sys.stdout, disable_stdout_buffering() isn't idempotent, and multiple calls will result in an error like this:
Traceback (most recent call last):
File "test/buffering.py", line 17, in <module>
print "hello"
IOError: [Errno 9] Bad file descriptor
close failed: [Errno 9] Bad file descriptor
Another possibility is:
def disable_stdout_buffering():
fileno = sys.stdout.fileno()
temp_fd = os.dup(fileno)
sys.stdout.close()
os.dup2(temp_fd, fileno)
os.close(temp_fd)
sys.stdout = os.fdopen(fileno, "w", 0)
(Appending to gc.garbage is not such a good idea because it's where unfreeable cycles get put, and you might want to check for those.)
The following works in Python 2.6, 2.7, and 3.2:
import os
import sys
buf_arg = 0
if sys.version_info[0] == 3:
os.environ['PYTHONUNBUFFERED'] = '1'
buf_arg = 1
sys.stdout = os.fdopen(sys.stdout.fileno(), 'a+', buf_arg)
sys.stderr = os.fdopen(sys.stderr.fileno(), 'a+', buf_arg)
Yes, it is enabled by default. You can disable it by using the -u option on the command line when calling python.
In Python 3, you can monkey-patch the print function, to always send flush=True:
_orig_print = print
def print(*args, **kwargs):
_orig_print(*args, flush=True, **kwargs)
As pointed out in a comment, you can simplify this by binding the flush parameter to a value, via functools.partial:
print = functools.partial(print, flush=True)
You can also run Python with stdbuf utility:
stdbuf -oL python <script>
You can create an unbuffered file and assign this file to sys.stdout.
import sys
myFile= open( "a.log", "w", 0 )
sys.stdout= myFile
You can't magically change the system-supplied stdout; since it's supplied to your python program by the OS.
You can also use fcntl to change the file flags in-fly.
fl = fcntl.fcntl(fd.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC # or os.O_DSYNC (if you don't care the file timestamp updates)
fcntl.fcntl(fd.fileno(), fcntl.F_SETFL, fl)
One way to get unbuffered output would be to use sys.stderr instead of sys.stdout or to simply call sys.stdout.flush() to explicitly force a write to occur.
You could easily redirect everything printed by doing:
import sys; sys.stdout = sys.stderr
print "Hello World!"
Or to redirect just for a particular print statement:
print >>sys.stderr, "Hello World!"
To reset stdout you can just do:
sys.stdout = sys.__stdout__
It is possible to override only write method of sys.stdout with one that calls flush. Suggested method implementation is below.
def write_flush(args, w=stdout.write):
w(args)
stdout.flush()
Default value of w argument will keep original write method reference. After write_flush is defined, the original write might be overridden.
stdout.write = write_flush
The code assumes that stdout is imported this way from sys import stdout.
Variant that works without crashing (at least on win32; python 2.7, ipython 0.12) then called subsequently (multiple times):
def DisOutBuffering():
if sys.stdout.name == '<stdout>':
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
if sys.stderr.name == '<stderr>':
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)
(I've posted a comment, but it got lost somehow. So, again:)
As I noticed, CPython (at least on Linux) behaves differently depending on where the output goes. If it goes to a tty, then the output is flushed after each '\n'
If it goes to a pipe/process, then it is buffered and you can use the flush() based solutions or the -u option recommended above.
Slightly related to output buffering:
If you iterate over the lines in the input with
for line in sys.stdin:
...
then the for implementation in CPython will collect the input for a while and then execute the loop body for a bunch of input lines. If your script is about to write output for each input line, this might look like output buffering but it's actually batching, and therefore, none of the flush(), etc. techniques will help that.
Interestingly, you don't have this behaviour in pypy.
To avoid this, you can use
while True:
line=sys.stdin.readline()
...
How do I force Python's print function to flush the buffered output to the screen?
See also: Disable output buffering if the goal is to change the buffering behaviour generally. This question is about explicitly flushing output after a specific print call, even though output is still being buffered.
For duplicate closers: if a beginner is asking a question about trying to make output appear immediately while not using a newline at the end, please instead use Why doesn't print output show up immediately in the terminal when there is no newline at the end? to close the question. The current question isn't good enough because the person asking will likely not have a concept of buffering or flushing; the other question is intended to explain those concepts first, whereas this question is about the technical details.
In Python 3, print can take an optional flush argument:
print("Hello, World!", flush=True)
In Python 2, after calling print, do:
import sys
sys.stdout.flush()
By default, print prints to sys.stdout (see the documentation for more about file objects).
Running python -h, I see a command line option:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
Here is the relevant documentation.
Since Python 3.3, you can force the normal print() function to flush without the need to use sys.stdout.flush(); just set the "flush" keyword argument to true. From the documentation:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
Print objects to the stream file, separated by sep and followed by end. sep, end and file, if present, must be given as keyword arguments.
All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no objects are given, print() will just write end.
The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.
How to flush output of Python print?
I suggest five ways of doing this:
In Python 3, call print(..., flush=True) (the flush argument is not available in Python 2's print function, and there is no analogue for the print statement).
Call file.flush() on the output file (we can wrap python 2's print function to do this), for example, sys.stdout
apply this to every print function call in the module with a partial function,
print = partial(print, flush=True) applied to the module global.
apply this to the process with a flag (-u) passed to the interpreter command
apply this to every python process in your environment with PYTHONUNBUFFERED=TRUE (and unset the variable to undo this).
Python 3.3+
Using Python 3.3 or higher, you can just provide flush=True as a keyword argument to the print function:
print('foo', flush=True)
Python 2 (or < 3.3)
They did not backport the flush argument to Python 2.7 So if you're using Python 2 (or less than 3.3), and want code that's compatible with both 2 and 3, may I suggest the following compatibility code. (Note the __future__ import must be at/very "near the top of your module"):
from __future__ import print_function
import sys
if sys.version_info[:2] < (3, 3):
old_print = print
def print(*args, **kwargs):
flush = kwargs.pop('flush', False)
old_print(*args, **kwargs)
if flush:
file = kwargs.get('file', sys.stdout)
# Why might file=None? IDK, but it works for print(i, file=None)
file.flush() if file is not None else sys.stdout.flush()
The above compatibility code will cover most uses, but for a much more thorough treatment, see the six module.
Alternatively, you can just call file.flush() after printing, for example, with the print statement in Python 2:
import sys
print 'delayed output'
sys.stdout.flush()
Changing the default in one module to flush=True
You can change the default for the print function by using functools.partial on the global scope of a module:
import functools
print = functools.partial(print, flush=True)
if you look at our new partial function, at least in Python 3:
>>> print = functools.partial(print, flush=True)
>>> print
functools.partial(<built-in function print>, flush=True)
We can see it works just like normal:
>>> print('foo')
foo
And we can actually override the new default:
>>> print('foo', flush=False)
foo
Note again, this only changes the current global scope, because the print name on the current global scope will overshadow the builtin print function (or unreference the compatibility function, if using one in Python 2, in that current global scope).
If you want to do this inside a function instead of on a module's global scope, you should give it a different name, e.g.:
def foo():
printf = functools.partial(print, flush=True)
printf('print stuff like this')
If you declare it a global in a function, you're changing it on the module's global namespace, so you should just put it in the global namespace, unless that specific behavior is exactly what you want.
Changing the default for the process
I think the best option here is to use the -u flag to get unbuffered output.
$ python -u script.py
or
$ python -um package.module
From the docs:
Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.
Changing the default for the shell operating environment
You can get this behavior for all python processes in the environment or environments that inherit from the environment if you set the environment variable to a nonempty string:
e.g., in Linux or OSX:
$ export PYTHONUNBUFFERED=TRUE
or Windows:
C:\SET PYTHONUNBUFFERED=TRUE
from the docs:
PYTHONUNBUFFERED
If this is set to a non-empty string it is equivalent to specifying the -u option.
Addendum
Here's the help on the print function from Python 2.7.12 - note that there is no flush argument:
>>> from __future__ import print_function
>>> help(print)
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
Also, as suggested in this blog post, one can reopen sys.stdout in unbuffered mode:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Each stdout.write and print operation will be automatically flushed afterwards.
With Python 3.x the print() function has been extended:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
So, you can just do:
print("Visiting toilet", flush=True)
Python Docs Entry
Using the -u command-line switch works, but it is a little bit clumsy. It would mean that the program would potentially behave incorrectly if the user invoked the script without the -u option. I usually use a custom stdout, like this:
class flushfile:
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
... Now all your print calls (which use sys.stdout implicitly), will be automatically flushed.
Use an unbuffered file:
f = open('xyz.log', 'a', 0)
Or
sys.stdout = open('out.log', 'a', 0)
In Python 3 you can overwrite the print function with the default set to flush = True
def print(*objects, sep=' ', end='\n', file=sys.stdout, flush=True):
__builtins__.print(*objects, sep=sep, end=end, file=file, flush=flush)
Dan's idea doesn't quite work:
#!/usr/bin/env python
class flushfile(file):
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
print "foo"
The result:
Traceback (most recent call last):
File "./passpersist.py", line 12, in <module>
print "foo"
ValueError: I/O operation on closed file
I believe the problem is that it inherits from the file class, which actually isn't necessary. According to the documentation for sys.stdout:
stdout and stderr needn’t be built-in
file objects: any object is acceptable
as long as it has a write() method
that takes a string argument.
so changing
class flushfile(file):
to
class flushfile(object):
makes it work just fine.
Here is my version, which provides writelines() and fileno(), too:
class FlushFile(object):
def __init__(self, fd):
self.fd = fd
def write(self, x):
ret = self.fd.write(x)
self.fd.flush()
return ret
def writelines(self, lines):
ret = self.writelines(lines)
self.fd.flush()
return ret
def flush(self):
return self.fd.flush
def close(self):
return self.fd.close()
def fileno(self):
return self.fd.fileno()
I did it like this in Python 3.4:
'''To write to screen in real-time'''
message = lambda x: print(x, flush=True, end="")
message('I am flushing out now...')
I first struggled to understand how the flush option was working. I wanted to do a 'loading display' and here is the solution I found:
for i in range(100000):
print('{:s}\r'.format(''), end='', flush=True)
print('Loading index: {:d}/100000'.format(i+1), end='')
The first line flushes the previous print and the second line prints a new updated message. I don't know if an one-line syntax exists here.
Is output buffering enabled by default in Python's interpreter for sys.stdout?
If the answer is positive, what are all the ways to disable it?
Suggestions so far:
Use the -u command line switch
Wrap sys.stdout in an object that flushes after every write
Set PYTHONUNBUFFERED env var
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Is there any other way to set some global flag in sys/sys.stdout programmatically during execution?
If you just want to flush after a specific write using print, see How can I flush the output of the print function?.
From Magnus Lycka answer on a mailing list:
You can skip buffering for a whole
python process using python -u
or by
setting the environment variable
PYTHONUNBUFFERED.
You could also replace sys.stdout with
some other stream like wrapper which
does a flush after every call.
class Unbuffered(object):
def __init__(self, stream):
self.stream = stream
def write(self, data):
self.stream.write(data)
self.stream.flush()
def writelines(self, datas):
self.stream.writelines(datas)
self.stream.flush()
def __getattr__(self, attr):
return getattr(self.stream, attr)
import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Hello'
I would rather put my answer in How to flush output of print function? or in Python's print function that flushes the buffer when it's called?, but since they were marked as duplicates of this one (what I do not agree), I'll answer it here.
Since Python 3.3, print() supports the keyword argument "flush" (see documentation):
print('Hello World!', flush=True)
# reopen stdout file descriptor with write mode
# and 0 as the buffer size (unbuffered)
import io, os, sys
try:
# Python 3, open as binary, then wrap in a TextIOWrapper with write-through.
sys.stdout = io.TextIOWrapper(open(sys.stdout.fileno(), 'wb', 0), write_through=True)
# If flushing on newlines is sufficient, as of 3.7 you can instead just call:
# sys.stdout.reconfigure(line_buffering=True)
except TypeError:
# Python 2
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Credits: "Sebastian", somewhere on the Python mailing list.
Yes, it is.
You can disable it on the commandline with the "-u" switch.
Alternatively, you could call .flush() on sys.stdout on every write (or wrap it with an object that does this automatically)
This relates to Cristóvão D. Sousa's answer, but I couldn't comment yet.
A straight-forward way of using the flush keyword argument of Python 3 in order to always have unbuffered output is:
import functools
print = functools.partial(print, flush=True)
afterwards, print will always flush the output directly (except flush=False is given).
Note, (a) that this answers the question only partially as it doesn't redirect all the output. But I guess print is the most common way for creating output to stdout/stderr in python, so these 2 lines cover probably most of the use cases.
Note (b) that it only works in the module/script where you defined it. This can be good when writing a module as it doesn't mess with the sys.stdout.
Python 2 doesn't provide the flush argument, but you could emulate a Python 3-type print function as described here https://stackoverflow.com/a/27991478/3734258 .
def disable_stdout_buffering():
# Appending to gc.garbage is a way to stop an object from being
# destroyed. If the old sys.stdout is ever collected, it will
# close() stdout, which is not good.
gc.garbage.append(sys.stdout)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# Then this will give output in the correct order:
disable_stdout_buffering()
print "hello"
subprocess.call(["echo", "bye"])
Without saving the old sys.stdout, disable_stdout_buffering() isn't idempotent, and multiple calls will result in an error like this:
Traceback (most recent call last):
File "test/buffering.py", line 17, in <module>
print "hello"
IOError: [Errno 9] Bad file descriptor
close failed: [Errno 9] Bad file descriptor
Another possibility is:
def disable_stdout_buffering():
fileno = sys.stdout.fileno()
temp_fd = os.dup(fileno)
sys.stdout.close()
os.dup2(temp_fd, fileno)
os.close(temp_fd)
sys.stdout = os.fdopen(fileno, "w", 0)
(Appending to gc.garbage is not such a good idea because it's where unfreeable cycles get put, and you might want to check for those.)
The following works in Python 2.6, 2.7, and 3.2:
import os
import sys
buf_arg = 0
if sys.version_info[0] == 3:
os.environ['PYTHONUNBUFFERED'] = '1'
buf_arg = 1
sys.stdout = os.fdopen(sys.stdout.fileno(), 'a+', buf_arg)
sys.stderr = os.fdopen(sys.stderr.fileno(), 'a+', buf_arg)
Yes, it is enabled by default. You can disable it by using the -u option on the command line when calling python.
In Python 3, you can monkey-patch the print function, to always send flush=True:
_orig_print = print
def print(*args, **kwargs):
_orig_print(*args, flush=True, **kwargs)
As pointed out in a comment, you can simplify this by binding the flush parameter to a value, via functools.partial:
print = functools.partial(print, flush=True)
You can also run Python with stdbuf utility:
stdbuf -oL python <script>
You can create an unbuffered file and assign this file to sys.stdout.
import sys
myFile= open( "a.log", "w", 0 )
sys.stdout= myFile
You can't magically change the system-supplied stdout; since it's supplied to your python program by the OS.
You can also use fcntl to change the file flags in-fly.
fl = fcntl.fcntl(fd.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC # or os.O_DSYNC (if you don't care the file timestamp updates)
fcntl.fcntl(fd.fileno(), fcntl.F_SETFL, fl)
One way to get unbuffered output would be to use sys.stderr instead of sys.stdout or to simply call sys.stdout.flush() to explicitly force a write to occur.
You could easily redirect everything printed by doing:
import sys; sys.stdout = sys.stderr
print "Hello World!"
Or to redirect just for a particular print statement:
print >>sys.stderr, "Hello World!"
To reset stdout you can just do:
sys.stdout = sys.__stdout__
It is possible to override only write method of sys.stdout with one that calls flush. Suggested method implementation is below.
def write_flush(args, w=stdout.write):
w(args)
stdout.flush()
Default value of w argument will keep original write method reference. After write_flush is defined, the original write might be overridden.
stdout.write = write_flush
The code assumes that stdout is imported this way from sys import stdout.
Variant that works without crashing (at least on win32; python 2.7, ipython 0.12) then called subsequently (multiple times):
def DisOutBuffering():
if sys.stdout.name == '<stdout>':
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
if sys.stderr.name == '<stderr>':
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)
(I've posted a comment, but it got lost somehow. So, again:)
As I noticed, CPython (at least on Linux) behaves differently depending on where the output goes. If it goes to a tty, then the output is flushed after each '\n'
If it goes to a pipe/process, then it is buffered and you can use the flush() based solutions or the -u option recommended above.
Slightly related to output buffering:
If you iterate over the lines in the input with
for line in sys.stdin:
...
then the for implementation in CPython will collect the input for a while and then execute the loop body for a bunch of input lines. If your script is about to write output for each input line, this might look like output buffering but it's actually batching, and therefore, none of the flush(), etc. techniques will help that.
Interestingly, you don't have this behaviour in pypy.
To avoid this, you can use
while True:
line=sys.stdin.readline()
...
How do I force Python's print function to flush the buffered output to the screen?
See also: Disable output buffering if the goal is to change the buffering behaviour generally. This question is about explicitly flushing output after a specific print call, even though output is still being buffered.
For duplicate closers: if a beginner is asking a question about trying to make output appear immediately while not using a newline at the end, please instead use Why doesn't print output show up immediately in the terminal when there is no newline at the end? to close the question. The current question isn't good enough because the person asking will likely not have a concept of buffering or flushing; the other question is intended to explain those concepts first, whereas this question is about the technical details.
In Python 3, print can take an optional flush argument:
print("Hello, World!", flush=True)
In Python 2, after calling print, do:
import sys
sys.stdout.flush()
By default, print prints to sys.stdout (see the documentation for more about file objects).
Running python -h, I see a command line option:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
Here is the relevant documentation.
Since Python 3.3, you can force the normal print() function to flush without the need to use sys.stdout.flush(); just set the "flush" keyword argument to true. From the documentation:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
Print objects to the stream file, separated by sep and followed by end. sep, end and file, if present, must be given as keyword arguments.
All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no objects are given, print() will just write end.
The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.
How to flush output of Python print?
I suggest five ways of doing this:
In Python 3, call print(..., flush=True) (the flush argument is not available in Python 2's print function, and there is no analogue for the print statement).
Call file.flush() on the output file (we can wrap python 2's print function to do this), for example, sys.stdout
apply this to every print function call in the module with a partial function,
print = partial(print, flush=True) applied to the module global.
apply this to the process with a flag (-u) passed to the interpreter command
apply this to every python process in your environment with PYTHONUNBUFFERED=TRUE (and unset the variable to undo this).
Python 3.3+
Using Python 3.3 or higher, you can just provide flush=True as a keyword argument to the print function:
print('foo', flush=True)
Python 2 (or < 3.3)
They did not backport the flush argument to Python 2.7 So if you're using Python 2 (or less than 3.3), and want code that's compatible with both 2 and 3, may I suggest the following compatibility code. (Note the __future__ import must be at/very "near the top of your module"):
from __future__ import print_function
import sys
if sys.version_info[:2] < (3, 3):
old_print = print
def print(*args, **kwargs):
flush = kwargs.pop('flush', False)
old_print(*args, **kwargs)
if flush:
file = kwargs.get('file', sys.stdout)
# Why might file=None? IDK, but it works for print(i, file=None)
file.flush() if file is not None else sys.stdout.flush()
The above compatibility code will cover most uses, but for a much more thorough treatment, see the six module.
Alternatively, you can just call file.flush() after printing, for example, with the print statement in Python 2:
import sys
print 'delayed output'
sys.stdout.flush()
Changing the default in one module to flush=True
You can change the default for the print function by using functools.partial on the global scope of a module:
import functools
print = functools.partial(print, flush=True)
if you look at our new partial function, at least in Python 3:
>>> print = functools.partial(print, flush=True)
>>> print
functools.partial(<built-in function print>, flush=True)
We can see it works just like normal:
>>> print('foo')
foo
And we can actually override the new default:
>>> print('foo', flush=False)
foo
Note again, this only changes the current global scope, because the print name on the current global scope will overshadow the builtin print function (or unreference the compatibility function, if using one in Python 2, in that current global scope).
If you want to do this inside a function instead of on a module's global scope, you should give it a different name, e.g.:
def foo():
printf = functools.partial(print, flush=True)
printf('print stuff like this')
If you declare it a global in a function, you're changing it on the module's global namespace, so you should just put it in the global namespace, unless that specific behavior is exactly what you want.
Changing the default for the process
I think the best option here is to use the -u flag to get unbuffered output.
$ python -u script.py
or
$ python -um package.module
From the docs:
Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.
Changing the default for the shell operating environment
You can get this behavior for all python processes in the environment or environments that inherit from the environment if you set the environment variable to a nonempty string:
e.g., in Linux or OSX:
$ export PYTHONUNBUFFERED=TRUE
or Windows:
C:\SET PYTHONUNBUFFERED=TRUE
from the docs:
PYTHONUNBUFFERED
If this is set to a non-empty string it is equivalent to specifying the -u option.
Addendum
Here's the help on the print function from Python 2.7.12 - note that there is no flush argument:
>>> from __future__ import print_function
>>> help(print)
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
Also, as suggested in this blog post, one can reopen sys.stdout in unbuffered mode:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Each stdout.write and print operation will be automatically flushed afterwards.
With Python 3.x the print() function has been extended:
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
So, you can just do:
print("Visiting toilet", flush=True)
Python Docs Entry
Using the -u command-line switch works, but it is a little bit clumsy. It would mean that the program would potentially behave incorrectly if the user invoked the script without the -u option. I usually use a custom stdout, like this:
class flushfile:
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
... Now all your print calls (which use sys.stdout implicitly), will be automatically flushed.
Use an unbuffered file:
f = open('xyz.log', 'a', 0)
Or
sys.stdout = open('out.log', 'a', 0)
In Python 3 you can overwrite the print function with the default set to flush = True
def print(*objects, sep=' ', end='\n', file=sys.stdout, flush=True):
__builtins__.print(*objects, sep=sep, end=end, file=file, flush=flush)
Dan's idea doesn't quite work:
#!/usr/bin/env python
class flushfile(file):
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
import sys
sys.stdout = flushfile(sys.stdout)
print "foo"
The result:
Traceback (most recent call last):
File "./passpersist.py", line 12, in <module>
print "foo"
ValueError: I/O operation on closed file
I believe the problem is that it inherits from the file class, which actually isn't necessary. According to the documentation for sys.stdout:
stdout and stderr needn’t be built-in
file objects: any object is acceptable
as long as it has a write() method
that takes a string argument.
so changing
class flushfile(file):
to
class flushfile(object):
makes it work just fine.
Here is my version, which provides writelines() and fileno(), too:
class FlushFile(object):
def __init__(self, fd):
self.fd = fd
def write(self, x):
ret = self.fd.write(x)
self.fd.flush()
return ret
def writelines(self, lines):
ret = self.writelines(lines)
self.fd.flush()
return ret
def flush(self):
return self.fd.flush
def close(self):
return self.fd.close()
def fileno(self):
return self.fd.fileno()
I did it like this in Python 3.4:
'''To write to screen in real-time'''
message = lambda x: print(x, flush=True, end="")
message('I am flushing out now...')
I first struggled to understand how the flush option was working. I wanted to do a 'loading display' and here is the solution I found:
for i in range(100000):
print('{:s}\r'.format(''), end='', flush=True)
print('Loading index: {:d}/100000'.format(i+1), end='')
The first line flushes the previous print and the second line prints a new updated message. I don't know if an one-line syntax exists here.