I recently switched OS and am using a newer Python (2.7). On my old system, I used to be able to print instantaneously. For instance, suppose I had a computationally intense for loop:
for i in range(10):
huge calculation
print i
then as the code completed each iteration, it would print i
However, on my current system, python seems to cache the stdout so that the terminal is blank for several minutes, after which it prints:
1
2
3
in short succession. Then, after a few more minutes, it prints:
4
5
6
and so on. How can I make python print as soon as it reaches the print statement?
Try to call flush of stdout after the print
import sys
...
sys.stdout.flush()
Or use a command line option -u which:
Force stdin, stdout and stderr to be totally unbuffered.
Since Python 3.3, you can simply pass flush=True to the print function.
Import the new print-as-function as in Python 3.x:
from __future__ import print_function
(put the statement at the top of your script/module)
This allows you to replace the new print function with your own:
def print(s, end='\n', file=sys.stdout):
file.write(s + end)
file.flush()
The advantage is that this way your script will work just the same when you upgrade one day to Python 3.x.
Ps1: I did not try it out, but the print-as-function might just flush by default.
PS2: you might also be interested in my progressbar example.
Related
Well, I have read that stdout is line-buffered. But the code works differently in Pydroid 3(unaware of the exact version) and Python 3.8.3.
import time
print('Hello', end = '')
time.sleep(5)
print('World')
In Pydroid 3, both Hello and World are printed after (at least after) 5 seconds while in Python 3.8.3, Hello is printed first, and World is printed after 5 seconds.
Why is the code working differently?
It is probably not a Python version issue, but a different terminal issue.
Some terminals (or more accurately files/streams, stdout included) only flush after a newline (which the first print doesn't have), while others can flush after every write.
to force a flush you can use flush=True as a param to print, try this:
import time
print('Hello', end='', flush=True)
time.sleep(5)
print('World')
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
I am working on a cluster and using Python to do some calculations.
However, python scripts does not print anything until they finish or get killed.
I have checked some ways (from Disable output buffering)
Use the -u command line switch
Wrap sys.stdout in an object that flushes after every write
Set PYTHONUNBUFFERED env var
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
But none of these works.
The python version is 2.7.8, 2.7.3 and 2.4.2 (There are three versions on the system, which must be put into PATH before running)
Also, because the cluster uses NFS, I have the exactly the same python among multiple machines. But on one machine, it prints unbuffered (at least it prints something before ending), without any setting at all (no PYTHONUNBUFFERD). On other machines it does not, which makes me even more confusing.
My testing script is
import os,sys,time
for i in range(100):
print(i)
time.sleep(1)
And
import os,sys,time
sys.stdout = os.fdopen(sys.stdout.fileno(),'w',0)
for i in range(100):
print(i)
time.sleep(1)
And
import os,sys,time
for i in range(100):
print(i)
sys.stdout.flush()
time.sleep(1)
Is there any other way to make Python print unbuffered (or buffered more reasonably, like buffer for one line) without modifing Python codes (because there are a lot of them)? Thanks.
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
I'm writing a parser in Python that outputs a bunch of database rows to standard out. In order for the DB to process them properly, each row needs to be fully printed to the console. I'm trying to prevent interrupts from making the print command stop halfway through printing a line.
I tried the solution that recommended using a signal handler override, but this still doesn't prevent the row from being partially printed when the program is interrupted. (I think the WRITE system call is cancelled to handle the interrupt).
I thought that the problem was solved by issue 10956 but I upgraded to Python 2.7.5 and the problem still happens.
You can see for yourself by running this example:
# Writer
import signal
interrupted = False
def signal_handler(signal, frame):
global interrupted
iterrupted = True
signal.signal(signal.SIGINT, signal_handler)
while True:
if interrupted:
break
print '0123456789'
In a terminal:
$ mkfifo --mode=0666 pipe
$ python writer.py > pipe
In another terminal:
$ cat pipe
Then Ctrl+C the first terminal. Some of the time the second terminal will end with an incomplete sequence of characters.
Is there any way of ensuring that full lines are written?
This seems less like an interrupt problem per se then a buffering issue. If I make a small change to your code, I don't get the partial lines.
# Writer
import sys
while True:
print '0123456789'
sys.stdout.flush()
It sounds like you don't really want to catch a signal but rather block it temporarily. This is supported by some *nix flavours. However Python explicitly does not support this.
You can write a C wrapper for sigmasks or look for a library. However if you are looking for a portable solution...