I've developed a Python script that performs several tasks in a row (mainly connecting to servers and retrieving information).
There are many steps, and for each of them I would like to display a dot, so that the user knows there is something going on.
At the end of each step, I do:
print('.', end='')
And in the final step, I write:
print('Done!')
It works, except nothing is displayed until the final print is executed, so it kind of defeats its original purpose :)
Basically, nothing is displayed on the screen, and at the very last moment, this pops up:
.......Done!
How can I force Python to print on the same line step after step?
By default, stdout is line buffered, meaning the buffer won't be flushed until you write a newline.
Flush the buffer explicitly each time you print a '.':
print('.', end='', flush=True)
The flush keyword was added in Python 3.3; for older versions, use sys.stdout.flush().
From the print() function documentation:
Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.
and from the sys.stdout documentation (the default value for the file argument of the print() function):
When interactive, standard streams are line-buffered. Otherwise, they are block-buffered like regular text files.
Instead of using print, you can write directly to stdout, (unbuffered):
import sys
import time
for i in range (10):
time.sleep (0.5)
sys.stdout.write('.')
print 'Done!'
for python 2.7.3 you can left a trailing comma which tells the idle to not insert a line after the statement is printed for example
print "hello",
print "world"
would return
>>> hello world
Related
I am running the following on Mac OSX and Python 3. I am running the snippet using Sublime Text 3's default build system for Python 3 and displaying the output in the same program's output window.
print('Starting test.')
for i in range(1,100):
print('This is a test. ' + str(i), end='\r')
print('Test done!')
My desired output is basically a repetition of that same line, overwriting the previous line in the console each time. By the end, I should ideally only see the following in my output window.
Starting test.
This is a test. 99
Test done!
I would expect the end='\r' gets me to this behavior, but it doesn't. Instead, I see:
Starting test.
This is a test. 1
This is a test. 2
This is a test. 3
[...]
This is a test. 99
Test done!
For background, if you use View Package File from the command palette to view Default/exec.py you can examine the code that's being used to execute programs and display their output into the panel.
As a short synopsis, the program is executed in a background thread, and the output of stdout and stderr are read from using this code:
def read_fileno(self, fileno, execute_finished):
decoder_cls = codecs.getincrementaldecoder(self.listener.encoding)
decoder = decoder_cls('replace')
while True:
data = decoder.decode(os.read(fileno, 2**16))
if len(data) > 0:
if self.listener:
self.listener.on_data(self, data)
else:
try:
os.close(fileno)
except OSError:
pass
if execute_finished and self.listener:
self.listener.on_finished(self)
break
That is, whenever they read any data, they invoke on_data() in the object that's listening for results, which in this case is the exec command itself. The implementation of that method looks like this:
def on_data(self, proc, data):
# Normalize newlines, Sublime Text always uses a single \n separator
# in memory.
data = data.replace('\r\n', '\n').replace('\r', '\n')
self.append_string(proc, data)
Internally Sublime uses \n as the line separator, so on windows \r\n is interpreted as \n, and on MacOS \r is interpreted as \n as well.
So on the one hand this doesn't do what you expect because your \r is effectively treated the same as \n is.
Aside of that, if you trace the call to append_string() through the guard code that keeps it thread safe and allows it to interact with the Sublime core, it ends up in a line that looks like this:
self.output_view.run_command(
'append',
{'characters': characters, 'force': True, 'scroll_to_end': True})
That is, whatever characters you provide, they're directly appended to the output buffer without regard to what they might contain.
If you were to provide a custom build target (a somewhat advanced plugin but still possible) this behaviour could be modified, although it might be tricky to get it to work on MacOS since there would be no way to distinguish what \r means on that platform.
The code displays correctly in a normal command line window. The problem is with Sublime's output window, which doesn't seem to be displaying the carriage return correctly (it replaces it with a newline).
Actually you need to add '\r' at the beginning of the string.
print('Starting test.')
for i in range(1,100):
print('\rThis is a test.'+str(i), end="")
print('\nTest done!')
Output
Starting test.
This is a test.99
Test done!
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
This code:
import time
for i in range(10):
print(i)
time.sleep(.5)
Causes my computer to hang for 5 seconds, and then print out 0-9, as opposed to printing a digit every half second. Am I doing something wrong?
print, by default, prints to sys.stdout and that buffers the output to be printed, internally.
Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.
Changed in version 3.3: Added the flush keyword argument.
Quoting sys.stdout's documentation,
When interactive, standard streams are line-buffered. Otherwise, they are block-buffered like regular text files.
So, in your case, you need to explicitly flush, like this
import time
for i in range(10):
print(i, flush=True)
time.sleep(.5)
Okay, there is a lot of confusion around this buffering. Let me explain as much as possible.
First of all, if you are trying this program in a terminal, they do line buffering (which basically means, whenever you encounter a newline character, send the buffered data to stdout), by default. So, you can reproduce this problem in Python 2.7, like this
>>> import time
>>> for i in range(10):
... print i,
... time.sleep(.5)
...
And in Python 3.x,
>>> for i in range(10):
... print(i, end='')
... time.sleep(.5)
We pass end='' because, the default end value is \n, as per the print's documentation,
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
Since the default end breaks the line buffering, the data will be sent to stdout immediately.
Another way to reproduce this problem is to store the actual program given by OP in a file and execute with Python 3.x interpreter, you will see that the stdout internally buffers the data and waits till the program finishes to print.
Try this:
for i in range(10):
sys.stdout.write('\r' + str(i))
time.sleep(.5)
here '/r' is carriage return, it brings the cursor first place again.
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
I got a simple python script which should read from stdin.
So if I redirect a stdout of a program to the stdin to my python script.
But the stuff that's logged by my program to the python script will only "reach" the python script when the program which is logging the stuff gets killed.
But actually I want to handle each line which is logged by my program as soon as it is available and not when my program which should actually run 24/7 quits.
So how can I make this happen? How can I make the stdin not wait for CTRL+D or EOF until they handle data?
Example
# accept_stdin.py
import sys
import datetime
for line in sys.stdin:
print datetime.datetime.now().second, line
# print_data.py
import time
print "1 foo"
time.sleep(3)
print "2 bar"
# bash
python print_data.py | python accept_stdin.py
Like all file objects, the sys.stdin iterator reads input in chunks; even if a line of input is ready, the iterator will try to read up to the chunk size or EOF before outputting anything. You can work around this by using the readline method, which doesn't have this behavior:
while True:
line = sys.stdin.readline()
if not line:
# End of input
break
do_whatever_with(line)
You can combine this with the 2-argument form of iter to use a for loop:
for line in iter(sys.stdin.readline, ''):
do_whatever_with(line)
I recommend leaving a comment in your code explaining why you're not using the regular iterator.
It is also an issue with your producer program, i.e. the one you pipe stdout to your python script.
Indeed, as this program only prints and never flushes, the data it prints is kept in the internal program buffers for stdout and not flushed to the system.
Add sys.stdout.flush() call right after you print statement in print_data.py.
You see the data when you quit the program as it automatically flushes on exit.
See this question for explanation,
As said by #user2357112 you need to use:
for line in iter(sys.stdin.readline, ''):
After that you need to start python with the -u flag to flush stdin and stdout immediately.
python -u print_data.py | python -u accept_stdin.py
You can also specify the flag in the shebang.