This question already has answers here:
Disable output buffering
(16 answers)
Closed 8 years ago.
I have a python script that looks something like this:
for item in collection:
print "what up"
#do complicated stuff that takes a long time.
In bash, I run this script by doing the following:
$ python my.py | tee my_file.txt
However, all I see in bash is a blank line until the program finishes. Then, all of the print statements come all at one.
Is this the expected operation of tee? Can I use tee to see the output in real-time?
Python, like many programs, tries to minimize the number of times it calls the write system call. It does this by collecting the output of several print statements before it actually writes them to its standard output file. This process is called buffering the output.
When Python is connected to a terminal, it doesn't buffer its output. This makes sense, because the human at the terminal wants to see the output right away.
When Python is writing to a file (or a pipe), it does buffer its output. This also makes sense, because no one will see the output until the process is complete
You can defeat this optimization by calling sys.stdout.flush() whenever you want to force Python to write its buffered output to its standard output file.
In your case, try this:
import sys
...
for item in collection:
print "what up"
sys.stdout.flush()
#do complicated stuff that takes a long time.
Related
I've got a Python script that uses os.system to run shell commands. The output from those commands is echoed to the screen; I like this and need to keep it. I would also like my script to be able to take action based on the contents of the output from the system call. How can I do this?
In my specific case, I'm calling os.system("svn update"). I need the output to go to the screen and (in case of conflicts, for example), the user needs to be able to interact with svn. I would like to be able to have the script take action based on the output - to trigger a build if it sees that a build script was updated, for example.
I'd prefer not to handle the I/O myself (that would seem unnecessarily complex) and I'd rather not send the output to a temporary file that I have to clean up later (though I will if I must).
Edit:
Here's my test script:
#!/usr/bin/python -t
import subprocess
output = subprocess.check_output(["echo","one"])
print "python:", output
output = subprocess.check_output(["echo", "two"], shell=True)
print "python:", output
output = subprocess.check_output("echo three", shell=True)
print "python:", output
and here's its output:
$ ./pytest
python: one
python:
python: three
(There's an extra blank line at the end that the code block doesn't show.) I expect something more like:
$ ./pytest
one
python: one
two
python:
three
python: three
To run a process, I would look into subprocess.check_output. In this case, something like:
output = subprocess.check_output(['svn','update'])
print output
This only works on python2.7 or newer though. If you want a version which works with older versions of python:
p = subprocess.Popen(['svn','update'],stdout=subprocess.PIPE)
output,stderr = p.communicate()
print output
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
I have a python script that performs a simulation. It takes a fairly long, varying time to run through each iteration, so I print a . after each loop as a way to monitor how fast it runs and how far it went through the for statement as the script runs. So the code has this general structure:
for step in steps:
run_simulation(step)
# Python 3.x version:
print('.', end='')
# for Python 2.x:
# print '.',
However, when I run the code, the dots do not appear one by one. Instead, all the dots are printed at once when the loop finishes, which makes the whole effort pointless. How can I print the dots inline as the code runs?
This problem can also occur when iterating over data fed from another process and trying to print results, for example to echo input from an Electron app. See Python not printing output.
The issue
By default, output from a Python program is buffered to improve performance. The terminal is a separate program from your code, and it is more efficient to store up text and communicate it all at once, rather than separately asking the terminal program to display each symbol.
Since terminal programs are usually meant to be used interactively, with input and output progressing a line at a time (for example, the user is expected to hit Enter to indicate the end of a single input item), the default is to buffer the output a line at a time.
So, if no newline is printed, the print function (in 3.x; print statement in 2.x) will simply add text to the buffer, and nothing is displayed.
Outputting in other ways
Every now and then, someone will try to output from a Python program by using the standard output stream directly:
import sys
sys.stdout.write('test')
This will have the same problem: if the output does not end with a newline, it will sit in the buffer until it is flushed.
Fixing the issue
For a single print
We can explicitly flush the output after printing.
In 3.x, the print function has a flush keyword argument, which allows for solving the problem directly:
for _ in range(10):
print('.', end=' ', flush=True)
time.sleep(.2) # or other time-consuming work
In 2.x, the print statement does not offer this functionality. Instead, flush the stream explicitly, using its .flush method. The standard output stream (where text goes when printed, by default) is made available by the sys standard library module, and is named stdout. Thus, the code will look like:
for _ in range(10):
print '.',
sys.stdout.flush()
time.sleep(.2) # or other time-consuming work
For multiple prints
Rather than flushing after every print (or deciding which ones need flushing afterwards), it is possible to disable the output line buffering completely. There are many ways to do this, so please refer to the linked question.
I've got a Python script that uses os.system to run shell commands. The output from those commands is echoed to the screen; I like this and need to keep it. I would also like my script to be able to take action based on the contents of the output from the system call. How can I do this?
In my specific case, I'm calling os.system("svn update"). I need the output to go to the screen and (in case of conflicts, for example), the user needs to be able to interact with svn. I would like to be able to have the script take action based on the output - to trigger a build if it sees that a build script was updated, for example.
I'd prefer not to handle the I/O myself (that would seem unnecessarily complex) and I'd rather not send the output to a temporary file that I have to clean up later (though I will if I must).
Edit:
Here's my test script:
#!/usr/bin/python -t
import subprocess
output = subprocess.check_output(["echo","one"])
print "python:", output
output = subprocess.check_output(["echo", "two"], shell=True)
print "python:", output
output = subprocess.check_output("echo three", shell=True)
print "python:", output
and here's its output:
$ ./pytest
python: one
python:
python: three
(There's an extra blank line at the end that the code block doesn't show.) I expect something more like:
$ ./pytest
one
python: one
two
python:
three
python: three
To run a process, I would look into subprocess.check_output. In this case, something like:
output = subprocess.check_output(['svn','update'])
print output
This only works on python2.7 or newer though. If you want a version which works with older versions of python:
p = subprocess.Popen(['svn','update'],stdout=subprocess.PIPE)
output,stderr = p.communicate()
print output
I have a ruby script that gets executed by a python script. From within the python script I want to access to return value of the ruby function.
Imagine, I would have this ruby script test.rb:
class TestClass
def self.test_function(some_var)
if case1
puts "This may take some time"
# something is done here with some_var
puts "Finished"
else
# just do something short with some_var
end
return some_var
end
end
Now, I want to get the return value of that function into my python script, the printed output should go to stdout.
I tried the following (example 1):
from subprocess import call
answer = call(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
However, this gives me the whole output on stdout and answer is just the exit code.
Therefore i tried this (example 2):
from subprocess import check_output
answer = check_output(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
This gives me the return value of the function in the else case (see test.rb) almost immediately. However, if case1 is true, answer contains the whole output, but while running test.rb nothing gets printed.
Is there any way to get the return value of the ruby function and the statements printed to stdout? Ideally, the solution requires no additional modules to install. Furthermore, I can't change the ruby code.
Edit:
Also tried this, but this also gives no output on stdout while running the ruby script (example 3):
import subprocess
process = subprocess.Popen(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"], stdout=subprocess.PIPE)
answer = process.communicate()
I also think that this is no matter of flushing the output to stdout in the ruby script. Example 1 gives me the output immediately.
Another way of doing this, without trying to call the ruby script as an external process is to set up a xmlrpc (or jsonrpc) server with the Ruby script, and call the remote functions from Python jsonrpc client (or xmlrpc)- the value would be available inside the Python program, nad even the sntax used would be just like you were dealing with a normal Python function.
Setting up such a server to expose a couple of functions remotely is very easy in Python, and should be the same from Ruby, but I had never tried it.
Check out http://docs.python.org/library/subprocess.html#popen-constructor and look into the ruby means of flushing stdout.