I have a python script test.py:
print "first"
import os
os.system("echo second")
On linux command line I execute
python test.py
which returns:
first
second
I then execute
python test.py > test.out; cat test.out
which returns
second
first
What about redirecting the output makes the os.system call print before the print statement??
When you're outputting to a pipe, Python buffers your output that's written to sys.stdout and outputs it after a flush, or after it's overflown, or upon closing (when the program exits). While it'll buffer the print calls, system calls output directly into stdout and their output won't be buffered. That's why you're seeing such precedence. To avoid that, use python -u:
python -u test.py > test.out; cat test.out
See more info here.
EDIT: explanation on when the buffer will be flushed.
Another way to prevent the os buffering is to flush the output after the first print:
#!/usr/bin/env python
import sys
print "first"
sys.stdout.flush()
import os
os.system("echo second")
When the output of the python script is a tty, its output is line buffered. When the output is a regular file, the output is block buffered.
Related
I have the following code in file abc.py:
import subprocess
def evaluate():
for i in range(5):
print("Printing", i)
subprocess.call(['ls', '-lrt'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
evaluate()
Now when I call using python abc.py > out.dat, the output file contains the result of 'ls -lrt' five times which is followed by printing statements in the code.
Why is it happening so and what should I do if I need to get output as:
printing 0
(results of ls -lrt here)
~~~~~~~~~~~~~~~~~~~~~~~
printing 1
.
.
.
and so on..?
Thank you..
You need to flush your stream before you call a subprocess:
import subprocess, sys
def evaluate():
for i in range(5):
print("Printing", i)
sys.stdout.flush()
subprocess.call(['ls', '-lrt'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
evaluate()
Flushing takes automatically place line-by-line as long as you write to a terminal (without redirection). Then you do not notice the problem because the print() flushes at the newline.
As soon as you redirect your output to a file the print() statement notices this and flushes automatically only when a certain buffer is full (and at termination time).
The subprocess (ls) does the same but uses its own buffer. And it terminates first, so its output is in the file first.
Usually I'd like to run my python code in linux as the following:
nohup python test.py > nohup.txt 2>&1 &
in the file test.py, I often use print to print out some messages to stdout. But actually I have to wait very long time then could see the messages were printed out to nohup.txt. How can I make it print out quickly.
You could call flush on stdout. If it is possible and practical for you to adjust your code to flush your buffers after the print call, in test.py:
from sys import stdout
from time import sleep
def log():
while True:
print('Test log message')
# flush your buffer
stdout.flush()
sleep(1)
While running this logging test, you can check the nohup.txt file and see the messages being printed out in realtime.
just make sure you have coreutils installed
stdbuf -oL nohup python test.py > nohup.txt 2>&1 &
this just sets buffering to off for this command ... you should see immediate output
(you might need nohup before stdbuf ... im not sure exactly)
alternatively ... just put at the top of test.py
import sys
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Write a simple python script test.py:
import time
print "begin"
time.sleep(10)
print "stop"
In bash, run
python test.py > log.txt
What I observe is that both "begin" and "stop" appear in log.txt at the same time, after 10 seconds.
Is this expected behavior?
Have you tried calling python with the -u option, which "forces stdin, stdout and stderr to be totally unbuffered" =>
python -u test.py > log.txt
You need to flush the buffer after printing.
import sys
import time
print "begin"
sys.stdout.flush()
time.sleep(10)
print "end"
sys.stdout.flush()
Or in Python 3:
# This can also be done in Python 2.6+ with
# from __future__ import print_function
import time
print("begin", flush=True)
time.sleep(10)
print("end", flush=True)
This is because the actual writing happens only when a buffer is full, or when the file is closed. When the program ends (after 10s), the buffer is emptied, the data gets written and the file closed.
I'm using Python 2.7.6 and IDLE on Windows 7.
I have 2 Python scripts:
script.py:
import subprocess, os, sys
print("hello 1")
mypath = os.path.abspath(__file__)
mydir = os.path.dirname(mypath)
start = os.path.join(mydir, "script2.py")
subprocess.call([sys.executable, start, "param"])
print("bye 1")
and script2.py that is being called by the previous script:
import sys
print "hello 2"
print (sys.argv[1])
print "bye 2"
If I run script.py with cmd.exe shell I get the expected result:
C:\tests>python ./script.py
hello 1
hello 2
param
bye 2
bye 1
But if I open script.py with the IDLE editor and run it with F5 I get this result:
>>> ================================ RESTART ================================
>>>
hello 1
bye 1
>>>
Why is the sub script not writing to the IDLE Python shell?
You're running the subprocess without providing any stdout or stderr.
When run in a terminal, the subprocess will inherit your stdout and stderr, so anything it prints will show up intermingled with your output.
When run in IDLE, the subprocess will also inherit your stdout and stderr, but those don't go anywhere. IDLE intercepts the Python-level wrappers sys.stdout and sys.stderr,* so anything you print to them from within Python will end up in the GUI window, but anything that goes to real stdout or stderr—like the output of any subprocess you run that inherits your streams—just goes nowhere.**
The simplest fix is to capture the stdout and stderr from the subprocess and print them yourself. For example:
out = subprocess.check_output([sys.executable, start, "param"],
stderr=subprocess.STDOUT)
print out
* IDLE is more complicated than it looks. It's actually running separate processes for the GUI window and for running your code, communicating over a socket. The sys.stdout (and likewise for the others) that IDLE provides for your script isn't a file object, it's a custom file-like object that redirects each write to the GUI process via remote procedure call over the socket.
** Actually, if you launched IDLE from a terminal rather than by double-clicking its icon, the subprocess's output may end up there. I'm not sure how it works on Windows. But at any rate, that isn't helpful to you.
I verified that abamert's change works in 2.7, on Win7, with Idle started normally from the icon. The slight glitch is that 'print out' inserts an extra blank line. This is easily changed by making print a function with a future import and use of the end parameter.
from __future__ import print_function
...
print(out, end='')
With Python 3, there is an additional issue that 'out' is bytes instead of str, so that it prints as
b'hello 2\r\nparam\r\nbye 2\r\n'
Since your output is all ascii, this can be fixed by changing the print call to
print(out.decode(), end='')
The resulting program works identically in 2.7 and 3.x.
I got a simple python script which should read from stdin.
So if I redirect a stdout of a program to the stdin to my python script.
But the stuff that's logged by my program to the python script will only "reach" the python script when the program which is logging the stuff gets killed.
But actually I want to handle each line which is logged by my program as soon as it is available and not when my program which should actually run 24/7 quits.
So how can I make this happen? How can I make the stdin not wait for CTRL+D or EOF until they handle data?
Example
# accept_stdin.py
import sys
import datetime
for line in sys.stdin:
print datetime.datetime.now().second, line
# print_data.py
import time
print "1 foo"
time.sleep(3)
print "2 bar"
# bash
python print_data.py | python accept_stdin.py
Like all file objects, the sys.stdin iterator reads input in chunks; even if a line of input is ready, the iterator will try to read up to the chunk size or EOF before outputting anything. You can work around this by using the readline method, which doesn't have this behavior:
while True:
line = sys.stdin.readline()
if not line:
# End of input
break
do_whatever_with(line)
You can combine this with the 2-argument form of iter to use a for loop:
for line in iter(sys.stdin.readline, ''):
do_whatever_with(line)
I recommend leaving a comment in your code explaining why you're not using the regular iterator.
It is also an issue with your producer program, i.e. the one you pipe stdout to your python script.
Indeed, as this program only prints and never flushes, the data it prints is kept in the internal program buffers for stdout and not flushed to the system.
Add sys.stdout.flush() call right after you print statement in print_data.py.
You see the data when you quit the program as it automatically flushes on exit.
See this question for explanation,
As said by #user2357112 you need to use:
for line in iter(sys.stdin.readline, ''):
After that you need to start python with the -u flag to flush stdin and stdout immediately.
python -u print_data.py | python -u accept_stdin.py
You can also specify the flag in the shebang.