I've written a python script to process some data. The script accepts two parameters, one for the input file and second for the output file. I want to extend to allow writing to stdout so output can be piped to other processes. My problem is that currently I also output progress information during processing as it can take some time. Ideally, I'd like to only suppress progress information if the script is configured to output to stdout. One advantage is that all the real output is emitted at once at the end of execution and not interspersed during execution.
The way I see it have two options, both of which I've tried (and work) but not sure they're the most pythonic way. Either I can overload calls to print() or I can redirect stdout to /dev/null during processing when the output file is configured to stdout.
Overload print()
if args.output_file.name == '<stdout>':
def silent(*args, **kwargs):
return
global print
print = silent
Redirect stdout
if args.output_file.name == '<stdout>':
out = open('/dev/null', 'w')
else:
out = sys.stdout
with redirect_stdout(out):
data = process_data()
write_output(data, args.output_file)
Overloading print() seems the least pythonic but at least it only effects calls to print() and not writing to stdout as a file. However, if I wanted to print() to stderr then that would also be suppressed.
Redirecting stdout seems cleaner except for the fact where I'm redirecting stdout to stdout if the script is writing to a regular file.
Related
I am including in my python code a function compiled in c via a cython wrapper. I have to take that function as given and cannot change it. Unfortunately, when I run that function, I see output that is bothering me.
I have tried a lot of tricks that are supposed to get rid of it, all of which play with sys.stdout or sys.stderr -- most noteably, the new contextlib.redirect_stdout. However, nothing I tried managed to silence the output.
At the most basic level, I simply tried setting
sys.stdout = open(os.devnull, 'w')
sys.stderr = open(os.devnull, 'w')
Which is not a safe, or practicable way of doing it, but it should shut the function up. Unfortunately, I can still see the output. What am I missing? Is there perhaps another "output type" besides stdout that this function might be using?
If it helps, I am inside a Pycharm debugging session and see this output in my debugging console.
Updated question to reflect that changing stderr did not help
A C function prints to a file descriptor (1 for stdout, 2 for stderr). If you want to prevent the printing, redirect that FD, that can be done also temporarily. Here is a litte demo:
import os
STDOUT = 1
saved_fd = os.dup(STDOUT)
null_fd = os.open(os.devnull, os.O_WRONLY)
os.dup2(null_fd, STDOUT)
os.system('echo TESTÂ 1') # redirected to /dev/null
os.dup2(saved_fd, STDOUT)
os.system('echo TEST 2') # normal
# note: close the null_fd, saved_fd when no longer needed
If the C code opens the terminal device itself, there is very little you can do to prevent it. But that would be very unusual (I would even say a bug) unless there is a specific reason to do so.
Is there perhaps another "output type" besides stdout that this
function might be using?
Yes, there exist stderr, which would be unaffected by stdout redirect, simple example, let printer.py content be
import sys
sys.stderr.write("printing to stderr")
then running in terminal
python printer.py > output.txt
lead to appearance of
printing to stderr
as > output.txt redirects only stdout.
I am trying to run a python file that prints something, waits 2 seconds, and then prints again. I want to catch these outputs live from my python script to then process them. I tried different things but nothing worked.
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if process.poll() is not None and output == '':
break
if output:
print(output.strip())
I'm at this point but it doesn't work. It waits until the code finishes and then prints all the outputs.
I just need to run a python file and get live outputs from it, if you have other ideas for doing it, without using the print function let me know, just know that I have to run the file separately. I just thought of the easiest way possible but, from what I'm seeing it can't be done.
There are three layers of buffering here, and you need to limit all three of them to guarantee you get live data:
Use the stdbuf command (on Linux) to wrap the subprocess execution (e.g. run ['stdbuf', '-oL'] + cmd instead of just cmd), or (if you have the ability to do so) alter the program itself to either explicitly change the buffering on stdout (e.g. using setvbuf for C/C++ code to switch stdout globally to line-buffered mode, rather than the default block buffering it uses when outputting to a non-tty) or to insert flush statements after critical output (e.g. fflush(stdout); for C/C++, fileobj.flush() for Python, etc.) the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in user-mode buffers of the sub-process.
Add bufsize=0 to the Popen arguments (probably not needed since you don't send anything to stdin, but harmless) so it unbuffers all piped handles. If the Popen is in text=True mode, switch to bufsize=1 (which is line-buffered, rather than unbuffered).
Add flush=True to the print arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter), or explicitly call sys.stdout.flush().
Between the three of these, you should be able to guarantee no data is stuck waiting in user-mode buffers; if at least one line has been output by the sub-process, it will reach you immediately, and any output triggered by it will also appear immediately. Item #1 is the hardest in most cases (when you can't use stdbuf, or the process reconfigures its own buffering internally and undoes the effect of stdbuf, and you can't modify the process executable to fix it); you have complete control over #2 and #3, but #1 may be outside your control.
This is the code I use for that same purpose:
def run_command(command, **kwargs):
"""Run a command while printing the live output"""
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
**kwargs,
)
while True: # Could be more pythonic with := in Python3.8+
line = process.stdout.readline()
if not line and process.poll() is not None:
break
print(line.decode(), end='')
An example of usage would be:
run_command(['git', 'status'], cwd=Path(__file__).parent.absolute())
process = subprocess.check_output(BACKEND+"mainbgw setup " + str(NUM_USERS), shell=True,\
stderr=subprocess.STDOUT)
I am using the above statement to run a C program in django-python based server for some computations, there are some printf() statements whose output I would like to see on stdout while the server is running and executing the subprocess, how can that be done ?
If you actually don't need the output to be available to your python code as a string, you can just use os.system, or subprocess.call without redirecting stdout elsewhere. Then stdout of your C program will just go directly to stdout of your python program.
If you need both streaming stdout and access to the output as a string, you should use subprocess.Popen (or the old popen2.popen4) to obtain a file descriptor of the output stream, then repeatedly read lines from the stream until you exhausted it. In the mean time, you keep a concatenated version of all data you grabbed. This is an example of the loop.
Python newbie here. I'm writing a script that can dump some output to either a file or stdout, depending on the arguments passed to it. When interpreting arguments, I assign either an open'ed file or stdout to a global variable named output_file, which can be used by the rest of the script to write output regardless of what type of stream was selected. At the very end of the script I close output_file. This is proper to do for a file stream, and though it's redundant for stdout, my experience with other programming languages suggests that there's no harm in explicitly closing stdout immediately before the program ends.
However, whenever stdout is used for output (and subsequently closed), I get a "ValueError: 'I/O operation on closed file.'". I know this error is not directly produced by my call to close stdout, but occurs after my script returns. My question is: Why does this happen, and is there a way to manually close stdout without triggering it? (I'm aware that I can easily work around the problem by conditionally closing the stream only when a file was selected, but I want to know if/why this is necessary.)
Very simple demonstrative snippet:
from sys import stdout
stdout.close()
The problem is that on python-3.2 there's an attempt at shutdown to flush stdout without checking if it was closed.
The issue13444 is about this.
You shouldn't have this problem in python-2.7 in releases after the fixing patch.
Once you've closed stdout in this manner, you have to be absolutely sure that nothing will attempt to print anything to stdout. If something does, you get that exception.
My recommendation would be to close output_file conditionally:
if output_file != sys.stdout:
output_file.close()
edit Here is an example where sys.stdout is closed right at the very end of the script, and that nonetheless produces a ValueError: 'I/O operation on closed file when run.
import atexit
#atexit.register
def goodbye():
print "You are now leaving the Python sector."
import sys
sys.stdout.close()
Before closing you can check output_file.closed file:
if not output_file.closed:
output_file.close()
And make sure you have no I/O calls to output_file after closing.
Two things seem necessary to avoid this error: reset (i) reset stdout; (ii) don't close stdout, close the file to which it was redirected.
f=open(filename, 'w')
sys.stdout = f
print("la la-la"), file = sys.stdout)
f.close()
sys.stdout = sys.__stdout__
Various solutions to this problem suggest copying the 'original' stdout pointer to a variable before assigning stdout to a file (i.e. original = stdout ... stdout = f) and then copying it back afterwards (stdout = original). But they neglect to mention the final operation in their routine, which is wasted hours pulling your hair out.
Found the solution here.
I'm writing a wrapper class for use with a workflow manager. I would like to log output from an application (child process executed via subprocess.Popen) in a certain way:
stdout of the child should go to a log file and to stdout of the parent,
stderr of the child should go to a different logfile, but also to stdout of the parent.
I.e. all output from the child should end up merged on stdout (like with subprocess.Popen(..., stderr=subprocess.STDOUT), so I can reserve stderr for log messages from the wrapper itself. On the other hand, the child's streams should go to different files to allow separate validation.
I've tried using a "Tee" helper class to tie two streams (stdout and the log file) together, so that Tee.write writes to both streams. However, this cannot be passed to Popen because "subprocess" uses OS-level functions for writing (see here: http://bugs.python.org/issue1631).
The problem with my current solution (code snippet below, adapted mostly from here) is that output on stdout may not appear in the right order.
How can I overcome this? Or should I use an altogether different approach?
(If I stick with the code below, how do I choose a value for the number of bytes in os.read?)
import subprocess, select, sys, os
call = ... # set this
process = subprocess.Popen(call, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
logs = {process.stdout: open("out.log", "w"), process.stderr: open("err.log", "w")}
done = {process.stdout: False, process.stderr: False}
while (process.poll() is None) or (not all(done.values())):
ready = select.select([process.stdout, process.stderr], [], [])[0]
for stream in ready:
data = os.read(stream.fileno(), 1)
if data:
sys.stdout.write(data)
logs[stream].write(data)
else:
done[stream] = True
logs[process.stdout].close()
logs[process.stderr].close()
By the way, this solution using "fcntl" has not worked for me. And I couldn't quite figure out how to adapt this solution to my case yet, so I haven't tried it.
If you set shell=True, you can pass a command string to subprocess that includes pipes, redirections, and the tee command.