I'm writing a code to analyze PDF file. I want to display the output on the console as well as to have a copy of the output in a file, I used this code save the output in a file:
import sys
sys.stdout = open('C:\\users\\Suleiman JK\\Desktop\\file.txt',"w")
print "test"
but could I display the output into console as well but without using classes because I'm not good with them?
(This answer uses Python 3 and you will have to adapt it if you prefer Python 2.)
Start by importing the Python logging package (and sys for accessing the standard output stream):
import logging
import sys
In your entry point, set up a handler for both the standard output stream and your output file:
targets = logging.StreamHandler(sys.stdout), logging.FileHandler('test.log')
and configure the logging package to output only the message without the log level:
logging.basicConfig(format='%(message)s', level=logging.INFO, handlers=targets)
Now you can use it:
>>> logging.info('testing the logging system')
testing the logging system
>>> logging.info('second message')
second message
>>> print(*open('test.log'), sep='')
testing the logging system
second message
sys.stdout can point to any object that has a write method, so you can create a class that writes to both a file and the console.
import sys
class LoggingPrinter:
def __init__(self, filename):
self.out_file = open(filename, "w")
self.old_stdout = sys.stdout
#this object will take over `stdout`'s job
sys.stdout = self
#executed when the user does a `print`
def write(self, text):
self.old_stdout.write(text)
self.out_file.write(text)
#executed when `with` block begins
def __enter__(self):
return self
#executed when `with` block ends
def __exit__(self, type, value, traceback):
#we don't want to log anymore. Restore the original stdout object.
sys.stdout = self.old_stdout
print "Entering section of program that will be logged."
with LoggingPrinter("result.txt"):
print "I've got a lovely bunch of coconuts."
print "Exiting logged section of program."
Result:
Console:
Entering section of program that will be logged.
I've got a lovely bunch of coconuts.
Exiting logged section of program.
result.txt:
I've got a lovely bunch of coconuts.
This method may be preferable to codesparkle's in some circumstances, because you don't have to replace all your existing prints with logging.info. Just put everything you want logged into a with block.
You could make a function which prints both to console and to file. You can either do it by switching stdout, e.g. like this:
def print_both(file, *args):
temp = sys.stdout #assign console output to a variable
print ' '.join([str(arg) for arg in args])
sys.stdout = file
print ' '.join([str(arg) for arg in args])
sys.stdout = temp #set stdout back to console output
or by using file write method (I suggest using this unless you have to use stdout)
def print_both(file, *args):
toprint = ' '.join([str(arg) for arg in args])
print toprint
file.write(toprint)
Note that:
The file argument passed to the function must be opened outside of the function (e.g. at the beginning of the program) and closed outside of the function (e.g. at the end of the program). You should open it in append mode.
Passing *args to the function allows you to pass arguments the same way you do to a print function. So you pass arguments to print...
...like this:
print_both(open_file_variable, 'pass arguments as if it is', 'print!', 1, '!')
Otherwise, you'd have to turn everything into a single argument i.e. a single string. It would look like this:
print_both(open_file_variable, 'you should concatenate'+str(4334654)+'arguments together')
I still suggest you learn to use classes properly, you'd benefit from that very much. Hope this helps.
I was too lazy to write a function, so when I needed to print to both the console and file I wrote this quick and (not so) dirty code:
import sys
...
with open('myreport.txt', 'w') as f:
for out in [sys.stdout, f]:
print('some data', file=out)
print('some mre data', file=out)
Related
How do I redirect stdout to an arbitrary file in Python?
When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.
I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.
If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:
# for python3
import sys
with open('file', 'w') as sys.stdout:
print('test')
A far more common method is to use shell redirection when executing (same on Windows and Linux):
$ python3 foo.py > file
There is contextlib.redirect_stdout() function in Python 3.4+:
from contextlib import redirect_stdout
with open('help.txt', 'w') as f:
with redirect_stdout(f):
print('it now prints to `help.text`')
It is similar to:
import sys
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_target):
old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
try:
yield new_target # run some code with the replaced stdout
finally:
sys.stdout = old_target # restore to the previous value
that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.
It doesn't redirect the stdout at the file descriptors level e.g.:
import os
from contextlib import redirect_stdout
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
print('redirected to a file')
os.write(stdout_fd, b'not redirected')
os.system('echo this also is not redirected')
b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.
To redirect at the file descriptor level, os.dup2() could be used:
import os
import sys
from contextlib import contextmanager
def fileno(file_or_fd):
fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
if not isinstance(fd, int):
raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
return fd
#contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
if stdout is None:
stdout = sys.stdout
stdout_fd = fileno(stdout)
# copy stdout_fd before it is overwritten
#NOTE: `copied` is inheritable on Windows when duplicating a standard stream
with os.fdopen(os.dup(stdout_fd), 'wb') as copied:
stdout.flush() # flush library buffers that dup2 knows nothing about
try:
os.dup2(fileno(to), stdout_fd) # $ exec >&to
except ValueError: # filename
with open(to, 'wb') as to_file:
os.dup2(to_file.fileno(), stdout_fd) # $ exec > to
try:
yield stdout # allow code to be run with the redirected stdout
finally:
# restore stdout to its previous value
#NOTE: dup2 makes stdout_fd inheritable unconditionally
stdout.flush()
os.dup2(copied.fileno(), stdout_fd) # $ exec >&copied
The same example works now if stdout_redirected() is used instead of redirect_stdout():
import os
import sys
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
print('redirected to a file')
os.write(stdout_fd, b'it is redirected now\n')
os.system('echo this is also redirected')
print('this is goes back to stdout')
The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.
Note: stdout.flush() does not flush
C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:
try:
import ctypes
from ctypes.util import find_library
except ImportError:
libc = None
else:
try:
libc = ctypes.cdll.msvcrt # Windows
except OSError:
libc = ctypes.cdll.LoadLibrary(find_library('c'))
def flush(stream):
try:
libc.fflush(None)
stream.flush()
except (AttributeError, ValueError, IOError):
pass # unsupported
You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:
def merged_stderr_stdout(): # $ exec 2>&1
return stdout_redirected(to=sys.stdout, stdout=sys.stderr)
Example:
from __future__ import print_function
import sys
with merged_stderr_stdout():
print('this is printed on stdout')
print('this is also printed on stdout', file=sys.stderr)
Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.
To answer, your edit: you could use python-daemon to daemonize your script and use logging module (as #erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.
you can try this too much better
import sys
class Logger(object):
def __init__(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt
The other answers didn't cover the case where you want forked processes to share your new stdout.
To do that:
from os import open, close, dup, O_WRONLY
old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1
..... do stuff and then restore
close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs
Quoted from PEP 343 -- The "with" Statement (added import statement):
Redirect stdout temporarily:
import sys
from contextlib import contextmanager
#contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Used as follows:
with open(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.
import sys
sys.stdout = open('stdout.txt', 'w')
Here is a variation of Yuda Prawira answer:
implement flush() and all the file attributes
write it as a contextmanager
capture stderr also
.
import contextlib, sys
#contextlib.contextmanager
def log_print(file):
# capture all outputs to a log file while still printing it
class Logger:
def __init__(self, file):
self.terminal = sys.stdout
self.log = file
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def __getattr__(self, attr):
return getattr(self.terminal, attr)
logger = Logger(file)
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdout = logger
sys.stderr = logger
try:
yield logger.log
finally:
sys.stdout = _stdout
sys.stderr = _stderr
with log_print(open('mylogfile.log', 'w')):
print('hello world')
print('hello world on stderr', file=sys.stderr)
# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
# ....
# print('[captured output]', log.getvalue())
You need a terminal multiplexer like either tmux or GNU screen
I'm surprised that a small comment by Ryan Amos' to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they've received. Further to Ryan's comment, tmux is a nice alternative to GNU screen.
But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you'd never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won't have lost the last 18 hours-worth of processing!
Based on this answer: https://stackoverflow.com/a/5916874/1060344, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.
Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python's logging facility (but you can really do anything with this):
class FileToLogInterface(file):
'''
Interface to make sure that everytime anything is written to stderr, it is
also forwarded to a file.
'''
def __init__(self, *args, **kwargs):
if 'cfg' not in kwargs:
raise TypeError('argument cfg is required.')
else:
if not isinstance(kwargs['cfg'], config.Config):
raise TypeError(
'argument cfg should be a valid '
'PostSegmentation configuration object i.e. '
'postsegmentation.config.Config')
self._cfg = kwargs['cfg']
kwargs.pop('cfg')
self._logger = logging.getlogger('access_log')
super(FileToLogInterface, self).__init__(*args, **kwargs)
def write(self, msg):
super(FileToLogInterface, self).write(msg)
self._logger.info(msg)
Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.
A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null
See this post for more info: What is the reason for performing a double fork when creating a daemon?
I know this question is answered (using python abc.py > output.log 2>&1 ), but I still have to say:
When writing your program, don't write to stdout. Always use logging to output whatever you want. That would give you a lot of freedom in the future when you want to redirect, filter, rotate the output files.
As mentioned by #jfs, most solutions will not properly handle some types of stdout output such as that from C extensions. There is a module that takes care of all this on PyPI called wurlitzer. You just need its sys_pipes context manager. It's as easy as using:
from contextlib import redirect_stdout
import os
from wurlitzer import sys_pipes
log = open("test.log", "a")
with redirect_stdout(log), sys_pipes():
print("print statement")
os.system("echo echo call")
Based on previous answers on this post I wrote this class for myself as a more compact and flexible way of redirecting the output of pieces of code - here just to a list - and ensure that the output is normalized afterwards.
class out_to_lt():
def __init__(self, lt):
if type(lt) == list:
self.lt = lt
else:
raise Exception("Need to pass a list")
def __enter__(self):
import sys
self._sys = sys
self._stdout = sys.stdout
sys.stdout = self
return self
def write(self,txt):
self.lt.append(txt)
def __exit__(self, type, value, traceback):
self._sys.stdout = self._stdout
Used as:
lt = []
with out_to_lt(lt) as o:
print("Test 123\n\n")
print(help(str))
Updating. Just found a scenario where I had to add two extra methods, but was easy to adapt:
class out_to_lt():
...
def isatty(self):
return True #True: You're running in a real terminal, False:You're being piped, redirected, cron
def flush(self):
pass
There are other versions using context but nothing this simple. I actually just googled to double check it would work and was surprised not to see it, so for other people looking for a quick solution that is safe and directed at only the code within the context block, here it is:
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
Tested like so:
$ cat redirect_stdout.py
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
$ python redirect_stdout.py
$ cat test_file
Testing 1 2 3
I have some code that parses command line options using argparse.
For example:
# mycode.py
import argparse
def parse_args():
parser = argparse.ArgumentParser('my code')
# list of arguments
# ...
# ...
return vars(parser.parse_args())
if __name__ == "__main__":
parse_args()
I would like to use unittest to check the output of the help function. I also don't want to change the actual code unless there is no other solution.
The help action has a SystemExit call built into it after printing to stdout, so I have had to try and catch it in the unittest.
Here is my unittest code with the following steps:
1) Set the sys.argv list to include the -h flag.
2) Wrap the function call in a context manager to prevent the SystemExit being viewed as an error.
3) Switch the sys.stdout temporarily to an io.StringIO object so I can inspect it without having it print to screen.
4) Call the function in a try...finally block so the SystemExit isn't fatal.
5) Switch sys.stdout back to the real stdout.
6) Open a file to which I had previously saved the help text (by entering python mycode.py -h > help_out.txt in the terminal) to verify it is the same as the captured output from the StringIO.
import unittest
import mycode
import sys
import io
class TestParams(unittest.TestCase):
def setUp(self):
pass
def test_help(self):
args = ["-h"]
sys.argv[1:] = args
with self.assertRaises(SystemExit):
captured_output = io.StringIO()
sys.stdout = captured_output
try:
mycode.parse_args()
finally:
sys.stdout = sys.__stdout__
with open("help_out.txt", "r") as f:
help_text = f.read()
self.assertEqual(captured_output, help_text)
def tearDown(self):
pass
This code works, but the captured_output StringIO object is empty, so the test fails.
I am looking for an explanation as to what is going wrong with the captured output and/or an alternative solution.
I was very close. The captured_output wasn't actually empty - I just wasn't accessing the contents correctly.
Substitute captured_output.get_value() for captured_value in my example code and it works perfectly.
Initially, I've a simple program to print out the whole output to the console.
Initial Code to display output in the console only
import os, subprocess
print("1. Before")
os.system('ver')
subprocess.run('whoami')
print('\n2. After')
Output in console
1. Before
Microsoft Windows [Version 10]
user01
2. After
Then, I decided to have a copy on a log file (log.txt) too while maintaining the original output to the console.
So, this is the new code.
import os, subprocess, sys
old_stdout = sys.stdout
log_file = open("log.txt","w")
sys.stdout = log_file
print("1. Before") # This appear in message.log only, but NOT in console
os.system('ver') # This appear in console only, but NOT in message.log
subprocess.run('whoami') # This appear in console only, but NOT in message.log
print('\n2. After') # This appear in message.log only, but NOT in console
sys.stdout = old_stdout
log_file.close()
Unfortunately, this didn't really work as expected. Some of the output only displayed on the console (os.system('ver') and subprocess.run('whoami')) while the print() function was only displayed on log.txt file and not in the console anymore.
Output in console
Microsoft Windows [Version 10]
user01
Output in log.txt file
1. Before
2. After
I was hoping to get similar output on both console and log.txt file. Is this possible?
What's wrong with my new code? Please let me know how to fix this.
Desired Output in both console and log.txt file
1. Before
Microsoft Windows [Version 10]
user01
2. After
The most appropriate way to handle this is with logging. Here's an example:
This is the python 2.6+ and 3.x version of how you can do it. (Can't override print() before 2.6)
log = logging.getLogger()
log.setLevel(logging.INFO)
# How should our message appear?
formatter = logging.Formatter('%(message)s')
# This prints to screen
ch = log.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
log.addHandler(ch)
# This prints to file
fh = log.FileHandler('/path/to/output_file.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
log.addHandler(fh)
def print(*args, **kwargs):
log.DEBUG(*args)
That option allows you the capability of using logging levels. For instance, you can put debug logging throughout your code for when the application starts acting funky. Swap logLevel to logging.DEBUG and suddenly, you're getting that output to screen. Notice in the above example, we have 2 different logging levels, one for screen and another for file. Yes, this will produce different output to each destination. You can remedy this by changing both to use logging.INFO (or logging.DEBUG, etc). (See full docs relating to log levels here.)
In the above example, I've overridden print(), but I'd recommend instead that you just reference your framework using log.DEBUG('Variable xyz: {}'.format(xyz)) or log.INFO('Some stuff that you want printed.)
Full logging documentation.
There's another, easier way to do it with overriding, but not quite so robust:
try:
# Python 2
import __builtin__
except ImportError:
# Python 3
import builtins as __builtin__
logfile = '/path/to/logging_file.log'
def print(*args, **kwargs):
"""Your custom print() function."""
with open(logfile) as f_out:
f_out.write(args[0])
f_out.write('\n')
# Uncomment the below line if you want to tail the log or something where you need that info written to disk ASAP.
# f_out.flush()
return __builtin__.print(*args, **kwargs)
There is no magic done by system, file pointer such as stdout and stderr need to be treated differently by your code. For example, stdout is one of the file pointer, you can do it in below:
log_file_pointer = open('log.txt', 'wt')
print('print_to_fp', file=log_file_pointer)
# Note: the print function will actually call log_file_pointer.write('print_to_fp')
Based on your requirement, you want to make the magic function to handle more than one file pointer in single line, you need a wrapper function in below:
def print_fps(content, files=[]):
for fi in files:
print(content, file=fi)
# the argument `file` of print does zero magic, it can only handle one file pointer once.
Then, you can make the magic happen now (make the output in both screen and file.)
import sys
log_file_pointer = open('log.txt', 'wt')
print_fps('1. Before', files=[log_file_pointer, sys.stdout])
print_fps('\n2. After', files=[log_file_pointer, sys.stdout])
After finishing the print part, let's move on to system call. Running any command in the operating system, you will get the return in default system file pointers: stdout and stderr. In python3, you can get those result in bytes by subprocess.Popen. And while running below code, what you want should be the result in stdout.
import subprocess
p = subprocess.Popen("whoami", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
# stdout: b'user01'
# stdout: b''
Yet again, you can call the wrapper function written in above and make output in both stdout and targeted file_pointer.
print_fps(stdout, files=[log_file_pointer, sys.stdout])
Finally, combining all the code in above. (Plus one more convenient function.)
import subprocess, sys
def print_fps(content, files=[]):
for fi in files:
print(content, file=fi)
def get_stdout(command):
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
# Note: Original idea is to return raw stdout
# return stdout
# Based on the scenario of the #Sabrina, the raw bytes of stdout needs decoding in utf-8 plus replacing newline '\r\n' to be pure
return stdout.decode().replace('\r\n', '')
log_file_pointer = open('log.txt', 'wt')
print_fps('1. Before', files=[log_file_pointer, sys.stdout])
print_fps(get_stdout('ver'), files=[log_file_pointer, sys.stdout])
print_fps(get_stdout('whoami'), files=[log_file_pointer, sys.stdout])
print_fps('\n2. After', files=[log_file_pointer, sys.stdout])
Note: because the output of Popen is in bytes, you might need to do decode to remove b''. You can run stdout.decode() to decode bytes to utf-8 decoded str.*
So I am trying to write a decorator which will log the standard output, while print it as well. The whole point is to fork the output so that it will show in the console, as well as get logged. So this is what I have so far:
from time import time
import tempfile
import sys
import datetime
def printUsingTempFile(f):
def wrapper(*a, **ka):
with open('sysLog.txt', 'a') as logFile:
with tempfile.NamedTemporaryFile() as f:
sys.stdout = f
sys.stderr = f
retVal = f(*a, **ka)
f.flush(); f.seek(0); logFile.write('\n'.join(f.readlines()))
f.flush(); f.seek(0); sys.__stdout__.write('\n'.join(f.readlines()))
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
return retVal
return wrapper
if __name__ == '__main__':
#printUsingTempFile
def someFunc(abc='Hello!!!'):
'''
This is the docstring of the someFunc function.
'''
print abc
return 1
someFunc('Print something here')
I run it like so:
In [18]: !python utility.py
because if I run is by using the run function the stupid iPython window closes.
I am getting a new file called 'sysLog.txt' in the directory, but that is empty. And there is no output anywhere to be seen either. There is no error or anything. Just ... nothing! Getting a little frustrated here with this. Not sure how to debug this code :(
Edit:
I realized that I wasn't seeking twice so I updated the code. I tried the following code:
def printUsingTempFile1():
with open('sysLog.txt', 'a') as logFile:
with tempfile.NamedTemporaryFile() as f:
sys.stdout = f
sys.stderr = f
print 'This is something ...'
f.flush(); f.seek(0); logFile.write('\n'.join(f.readlines()))
f.flush(); f.seek(0); sys.__stdout__.write('\n'.join(f.readlines()))
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
And this function seems to be working ok.
Why are you using f as the name for the passed in function to the decorator as well as the name of the temporary file? I'm assuming that the call to f(*a, **ka) errors out and since you have redirected the error stream to the temp file, it gets closed without writing out anything and hence nothing on the console.
A tip for debugging these sort of situations is to start stripping pieces of code which doesn't work one by one and bringing it down to the bare minimum which works. And then again work your way up till it starts failing. :)
EDIT
I would also like to recommend running your code under a lint tool to catch issues like these. For e.g. the code posted below:
''' This is a test module '''
import tempfile
import sys
def print_tmp_file(arg):
''' Print out the things using temp file '''
def wrapper(*a, **ka):
''' Wrapper function '''
with open('sysLog.txt', 'a') as log_file:
with tempfile.NamedTemporaryFile() as arg:
sys.stdout = arg
sys.stderr = arg
ret_val = arg(*a, **ka)
arg.flush()
arg.seek(0)
log_file.write('\n'.join(arg.readlines()))
arg.flush()
arg.seek(0)
sys.__stdout__.write('\n'.join(arg.readlines()))
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
return ret_val
return wrapper
if __name__ == '__main__':
#print_tmp_file
def some_func(abc='Hello!!!'):
''' This is the docstring of the someFunc function. '''
print abc
return 1
some_func('Print something here')
Running pylint on the above code gives the following warning:
W: 6,19: Unused argument 'arg' (unused-argument)
which is a clear indication that something has gone wrong.
How do I redirect stdout to an arbitrary file in Python?
When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.
I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.
If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:
# for python3
import sys
with open('file', 'w') as sys.stdout:
print('test')
A far more common method is to use shell redirection when executing (same on Windows and Linux):
$ python3 foo.py > file
There is contextlib.redirect_stdout() function in Python 3.4+:
from contextlib import redirect_stdout
with open('help.txt', 'w') as f:
with redirect_stdout(f):
print('it now prints to `help.text`')
It is similar to:
import sys
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_target):
old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
try:
yield new_target # run some code with the replaced stdout
finally:
sys.stdout = old_target # restore to the previous value
that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.
It doesn't redirect the stdout at the file descriptors level e.g.:
import os
from contextlib import redirect_stdout
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
print('redirected to a file')
os.write(stdout_fd, b'not redirected')
os.system('echo this also is not redirected')
b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.
To redirect at the file descriptor level, os.dup2() could be used:
import os
import sys
from contextlib import contextmanager
def fileno(file_or_fd):
fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
if not isinstance(fd, int):
raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
return fd
#contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
if stdout is None:
stdout = sys.stdout
stdout_fd = fileno(stdout)
# copy stdout_fd before it is overwritten
#NOTE: `copied` is inheritable on Windows when duplicating a standard stream
with os.fdopen(os.dup(stdout_fd), 'wb') as copied:
stdout.flush() # flush library buffers that dup2 knows nothing about
try:
os.dup2(fileno(to), stdout_fd) # $ exec >&to
except ValueError: # filename
with open(to, 'wb') as to_file:
os.dup2(to_file.fileno(), stdout_fd) # $ exec > to
try:
yield stdout # allow code to be run with the redirected stdout
finally:
# restore stdout to its previous value
#NOTE: dup2 makes stdout_fd inheritable unconditionally
stdout.flush()
os.dup2(copied.fileno(), stdout_fd) # $ exec >&copied
The same example works now if stdout_redirected() is used instead of redirect_stdout():
import os
import sys
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
print('redirected to a file')
os.write(stdout_fd, b'it is redirected now\n')
os.system('echo this is also redirected')
print('this is goes back to stdout')
The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.
Note: stdout.flush() does not flush
C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:
try:
import ctypes
from ctypes.util import find_library
except ImportError:
libc = None
else:
try:
libc = ctypes.cdll.msvcrt # Windows
except OSError:
libc = ctypes.cdll.LoadLibrary(find_library('c'))
def flush(stream):
try:
libc.fflush(None)
stream.flush()
except (AttributeError, ValueError, IOError):
pass # unsupported
You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:
def merged_stderr_stdout(): # $ exec 2>&1
return stdout_redirected(to=sys.stdout, stdout=sys.stderr)
Example:
from __future__ import print_function
import sys
with merged_stderr_stdout():
print('this is printed on stdout')
print('this is also printed on stdout', file=sys.stderr)
Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.
To answer, your edit: you could use python-daemon to daemonize your script and use logging module (as #erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.
you can try this too much better
import sys
class Logger(object):
def __init__(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt
The other answers didn't cover the case where you want forked processes to share your new stdout.
To do that:
from os import open, close, dup, O_WRONLY
old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1
..... do stuff and then restore
close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs
Quoted from PEP 343 -- The "with" Statement (added import statement):
Redirect stdout temporarily:
import sys
from contextlib import contextmanager
#contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Used as follows:
with open(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.
import sys
sys.stdout = open('stdout.txt', 'w')
Here is a variation of Yuda Prawira answer:
implement flush() and all the file attributes
write it as a contextmanager
capture stderr also
.
import contextlib, sys
#contextlib.contextmanager
def log_print(file):
# capture all outputs to a log file while still printing it
class Logger:
def __init__(self, file):
self.terminal = sys.stdout
self.log = file
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def __getattr__(self, attr):
return getattr(self.terminal, attr)
logger = Logger(file)
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdout = logger
sys.stderr = logger
try:
yield logger.log
finally:
sys.stdout = _stdout
sys.stderr = _stderr
with log_print(open('mylogfile.log', 'w')):
print('hello world')
print('hello world on stderr', file=sys.stderr)
# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
# ....
# print('[captured output]', log.getvalue())
You need a terminal multiplexer like either tmux or GNU screen
I'm surprised that a small comment by Ryan Amos' to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they've received. Further to Ryan's comment, tmux is a nice alternative to GNU screen.
But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you'd never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won't have lost the last 18 hours-worth of processing!
Based on this answer: https://stackoverflow.com/a/5916874/1060344, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.
Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python's logging facility (but you can really do anything with this):
class FileToLogInterface(file):
'''
Interface to make sure that everytime anything is written to stderr, it is
also forwarded to a file.
'''
def __init__(self, *args, **kwargs):
if 'cfg' not in kwargs:
raise TypeError('argument cfg is required.')
else:
if not isinstance(kwargs['cfg'], config.Config):
raise TypeError(
'argument cfg should be a valid '
'PostSegmentation configuration object i.e. '
'postsegmentation.config.Config')
self._cfg = kwargs['cfg']
kwargs.pop('cfg')
self._logger = logging.getlogger('access_log')
super(FileToLogInterface, self).__init__(*args, **kwargs)
def write(self, msg):
super(FileToLogInterface, self).write(msg)
self._logger.info(msg)
Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.
A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null
See this post for more info: What is the reason for performing a double fork when creating a daemon?
I know this question is answered (using python abc.py > output.log 2>&1 ), but I still have to say:
When writing your program, don't write to stdout. Always use logging to output whatever you want. That would give you a lot of freedom in the future when you want to redirect, filter, rotate the output files.
As mentioned by #jfs, most solutions will not properly handle some types of stdout output such as that from C extensions. There is a module that takes care of all this on PyPI called wurlitzer. You just need its sys_pipes context manager. It's as easy as using:
from contextlib import redirect_stdout
import os
from wurlitzer import sys_pipes
log = open("test.log", "a")
with redirect_stdout(log), sys_pipes():
print("print statement")
os.system("echo echo call")
Based on previous answers on this post I wrote this class for myself as a more compact and flexible way of redirecting the output of pieces of code - here just to a list - and ensure that the output is normalized afterwards.
class out_to_lt():
def __init__(self, lt):
if type(lt) == list:
self.lt = lt
else:
raise Exception("Need to pass a list")
def __enter__(self):
import sys
self._sys = sys
self._stdout = sys.stdout
sys.stdout = self
return self
def write(self,txt):
self.lt.append(txt)
def __exit__(self, type, value, traceback):
self._sys.stdout = self._stdout
Used as:
lt = []
with out_to_lt(lt) as o:
print("Test 123\n\n")
print(help(str))
Updating. Just found a scenario where I had to add two extra methods, but was easy to adapt:
class out_to_lt():
...
def isatty(self):
return True #True: You're running in a real terminal, False:You're being piped, redirected, cron
def flush(self):
pass
There are other versions using context but nothing this simple. I actually just googled to double check it would work and was surprised not to see it, so for other people looking for a quick solution that is safe and directed at only the code within the context block, here it is:
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
Tested like so:
$ cat redirect_stdout.py
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
$ python redirect_stdout.py
$ cat test_file
Testing 1 2 3