Related
How do I redirect stdout to an arbitrary file in Python?
When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.
I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.
If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:
# for python3
import sys
with open('file', 'w') as sys.stdout:
print('test')
A far more common method is to use shell redirection when executing (same on Windows and Linux):
$ python3 foo.py > file
There is contextlib.redirect_stdout() function in Python 3.4+:
from contextlib import redirect_stdout
with open('help.txt', 'w') as f:
with redirect_stdout(f):
print('it now prints to `help.text`')
It is similar to:
import sys
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_target):
old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
try:
yield new_target # run some code with the replaced stdout
finally:
sys.stdout = old_target # restore to the previous value
that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.
It doesn't redirect the stdout at the file descriptors level e.g.:
import os
from contextlib import redirect_stdout
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
print('redirected to a file')
os.write(stdout_fd, b'not redirected')
os.system('echo this also is not redirected')
b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.
To redirect at the file descriptor level, os.dup2() could be used:
import os
import sys
from contextlib import contextmanager
def fileno(file_or_fd):
fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
if not isinstance(fd, int):
raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
return fd
#contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
if stdout is None:
stdout = sys.stdout
stdout_fd = fileno(stdout)
# copy stdout_fd before it is overwritten
#NOTE: `copied` is inheritable on Windows when duplicating a standard stream
with os.fdopen(os.dup(stdout_fd), 'wb') as copied:
stdout.flush() # flush library buffers that dup2 knows nothing about
try:
os.dup2(fileno(to), stdout_fd) # $ exec >&to
except ValueError: # filename
with open(to, 'wb') as to_file:
os.dup2(to_file.fileno(), stdout_fd) # $ exec > to
try:
yield stdout # allow code to be run with the redirected stdout
finally:
# restore stdout to its previous value
#NOTE: dup2 makes stdout_fd inheritable unconditionally
stdout.flush()
os.dup2(copied.fileno(), stdout_fd) # $ exec >&copied
The same example works now if stdout_redirected() is used instead of redirect_stdout():
import os
import sys
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
print('redirected to a file')
os.write(stdout_fd, b'it is redirected now\n')
os.system('echo this is also redirected')
print('this is goes back to stdout')
The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.
Note: stdout.flush() does not flush
C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:
try:
import ctypes
from ctypes.util import find_library
except ImportError:
libc = None
else:
try:
libc = ctypes.cdll.msvcrt # Windows
except OSError:
libc = ctypes.cdll.LoadLibrary(find_library('c'))
def flush(stream):
try:
libc.fflush(None)
stream.flush()
except (AttributeError, ValueError, IOError):
pass # unsupported
You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:
def merged_stderr_stdout(): # $ exec 2>&1
return stdout_redirected(to=sys.stdout, stdout=sys.stderr)
Example:
from __future__ import print_function
import sys
with merged_stderr_stdout():
print('this is printed on stdout')
print('this is also printed on stdout', file=sys.stderr)
Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.
To answer, your edit: you could use python-daemon to daemonize your script and use logging module (as #erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.
you can try this too much better
import sys
class Logger(object):
def __init__(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt
The other answers didn't cover the case where you want forked processes to share your new stdout.
To do that:
from os import open, close, dup, O_WRONLY
old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1
..... do stuff and then restore
close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs
Quoted from PEP 343 -- The "with" Statement (added import statement):
Redirect stdout temporarily:
import sys
from contextlib import contextmanager
#contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Used as follows:
with open(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.
import sys
sys.stdout = open('stdout.txt', 'w')
Here is a variation of Yuda Prawira answer:
implement flush() and all the file attributes
write it as a contextmanager
capture stderr also
.
import contextlib, sys
#contextlib.contextmanager
def log_print(file):
# capture all outputs to a log file while still printing it
class Logger:
def __init__(self, file):
self.terminal = sys.stdout
self.log = file
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def __getattr__(self, attr):
return getattr(self.terminal, attr)
logger = Logger(file)
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdout = logger
sys.stderr = logger
try:
yield logger.log
finally:
sys.stdout = _stdout
sys.stderr = _stderr
with log_print(open('mylogfile.log', 'w')):
print('hello world')
print('hello world on stderr', file=sys.stderr)
# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
# ....
# print('[captured output]', log.getvalue())
You need a terminal multiplexer like either tmux or GNU screen
I'm surprised that a small comment by Ryan Amos' to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they've received. Further to Ryan's comment, tmux is a nice alternative to GNU screen.
But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you'd never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won't have lost the last 18 hours-worth of processing!
Based on this answer: https://stackoverflow.com/a/5916874/1060344, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.
Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python's logging facility (but you can really do anything with this):
class FileToLogInterface(file):
'''
Interface to make sure that everytime anything is written to stderr, it is
also forwarded to a file.
'''
def __init__(self, *args, **kwargs):
if 'cfg' not in kwargs:
raise TypeError('argument cfg is required.')
else:
if not isinstance(kwargs['cfg'], config.Config):
raise TypeError(
'argument cfg should be a valid '
'PostSegmentation configuration object i.e. '
'postsegmentation.config.Config')
self._cfg = kwargs['cfg']
kwargs.pop('cfg')
self._logger = logging.getlogger('access_log')
super(FileToLogInterface, self).__init__(*args, **kwargs)
def write(self, msg):
super(FileToLogInterface, self).write(msg)
self._logger.info(msg)
Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.
A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null
See this post for more info: What is the reason for performing a double fork when creating a daemon?
I know this question is answered (using python abc.py > output.log 2>&1 ), but I still have to say:
When writing your program, don't write to stdout. Always use logging to output whatever you want. That would give you a lot of freedom in the future when you want to redirect, filter, rotate the output files.
As mentioned by #jfs, most solutions will not properly handle some types of stdout output such as that from C extensions. There is a module that takes care of all this on PyPI called wurlitzer. You just need its sys_pipes context manager. It's as easy as using:
from contextlib import redirect_stdout
import os
from wurlitzer import sys_pipes
log = open("test.log", "a")
with redirect_stdout(log), sys_pipes():
print("print statement")
os.system("echo echo call")
Based on previous answers on this post I wrote this class for myself as a more compact and flexible way of redirecting the output of pieces of code - here just to a list - and ensure that the output is normalized afterwards.
class out_to_lt():
def __init__(self, lt):
if type(lt) == list:
self.lt = lt
else:
raise Exception("Need to pass a list")
def __enter__(self):
import sys
self._sys = sys
self._stdout = sys.stdout
sys.stdout = self
return self
def write(self,txt):
self.lt.append(txt)
def __exit__(self, type, value, traceback):
self._sys.stdout = self._stdout
Used as:
lt = []
with out_to_lt(lt) as o:
print("Test 123\n\n")
print(help(str))
Updating. Just found a scenario where I had to add two extra methods, but was easy to adapt:
class out_to_lt():
...
def isatty(self):
return True #True: You're running in a real terminal, False:You're being piped, redirected, cron
def flush(self):
pass
There are other versions using context but nothing this simple. I actually just googled to double check it would work and was surprised not to see it, so for other people looking for a quick solution that is safe and directed at only the code within the context block, here it is:
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
Tested like so:
$ cat redirect_stdout.py
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
$ python redirect_stdout.py
$ cat test_file
Testing 1 2 3
I am aware that sys.stdout is a Python object that wraps the output file handle but I am wondering if those file handles are "synced" and always the same?
For example, say sys.stdout.isatty() is True. I call GetStdHandle(-11) (-11 is STDOUT on Windows) and then some Windows Console API that fails and find that the error's errno is 6 (The handle is invalid). AFAIK, this means that the handle is not a valid console handle. In that case, they are not "synced". In other words, is it possible to redirect sys.stdout while the STDOUT handle returned by GetStdHandle is not redirected? My code uses GetStdHandle so ultimately I should test for errno 6 but it would be nice if I could just rely on sys.stdout.isatty.
Here is an example (I don't have access to a windows machine at the moment but hopefully the code is correct). Run with and without redirection (or normally and within a call to subprocess.check_output.
import sys
from ctypes import WinError, wintypes
STDOUT = -11
ERROR_INVALID_HANDLE = 6
kernel32 = ctypes.WinDLL('kernel32', use_errno=True, use_last_error=True)
handle = kernel32.GetStdHandle(STDOUT)
# Assume we set argtypes/restype for all win api functions here
if handle == wintypes.HANDLE(-1).value:
raise WinError()
console_mode = wintypes.DWORD(0)
# We use GetConsoleMode here but it could be any function that expects a
# valid console handle
retval = kernel32.GetConsoleMode(handle, ctypes.byref(console_mode))
# Are the following assertions always true?
if retval == 0:
errno = ctypes.get_last_error()
if errno == ERROR_INVALID_HANDLE:
print('Invalid handle')
assert not sys.stdout.isatty()
else:
# Another error happened
raise WinError()
else:
assert sys.stdout.isatty()
I tried to scour the CPython source code but could not find anything that could confirm or deny this. Perhaps someone more experienced with the codebase could point me in the right direction?
EDIT: I know about the CONOUT$ + CreateFile API. I am not interested in getting the input or output handle under redirection but in understanding the relationship between the Windows console handle APIs and sys.stdout.
Yes, I can reproduce this problem in C++.
You can use CreateFile to get the output handle of the console，and then use the handle as a parameter when calling the windows console apis .
The CreateFile function enables a process to get a handle to its
console's input buffer and active screen buffer, even if STDIN and
STDOUT have been redirected. To open a handle to a console's input
buffer, specify the CONIN$ value in a call to CreateFile. Specify the
CONOUT$ value in a call to CreateFile to open a handle to a console's
active screen buffer. CreateFile enables you to specify the read/write
access of the handle that it returns.
Refer: Console Handles
In C++ it looks like this,
HANDLE hConsole = CreateFile("CONOUT$",
GENERIC_WRITE | GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
It works well, and you can convert it to python code as needed.
Updated:
import sys
import ctypes
from ctypes import WinError, wintypes
STDOUT = -11
sys.stdout = open('test.txt', 'w')
kernel32 = ctypes.WinDLL('kernel32', use_errno=True, use_last_error=True)
handle = kernel32.GetStdHandle(STDOUT)
if handle == wintypes.HANDLE(-1).value:
raise WinError()
console_mode = wintypes.DWORD(0)
retval = kernel32.GetConsoleMode(handle, ctypes.byref(console_mode))
print(retval)
if sys.stdout.isatty():
print('You are running in a real terminal')
else:
print('You are being piped or redirected')
retval returns 1. They will all be printed in test.txt.
When you delete sys.stdout = open('test.txt', 'w').
Let us consider the following Python code, to be executed by cpython on a Linux system (warning: it will try to create or overwrite files in /tmp/first, /tmp/second and /tmp/third).
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
import subprocess
import os
import sys
import threading
class ThreadizedPopen(threading.Thread):
def __init__(self, command, stdin_name, stdout_name):
super(ThreadizedPopen, self).__init__()
self.command = command
self.stdin_name = stdin_name
self.stdout_name = stdout_name
self.returncode = None
def run(self):
with open(self.stdin_name, 'rb') as fin:
with open(self.stdout_name, 'wb') as fout:
popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None)
popen.communicate()
self.returncode = popen.returncode
def main():
os.system('mkfifo /tmp/first')
os.system('mkfifo /tmp/second')
os.system('mkfifo /tmp/third')
popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
popen1.start()
popen2.start()
with open('/tmp/third') as fin:
print fin.read()
popen1.join()
popen2.join()
if __name__ == '__main__':
main()
I execute it then, on another shell, I write something in /tmp/first (say with echo test > /tmp/first). I would expect the Python program to quickly exit and print the same thing I fed to the first FIFO.
In theory it should happen that the string I wrote in /tmp/first gets copied over by the two cat processes spawned by my program to the other two FIFOs and then picked up by the main Python program to be wrote on its stdout. As soon as every cat process finished, it should close its end of the writing FIFO, making the corresponding reading end return EOF and triggering the termination of the following cat process. Looking at the program with strace reveals that the test string is copied correctly through all the three FIFOs and is read by the main Python program. The first FIFO is also correctly closed (and the first cat process exits, together with its manager Python thread). However the second cat process is stuck in a read() call, expecting data from its reading FIFO.
I do not understand why this happens. From the pipe(t) man page (which, I understand, covers also this kind of FIFOs) it seems that a read on a FIFO is returned EOF as soon as the writing end (and all its duplicates) are closed. According to strace this appears to be the trace (in particular, the cat process is dead, thus all its file descriptors are closed; its managing thread has closed its descriptors as well, I can see it in the strace output).
Can you suggest me why that happens? I can post the strace output if it can be useful.
I found this question
and simply added close_fds=True to your subprocess call. Your code now reads:
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
import subprocess
import os
import sys
import threading
class ThreadizedPopen(threading.Thread):
def __init__(self, command, stdin_name, stdout_name):
super(ThreadizedPopen, self).__init__()
self.command = command
self.stdin_name = stdin_name
self.stdout_name = stdout_name
self.returncode = None
def run(self):
with open(self.stdin_name, 'rb') as fin:
with open(self.stdout_name, 'wb') as fout:
popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None, close_fds=True)
popen.communicate()
self.returncode = popen.returncode
def main():
os.system('mkfifo /tmp/first')
os.system('mkfifo /tmp/second')
os.system('mkfifo /tmp/third')
popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
popen1.start()
popen2.start()
with open('/tmp/third') as fin:
print fin.read()
popen1.join()
popen2.join()
if __name__ == '__main__':
main()
I placed your code in a script called fifo_issue.py and ran it in a terminal. The script was idling as you'd expect (ignore mkfifo: cannot create fifo):
$ python fifo_issue.py
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
Then, in a second terminal, I typed:
$ echo "I was echoed to /tmp/first!" > /tmp/first
Back to the first terminal that's still running your idling threads:
$ python fifo_issue.py
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
I was echoed to /tmp/first!
After which python exited correctly
I'm trying to redirect the output of printf functions to a file on Windows. I'm using ctypes with python3 to invoke the functions. My code is:
import os, sys
from ctypes import *
if __name__ == '__main__':
print("begin")
saved_stdout=os.dup(1)
test_file=open("TEST.TXT", "w")
os.dup2(test_file.fileno(), 1)
test_file.close()
print("python print")
cdll.msvcrt.printf(b"Printf function 1\n")
cdll.msvcrt.printf(b"Printf function 2\n")
cdll.msvcrt.printf(b"Printf function 3\n")
os.dup2(saved_stdout, 1)
print("end")
But when I run the code from Eclipse I get the following on the screen:
begin
end
Printf function 1
Printf function 2
Printf function 3
...and the following in the TEST.txt
python print
When I run this from cmd, this is what is on the screen:
begin
end
..and this is in the TEST.txt:
python print
When I comment out the second dup2() statement e.g.
import os, sys
from ctypes import *
if __name__ == '__main__':
print("begin")
saved_stdout=os.dup(1)
test_file=open("TEST.TXT", "w")
os.dup2(test_file.fileno(), 1)
test_file.close()
print("python print")
cdll.msvcrt.printf(b"Printf function 1\n")
cdll.msvcrt.printf(b"Printf function 2\n")
cdll.msvcrt.printf(b"Printf function 3\n")
#os.dup2(saved_stdout, 1)
print("end")
From Eclipse, on the screen:
begin
...and in the TEST.txt file:
python print
end
Printf function 1
Printf function 2
Printf function 3
From cmd, on the screen:
begin
...and in the TEST.txt file:
python print
end
I'm totally confused now. I read all the redirection threads here on StackOverflow and I can't understand what's going on.
Anyway, what I've gathered is that C functions access the stdout that is bind directly to the file descriptor, while python uses a special object for that - stdout File Object. So the elementary sys.stdout=*something* doesn't work with ctypes.
I've even tried os.fdopen(1) on the dup2-ed output and then calling flush() after every printf statement but this isn't working again.
I'm totally out of ideas now and would appreciate if someone have a solution for this.
Use the same C runtime that CPython 3.x uses (e.g. msvcr100.dll for 3.3). Also include a call to fflush(NULL) before and after redirecting stdout. For good measure, redirect the Windows StandardOutput handle, in case a program uses the Windows API directly.
This can get complicated if the DLL uses a different C runtime, which has its own set of POSIX file descriptors. That said, it should be OK if it gets loaded after you've redirected Windows StandardOutput.
Edit:
I've modified the example to run in Python 3.5+. VC++ 14's new "Universal CRT" makes it much more difficult to use C standard I/O via ctypes.
import os
import sys
import ctypes, ctypes.util
kernel32 = ctypes.WinDLL('kernel32')
STD_OUTPUT_HANDLE = -11
if sys.version_info < (3, 5):
libc = ctypes.CDLL(ctypes.util.find_library('c'))
else:
if hasattr(sys, 'gettotalrefcount'): # debug build
libc = ctypes.CDLL('ucrtbased')
else:
libc = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')
# VC 14.0 doesn't implement printf dynamically, just
# __stdio_common_vfprintf. This take a va_array arglist,
# which I won't implement, so I escape format specificiers.
class _FILE(ctypes.Structure):
"""opaque C FILE type"""
libc.__acrt_iob_func.restype = ctypes.POINTER(_FILE)
def _vprintf(format, arglist_ignored):
options = ctypes.c_longlong(0) # no legacy behavior
stdout = libc.__acrt_iob_func(1)
format = format.replace(b'%%', b'\0')
format = format.replace(b'%', b'%%')
format = format.replace(b'\0', b'%%')
arglist = locale = None
return libc.__stdio_common_vfprintf(
options, stdout, format, locale, arglist)
def _printf(format, *args):
return _vprintf(format, args)
libc.vprintf = _vprintf
libc.printf = _printf
def do_print(label):
print("%s: python print" % label)
s = ("%s: libc _write\n" % label).encode('ascii')
libc._write(1, s, len(s))
s = ("%s: libc printf\n" % label).encode('ascii')
libc.printf(s)
libc.fflush(None) # flush all C streams
if __name__ == '__main__':
# save POSIX stdout and Windows StandardOutput
fd_stdout = os.dup(1)
hStandardOutput = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
do_print("begin")
# redirect POSIX and Windows
with open("TEST.TXT", "w") as test:
os.dup2(test.fileno(), 1)
kernel32.SetStdHandle(STD_OUTPUT_HANDLE, libc._get_osfhandle(1))
do_print("redirected")
# restore POSIX and Windows
os.dup2(fd_stdout, 1)
kernel32.SetStdHandle(STD_OUTPUT_HANDLE, hStandardOutput)
do_print("end")
How do I redirect stdout to an arbitrary file in Python?
When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.
I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.
If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:
# for python3
import sys
with open('file', 'w') as sys.stdout:
print('test')
A far more common method is to use shell redirection when executing (same on Windows and Linux):
$ python3 foo.py > file
There is contextlib.redirect_stdout() function in Python 3.4+:
from contextlib import redirect_stdout
with open('help.txt', 'w') as f:
with redirect_stdout(f):
print('it now prints to `help.text`')
It is similar to:
import sys
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_target):
old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
try:
yield new_target # run some code with the replaced stdout
finally:
sys.stdout = old_target # restore to the previous value
that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.
It doesn't redirect the stdout at the file descriptors level e.g.:
import os
from contextlib import redirect_stdout
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
print('redirected to a file')
os.write(stdout_fd, b'not redirected')
os.system('echo this also is not redirected')
b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.
To redirect at the file descriptor level, os.dup2() could be used:
import os
import sys
from contextlib import contextmanager
def fileno(file_or_fd):
fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
if not isinstance(fd, int):
raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
return fd
#contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
if stdout is None:
stdout = sys.stdout
stdout_fd = fileno(stdout)
# copy stdout_fd before it is overwritten
#NOTE: `copied` is inheritable on Windows when duplicating a standard stream
with os.fdopen(os.dup(stdout_fd), 'wb') as copied:
stdout.flush() # flush library buffers that dup2 knows nothing about
try:
os.dup2(fileno(to), stdout_fd) # $ exec >&to
except ValueError: # filename
with open(to, 'wb') as to_file:
os.dup2(to_file.fileno(), stdout_fd) # $ exec > to
try:
yield stdout # allow code to be run with the redirected stdout
finally:
# restore stdout to its previous value
#NOTE: dup2 makes stdout_fd inheritable unconditionally
stdout.flush()
os.dup2(copied.fileno(), stdout_fd) # $ exec >&copied
The same example works now if stdout_redirected() is used instead of redirect_stdout():
import os
import sys
stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
print('redirected to a file')
os.write(stdout_fd, b'it is redirected now\n')
os.system('echo this is also redirected')
print('this is goes back to stdout')
The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.
Note: stdout.flush() does not flush
C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:
try:
import ctypes
from ctypes.util import find_library
except ImportError:
libc = None
else:
try:
libc = ctypes.cdll.msvcrt # Windows
except OSError:
libc = ctypes.cdll.LoadLibrary(find_library('c'))
def flush(stream):
try:
libc.fflush(None)
stream.flush()
except (AttributeError, ValueError, IOError):
pass # unsupported
You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:
def merged_stderr_stdout(): # $ exec 2>&1
return stdout_redirected(to=sys.stdout, stdout=sys.stderr)
Example:
from __future__ import print_function
import sys
with merged_stderr_stdout():
print('this is printed on stdout')
print('this is also printed on stdout', file=sys.stderr)
Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.
To answer, your edit: you could use python-daemon to daemonize your script and use logging module (as #erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.
you can try this too much better
import sys
class Logger(object):
def __init__(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt
The other answers didn't cover the case where you want forked processes to share your new stdout.
To do that:
from os import open, close, dup, O_WRONLY
old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1
..... do stuff and then restore
close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs
Quoted from PEP 343 -- The "with" Statement (added import statement):
Redirect stdout temporarily:
import sys
from contextlib import contextmanager
#contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Used as follows:
with open(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.
import sys
sys.stdout = open('stdout.txt', 'w')
Here is a variation of Yuda Prawira answer:
implement flush() and all the file attributes
write it as a contextmanager
capture stderr also
.
import contextlib, sys
#contextlib.contextmanager
def log_print(file):
# capture all outputs to a log file while still printing it
class Logger:
def __init__(self, file):
self.terminal = sys.stdout
self.log = file
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def __getattr__(self, attr):
return getattr(self.terminal, attr)
logger = Logger(file)
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdout = logger
sys.stderr = logger
try:
yield logger.log
finally:
sys.stdout = _stdout
sys.stderr = _stderr
with log_print(open('mylogfile.log', 'w')):
print('hello world')
print('hello world on stderr', file=sys.stderr)
# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
# ....
# print('[captured output]', log.getvalue())
You need a terminal multiplexer like either tmux or GNU screen
I'm surprised that a small comment by Ryan Amos' to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they've received. Further to Ryan's comment, tmux is a nice alternative to GNU screen.
But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you'd never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won't have lost the last 18 hours-worth of processing!
Based on this answer: https://stackoverflow.com/a/5916874/1060344, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.
Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python's logging facility (but you can really do anything with this):
class FileToLogInterface(file):
'''
Interface to make sure that everytime anything is written to stderr, it is
also forwarded to a file.
'''
def __init__(self, *args, **kwargs):
if 'cfg' not in kwargs:
raise TypeError('argument cfg is required.')
else:
if not isinstance(kwargs['cfg'], config.Config):
raise TypeError(
'argument cfg should be a valid '
'PostSegmentation configuration object i.e. '
'postsegmentation.config.Config')
self._cfg = kwargs['cfg']
kwargs.pop('cfg')
self._logger = logging.getlogger('access_log')
super(FileToLogInterface, self).__init__(*args, **kwargs)
def write(self, msg):
super(FileToLogInterface, self).write(msg)
self._logger.info(msg)
Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.
A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null
See this post for more info: What is the reason for performing a double fork when creating a daemon?
I know this question is answered (using python abc.py > output.log 2>&1 ), but I still have to say:
When writing your program, don't write to stdout. Always use logging to output whatever you want. That would give you a lot of freedom in the future when you want to redirect, filter, rotate the output files.
As mentioned by #jfs, most solutions will not properly handle some types of stdout output such as that from C extensions. There is a module that takes care of all this on PyPI called wurlitzer. You just need its sys_pipes context manager. It's as easy as using:
from contextlib import redirect_stdout
import os
from wurlitzer import sys_pipes
log = open("test.log", "a")
with redirect_stdout(log), sys_pipes():
print("print statement")
os.system("echo echo call")
Based on previous answers on this post I wrote this class for myself as a more compact and flexible way of redirecting the output of pieces of code - here just to a list - and ensure that the output is normalized afterwards.
class out_to_lt():
def __init__(self, lt):
if type(lt) == list:
self.lt = lt
else:
raise Exception("Need to pass a list")
def __enter__(self):
import sys
self._sys = sys
self._stdout = sys.stdout
sys.stdout = self
return self
def write(self,txt):
self.lt.append(txt)
def __exit__(self, type, value, traceback):
self._sys.stdout = self._stdout
Used as:
lt = []
with out_to_lt(lt) as o:
print("Test 123\n\n")
print(help(str))
Updating. Just found a scenario where I had to add two extra methods, but was easy to adapt:
class out_to_lt():
...
def isatty(self):
return True #True: You're running in a real terminal, False:You're being piped, redirected, cron
def flush(self):
pass
There are other versions using context but nothing this simple. I actually just googled to double check it would work and was surprised not to see it, so for other people looking for a quick solution that is safe and directed at only the code within the context block, here it is:
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
Tested like so:
$ cat redirect_stdout.py
import sys
with open('test_file', 'w') as sys.stdout:
print('Testing 1 2 3')
$ python redirect_stdout.py
$ cat test_file
Testing 1 2 3