Simple unittest below.
If I run it (e.g., python -m unittest module_name) without 'test' as an argument, it passes. If I run it with 'test' as an argument, I get "TypeError: bad argument type for built-in operation". Why?
from io import StringIO
import sys
from unittest import TestCase
class TestSimple(TestCase):
def test_simple(self):
old_stdout = sys.stdout
buf = StringIO()
try:
sys.stdout = buf
print('hi')
finally:
import pdb
if 'test' in sys.argv:
pdb.set_trace()
sys.stdout = old_stdout
contextlib.redirect_stdout version:
from contextlib import redirect_stdout
from io import StringIO
import pdb
import sys
from unittest import TestCase
class TestSimple(TestCase):
def test_simple(self):
buf = StringIO()
with redirect_stdout(buf):
print('hi')
pdb.set_trace()
print('finis')
Thanks in advance.
Edit:
The original program was tested in Python 3.4 in both Debian and Windows 7.
Something similar (using environment flags instead of a command line argument) appears to hang in Python 2, but pressing c allows it to finish, so I'm guessing it might just be that pdb's UI has been redirected.But the Python 3 version has the behavior initial described (crashes), although a colleague tested on 3.4 on Mac OS and saw the "hang" behavior.
You need to give pdb the original stdout:
pdb.Pdb(stdout=sys.__stdout__).set_trace()
Related
I am trying to write a unit test that executes a function that writes to stdout, capture that output, and check the result. The function in question is a black box: we can't change how it is writing it's output. For purposes of this example I've simplified it quite a bit, but essentially the function generates its output using subprocess.call().
No matter what I try I can't capture the output. It is always written to the screen, and the test fails because it captures nothing. I experimented with both print() and os.system(). With print() I can capture stdout, but not with os.system() either.
It's also not specific to unittesting. I've written my test example without that with the same results.
Questions similar to this have been asked a lot, and the answers all seem to boil down to use subprocess.Popen() and communicate(), but that would require changing the black box. I'm sure there's an answer I just haven't come across, but I'm stumped.
We are using Python-2.7.
Anyway my example code is this:
#!/usr/bin/env python
from __future__ import print_function
import sys
sys.dont_write_bytecode = True
import os
import unittest
import subprocess
from contextlib import contextmanager
from cStringIO import StringIO
# from somwhere import my_function
def my_function(arg):
#print('my_function:', arg)
subprocess.call(['/bin/echo', 'my_function: ', arg], shell=False)
#os.system('echo my_function: ' + arg)
#contextmanager
def redirect_cm(new_stdout):
old_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield
finally:
sys.stdout = old_stdout
class Test_something(unittest.TestCase):
def test(self):
fptr = StringIO()
with redirect_cm(fptr):
my_function("some_value")
self.assertEqual("my_function: some_value\n", fptr.getvalue())
if __name__ == '__main__':
unittest.main()
There are two issues in the above code
StringIO fptr does not shared by the current and the spawned process, we could not get the result in current process even if the spawned process has written result to StringIO object
Changing sys.stdout doesn’t affect the standard I/O streams of processes executed by os.popen(), os.system() or the exec*() family of functions in the os module
A simple solution is
use os.pipe to share result between the two processes
use os.dup2 instead of changing sys.stdout
A demo example as following shown
import sys
import os
import subprocess
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_out):
old_stdout = os.dup(1)
try:
os.dup2(new_out, sys.stdout.fileno())
yield
finally:
os.dup2(old_stdout, 1)
def test():
reader, writer = os.pipe()
with redirect_stdout(writer):
subprocess.call(['/bin/echo', 'something happened what'], shell=False)
print os.read(reader, 1024)
test()
I'm importing a module foo which uses Python's logging module. However, foo produces a huge amount of logging output, and I need to use stdout to communicate important information to the user, which is largely being drowned out by the ridiculous output of the module I'm importing.
How can I disable the module's ability to log to stdout without modifying foo's code? I still want it to log to the files it logs to, but I don't want it logging to stdout.
I have tried the following:
logging.getLogger("foo").propagate = False
and
#contextlib.contextmanager
def nostdout():
class DummyFile(object):
def write(self, x): pass
save_stdout = sys.stdout
sys.stdout = DummyFile()
yield
sys.stdout = save_stdout
with nostdout(): import foo
Try the following:
logging.getLogger(<logger_name_used_in_foo>).propagate = False
I'm referencing this article.
In general, if you want to capture anything written to stdout you can use the contextlib in Python 3:
from contextlib import redirect_stdout
f = io.StringIO()
with redirect_stdout(f):
print('foobar')
call_annoying_module()
print('Stdout: "{0}"'.format(f.getvalue()))
On Python 3.4 and older, redirect_stdout can be implemented like this:
from contextlib import contextmanager
#contextmanager
def stdout_redirector(stream):
old_stdout = sys.stdout
sys.stdout = stream
try:
yield
finally:
sys.stdout = old_stdout
If the library has any C bindings that print using puts then it gets more complicated. See the article.
The easiest case is when you're running another program using subprocess,
then all stdout output can be easily captured.
proc = subprocess.Popen("echo output".split(), stdout=subprocess.PIPE)
std_output, err_output = proc.communicate()
There was a redirect_output function in IPython.utils, and there was a %%capture magic function, but these are now gone, and this thread on the topic is now outdated.
I'd like to do something like the following:
from IPython.utils import io
from __future__ import print_function
with io.redirect_output(stdout=False, stderr="stderr_test.txt"):
while True:
print('hello!', file=sys.stderr)
Thoughts? For more context, I am trying to capture the output of some ML functions that run for hours or days, and output a line every 5-10 seconds to stderr. I then want to take the output, munge it, and plot the data.
You could probably try replacing sys.stderr with some other file descriptor the same way as suggested here.
import sys
oldstderr = sys.stderr
sys.stderr = open('log.txt', 'w')
# do something
sys.stderr = oldstderr
Update: starting form Python 3.4, you should consuder using contextlib.redirect_stdout() instead, like this:
f = io.StringIO()
with redirect_stdout(f):
print('a')
s = f.getvalue()
#Ben, just replacing sys.stderr did not work, and the full flush logic suggested in the post was necessary. But thank you for the pointer as it finally gave me a working version:
import sys
oldstderr = sys.stderr
sys.stderr = open('log.txt', 'w')
class flushfile():
def __init__(self, f):
self.f = f
def __getattr__(self,name):
return object.__getattribute__(self.f, name)
def write(self, x):
self.f.write(x)
self.f.flush()
def flush(self):
self.f.flush()
sys.sterr = flushfile(sys.stderr)
from __future__ import print_function
# some long running function here, e.g.
for i in range(1000000):
print('hello!', file=sys.stderr)
sys.stderr = oldstderr
It would have been nice if Jupyter kept the redirect_output() function and/or the %%capture magic.
I was wondering if this is possible in python:
# module1
def test():
print('hey')
# module2
import module1
module1.test() # prints to stdout
Without modifying module1 is there any way to wrap this in module2 so that I can capture the
print('hey') inside a variable? Apart from running module1 as a script?
I don't want to be responsible for modifying sys.stdout and then restoring it to its previous values. The above answers don't have any finally: clause, which can be dangerous integrating this into other important code.
https://docs.python.org/3/library/contextlib.html
import contextlib, io
f = io.StringIO()
with contextlib.redirect_stdout(f):
module1.test()
output = f.getvalue()
You probably want the variable output which is <class 'str'> with the redirected stdout.
Note: this code is lifted from the official docs with trivial modifications (but tested). Another version of this answer was already given to a mostly duplicated question here: https://stackoverflow.com/a/22434594/1092940
I leave the answer here because it is a much better solution than the others here IMO.
Yes, all you need is to redirect the stdout to a memory buffer that complies with the interface of stdout, you can do it with StringIO. This works for me in 2.7:
import sys
import cStringIO
stdout_ = sys.stdout #Keep track of the previous value.
stream = cStringIO.StringIO()
sys.stdout = stream
print "hello" # Here you can do whatever you want, import module1, call test
sys.stdout = stdout_ # restore the previous stdout.
variable = stream.getvalue() # This will get the "hello" string inside the variable
Yes, you can. You need to take control of sys.stdout. Something like this:
import sys
stdout_ = sys.stdout #Keep track of the previous value.
sys.stdout = open('myoutputfile.txt', 'w') # Something here that provides a write method.
# calls to print, ie import module1
sys.stdout = stdout_ # restore the previous stdout.
For Python 3:
# redirect sys.stdout to a buffer
import sys, io
stdout = sys.stdout
sys.stdout = io.StringIO()
# call module that calls print()
import module1
module1.test()
# get output and restore sys.stdout
output = sys.stdout.getvalue()
sys.stdout = stdout
print(output)
There's No need to use another module, just class object with write attribute, with one input, which you can save in another variable. for ecample
CLASS:
class ExClass:
def __init__(self):
self.st = ''
def write(self, o): #here o is the output that goes to stdout
self.st += str(o)
MAIN Program:
import sys
stdout_ = sys.stdout
var = ExClass()
sys.stdout = var
print("Hello") # these will not be pronted
print("Hello2") # instead will be written in var.st
sys.stdout = stdout_
print(var.st)
output will be
Hello
Hello2
Sending ftplib debug output to the logging module
Based on the approach taken by jimmy kumar ahalpara answer, I was able to capture ftplib's debug output into logging. ftplib was around before the logging module and uses print to emit debug messages.
I'd tried reassigning the print function to a logging method but I couldn't get that to work. The code below works for me.
I should think this will work with other modules as well but there would not be any granularity between different module's output as it's capturing everything sent to stdout to the same logger.
# convenience class to redirect stdout to logging
class SendToLog:
def __init__(self, logging_method):
self.logger = logging
_method
def write(self, o):
if str(o).strip(): # ignore empty lines
self.logger(str(o))
import logging
import sys
# code to initialise logging output and handlers ...
# ...
# get logger for ftplib and redirect it's print output to our log
ftp_logger = logging.getLogger('ftplib')
# note: logging's debug method is passed to the class, the instance then calls this method
sys.stdout = SendToLog(ftp_logger.debug)
# code to do stuff with ftplib ...
# remember to set ftplib's debug level > 0 or there will be no output
# FTP.set_debuglevel(1)
# ...
# important to finalise logging and restore stdout
logging.shutdown()
sys.stdout = sys.__stdout__
python3 stdout ftplib logging
I'm on Windows and Python is (very effectively) preventing me from sending a stand-alone '\n' character to STDOUT. For example, the following will output foo\r\nvar:
sys.stdout.write("foo\nvar")
How can I turn this "feature" off? Writing to a file first is not an option, because the output is being piped.
Try the following before writing anything:
import sys
if sys.platform == "win32":
import os, msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
If you only want to change to binary mode temporarily, you can write yourself a wrapper:
import sys
from contextlib import contextmanager
#contextmanager
def binary_mode(f):
if sys.platform != "win32":
yield; return
import msvcrt, os
def setmode(mode):
f.flush()
msvcrt.setmode(f.fileno(), mode)
setmode(os.O_BINARY)
try:
yield
finally:
setmode(os.O_TEXT)
with binary_mode(sys.stdout), binary_mode(sys.stderr):
# code
Add 'r' before the string:
sys.stdout.write(r"foo\nvar")
As expected, it also works for print.