Long story short, I am writing python code that occasionally causes an underlying module to spit out complaints in the terminal that I want my code to respond to. My question is if there is some way that I can take in all terminal outputs as a string while the program is running so that I might parse it and execute some handler code. Its not errors that crash the program entirely and not a situation where I can simply do a try catch. Thanks for any help!
Edit: Running on Linux
there are several solutions to your need. the easiest would be to use a shared buffer of sort and get all your package output there instead of stdout (with regular print) thus keeping your personal streams under your package control.
since you probably already have some code with print or you want for it to work with minimal change, i suggest use the contextlib.redirect_stdout with a context manager.
give it a shared io.StringIO instance and wrap all your method with it.
you can even create a decorator to do it automatically.
something like:
// decorator
from contextlib import redirect_stdout
import io
import functools
SHARED_BUFFER = io.StringIO()
def std_redirecter(func):
#functools.wrap(func)
def inner(*args, **kwargs):
with redirect_stdout(SHARED_BUFFER) as buffer:
print('foo')
print('bar')
func(*args, **kwargs)
return inner
// your files
#std_redirecter
def writing_to_stdout_func():
print('baz')
// invocation
writing_to_stdout_func()
string = SHARED_BUFFER.getvalue() // 'foo\nbar\nbaz\n'
Related
I'm working on an open source python library that uses a verbose_print command to log outputs in the console. Currently it looks like this:
def sys_write_flush(s):
""" Writes and flushes without delay a text in the console """
sys.stdout.write(s)
sys.stdout.flush()
def verbose_print(verbose, s):
""" Only prints s (with sys_write_flush) if verbose is True."""
if verbose:
sys_write_flush(s)
I proposed a change that looks like this:
def verbose_print(verbose, *args):
""" Prints everything passed except the first argument if verbose is True."""
if verbose:
print(*args)
Apart from the fact that it fails on Python 2 (bonus point for fixing this!), I thought that this would be better and more idiomatic. The advantages are, that you can treat verbose_print exactly like print, except that the first argument has to be True or False.
The repo owner replied with this message:
I should have documented this one, but basically the issue was that in
some consoles (and in the IPython notebook, at least at the time),
"print" commands get delayed, while stdout.flush are instantaneous, so
my method was better at providing feedback.
I would be against changing it to print unless it solves some known
issues.
Is this still a valid concern? Would print() followed by sys.stdout.flush() avoid the delay? Are there any better ways to write this?
Source
Quote from the docs:
print evaluates each expression in turn and writes the resulting
object to standard output.
Standard output is defined as the file object named stdout in the
built-in module sys. If no such object exists, or if it does not
have a write() method, a RuntimeError exception is raised.
According to this, print writes to sys.stdout, so, yes, doing a sys.stdout.flush() after printing will have the same effect as flushing after sys.stdout.write-ing.
The syntax print(*a) fails in Python 2 because print isn't a function, but a statement, and that fun(*stuff) construct is only applicable to functions.
In Python 3 print(*a) passes whatever a contains to the function print as separate arguments, but this is equal to passing a big string:
separator = ' '
print separator.join(map(str, iterable))
So, your code could look like this:
def verbose_print(verbose, *args):
""" Prints everything passed except the first argument if verbose is True."""
if verbose:
print " ".join(map(str, args))
sys.stdout.flush()
Although I don't see why this can be faster or more readable than the original.
I have a Python script that accepts one input (a text file): ./myprog.py file.txt. The script outputs a string based on the given input.
I have a set of test files that I would like to test my program with. I know the expected output for each file and want to make sure my script produces the correct output for each file.
What is the generally accepted way to do this type of testing?
I was thinking of using Python's unittest module as the testing framework, and then running my script through subprocess.check_output(stderr=subprocess.STDOUT), capturing stdout and stderr, and then doing a unittest assertEqual to compare the actual and expected strings. I want to make sure I'm not missing some nicer solution.
There's two problems here. Testing a program, as opposed to a library of functions, and testing something that prints, as opposed to values returned from a function. Both make testing more difficult, so it's best to side step these problems as much as possible.
The usual technique is to create a library of functions and then have your program be a thin wrapper around that. These functions return their results, and only the program does the printing. This means you can use normal unit testing techniques for most of the code.
You can have a single file which is both a library and a program. Here's a simple example as hello.py.
def hello(greeting, place):
return greeting + ", " + place + "!"
def main():
print(hello("Hello", "World"))
if __name__ == '__main__':
main()
That last bit is how a file can tell if it was run as a program or if it was imported as a library. It allows access to the individual functions with import hello, and it also allows the file to run as a program. See this answer for more information.
Then you can write a mostly normal unit test.
import hello
import unittest
import sys
from StringIO import StringIO
import subprocess
class TestHello(unittest.TestCase):
def test_hello(self):
self.assertEqual(
hello.hello("foo", "bar"),
"foo, bar!"
)
def test_main(self):
saved_stdout = sys.stdout
try:
out = StringIO()
sys.stdout = out
hello.main()
output = out.getvalue()
self.assertEqual(output, "Hello, World!\n")
finally:
sys.stdout = saved_stdout
def test_as_program(self):
self.assertEqual(
subprocess.check_output(["python", "hello.py"]),
"Hello, World!\n"
)
if __name__ == '__main__':
unittest.main()
Here test_hello is unit testing hello directly as a function; and in a more complicated program there would be more functions to test. We also have test_main to unit test main using StringIO to capture its output. Finally, we ensure the program will run as a program with test_as_program.
The important thing is to test as much of the functionality as functions returning data, and to test as little as possible as printed and formatted strings, and almost nothing via running the program itself. By the time we're actually testing the program, all we need to do is check that it calls main.
I am trying to gather the files: output.xml, report.html and log.html when the whole execution ends.
I can use a listener for this purpose but I want to avoid write a line like:
robot --listener "SomeListener.py" YourTestSuite.robot
Then, I looked at the documentation and I found that I can use a Test Library as a Listener and import it in my Test Suite, for example:
class SomeLibrary(object):
ROBOT_LISTENER_API_VERSION = 2 # or 3
ROBOT_LIBRARY_SCOPE = "GLOBAL" # or TEST SUITE
def __init__(self):
self.ROBOT_LIBRARY_LISTENER = self
def start_suite(self, data, result):
pass
def close(self):
pass
My problem is that the close method is called when the library goes out of the scope. So, I cannot gather the report files because they don't exist in that moment and I receive an Error. I also tried with the method:
def report_file(self, path):
pass
But, nothing happens, I guess that it could be because a Library as a Listener cannot use this methods from the Listener API or because the file is still not created.
Any idea about How can I gather those files using a library as a Listener?
I am open to ideas or suggestions.
Thanks.
The Listener specification actually specifies three methods that are triggered when the three files are generated:
startSuite
endSuite
startTest
endTest
startKeyword
endKeyword
logMessage
message
outputFile
logFile
reportFile
debugFile
Have a look at the last 4 ones, which should be helpful. For more details look at the listener API specification. 4.3.3 Listener interface methods
I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm
I have a Python-based app that can accept a few commands in a simple read-eval-print-loop. I'm using raw_input('> ') to get the input. On Unix-based systems, I also import readline to make things behave a little better. All this is working fine.
The problem is that there are asynchronous events coming in, and I'd like to print output as soon as they happen. Unfortunately, this makes things look ugly. The "> " string doesn't show up again after the output, and if the user is halfway through typing something, it chops their text in half. It should probably redraw the user's text-in-progress after printing something.
This seems like it must be a solved problem. What's the proper way to do this?
Also note that some of my users are Windows-based.
TIA
Edit: The accepted answer works under Unixy platforms (when the readline module is available), but if anyone knows how to make this work under Windows, it would be much appreciated!
Maybe something like this will do the trick:
#!/usr/bin/env python2.6
from __future__ import print_function
import readline
import threading
PROMPT = '> '
def interrupt():
print() # Don't want to end up on the same line the user is typing on.
print('Interrupting cow -- moo!')
print(PROMPT, readline.get_line_buffer(), sep='', end='')
def cli():
while True:
cli = str(raw_input(PROMPT))
if __name__ == '__main__':
threading.Thread(target=cli).start()
threading.Timer(2, interrupt).start()
I don't think that stdin is thread-safe, so you can end up losing characters to the interrupting thread (that the user will have to retype at the end of the interrupt). I exaggerated the amount of interrupt time with the time.sleep call. The readline.get_line_buffer call won't display the characters that get lost, so it all turns out alright.
Note that stdout itself isn't thread safe, so if you've got multiple interrupting threads of execution, this can still end up looking gross.
Why are you writing your own REPL using raw_input()? Have you looked at the cmd.Cmd class? Edit: I just found the sclapp library, which may also be useful.
Note: the cmd.Cmd class (and sclapp) may or may not directly support your original goal; you may have to subclass it and modify it as needed to provide that feature.
run this:
python -m twisted.conch.stdio
You'll get a nice, colored, async REPL, without using threads. While you type in the prompt, the event loop is running.
look into the code module, it lets you create objects for interpreting python code also (shameless plug) https://github.com/iridium172/PyTerm lets you create interactive command line programs that handle raw keyboard input (like ^C will raise a KeyboardInterrupt).
It's kind of a non-answer, but I would look at IPython's code to see how they're doing it.
I think you have 2 basic options:
Synchronize your output (i.e. block until it comes back)
Separate your input and your (asyncronous) output, perhaps in two separate columns.