Long story short, I am writing python code that occasionally causes an underlying module to spit out complaints in the terminal that I want my code to respond to. My question is if there is some way that I can take in all terminal outputs as a string while the program is running so that I might parse it and execute some handler code. Its not errors that crash the program entirely and not a situation where I can simply do a try catch. Thanks for any help!
Edit: Running on Linux
there are several solutions to your need. the easiest would be to use a shared buffer of sort and get all your package output there instead of stdout (with regular print) thus keeping your personal streams under your package control.
since you probably already have some code with print or you want for it to work with minimal change, i suggest use the contextlib.redirect_stdout with a context manager.
give it a shared io.StringIO instance and wrap all your method with it.
you can even create a decorator to do it automatically.
something like:
// decorator
from contextlib import redirect_stdout
import io
import functools
SHARED_BUFFER = io.StringIO()
def std_redirecter(func):
#functools.wrap(func)
def inner(*args, **kwargs):
with redirect_stdout(SHARED_BUFFER) as buffer:
print('foo')
print('bar')
func(*args, **kwargs)
return inner
// your files
#std_redirecter
def writing_to_stdout_func():
print('baz')
// invocation
writing_to_stdout_func()
string = SHARED_BUFFER.getvalue() // 'foo\nbar\nbaz\n'
I have the following function,
def load():
with open(PATH_CONFIG, 'r') as file:
return json.loads(file.read())
Will there be a file.close() called? I know that the with keyword normally calls the close() method for the file at the end of the indented block, but at the same time the return keyword means that the rest of the function does not run.
Just like try/finally, anything that exits the with block (return, break/continue that affects a loop surrounding it, exception thrown, sys.exit called, etc.) will perform appropriate cleanup as execution bubbles out of the with block.
The only exceptions are:
When there are actual bugs (in the interpreter, or in misuse of intrinsically dangerous tools like ctypes) where the interpreter itself crashes or otherwise exits "forcefully" (e.g. due to a segfault)
Calling os._exit bypasses all cleanup procedures (that's why it should never be used in anything but forked worker processes)
Return exits the with block like a normal dedent
Yes.
If an exception is raised, a context manager has the option of changing its behavior, but there's no difference between a return and falling off the end of the statement body, and with few exceptions most context managers will perform their cleanup and allow the exception to propagate.
The idea is that it's comparable to a finally statement, and will be executed no matter how the block is exited. The contextmanager class from the standard library makes this analogy concrete.
from contextlib import contextmanager
#contextmanager
def example():
print('enter')
try:
yield
print('unexceptional return')
finally:
print('exit')
We can use with example(): in various ways to see how the with statement performs in a more visible example than closing a file.
I end up having to write and support short python wrapper scripts with the following high-level structure:
try:
code
...
...
except:
raise
finally:
file_handle.close()
db_conn.close()
Notice that all I do in the except block is re-raise the exception to the script caller sans window-dressing; this is not a problem in my particular context. The idea here is that cleanup code should always be executed by means of the finally block, exception or not.
Am I better off using an atexit handler for this purpose? I could do without the extra level of indentation introduced by try.
The atexit module provides a simple interface to register functions to be called when a program closes down normally. Functions registered are automatically executed upon normal interpreter termination.
import atexit
def cleanup():
print 'performimg cleanup'
# multiple functions can be registered here...
atexit.register(cleanup)
The sys module also provides a hook, sys.exitfunc, but only one function can be registered there.
Finally is accompanied by try except block, functionality of finally can also be used for something similar like cleanup, however at finally block sys.exc_info is all-None.
If the finally clause raises another exception, the saved exception is discarded however you can put try except in the function registered with atexit to handle them.
Another pro-con is atexit functions are only executes when program terminates, however you can use finally (with try-except) anywhere in the code and perform the cleanup
In you scenario, where you want to raise an exception from cleanup content, usage of atexit would be helpful, if you are ok for cleanup to happen at the end of the program
Just use contextlib.closing
with closing(resource1) as f1, closing(resource2) as f2:
f1.something()
f2.something()
And they will be automatically closed. Files objects can be used directly as contexts so you don't need the closing call.
If close is not the only method used by your resources, you can create custom functions with the contextlib.contextmanager decorator.
atexit is called upon program termination, so this is not what you are looking for.
I'm writing an MPI-based application (but MPI doesn't matter in my question, I mention it only to expose the rationale) and in some cases, when there is less work items than processes, I need to create a new communicator excluding the processes that have nothing to do. Finally, the new communicator has to be freed by the processes that have work to do (and only by them).
A neat way to do that would be to write:
with filter_comm(comm, nworkitems) as newcomm:
... do work with communicator newcomm...
the body being executed only by the processes that have work to do.
Is there a way in a context manager to avoid executing the body?
I understand that context managers have rightfully been designed to avoid hiding control flows, but I wonder if it is possible to circumvent that, since in my case I think it would be justified for clarity sake.
The ability to conditionally skip context manager body has been proposed but rejected as documented in PEP 377.
I did some research about alternatives. Here are my findings.
First let me explain the background of my code examples. You have a bunch of devices you want to work with. For every device you have to acquire the driver for the device; then work with the device using the driver; and lastly release the driver so others can acquire the driver and work with the device.
Nothing out of the ordinary here. The code looks roughly like this:
driver = getdriver(devicename)
try:
dowork(driver)
finally:
releasedriver(driver)
But once every full moon when the planets are not aligned correctly the acquired driver for a device is bad and no work can be done with the device. This is no big deal. Just skip the device this round and try again next round. Usually the driver is good then. But even a bad driver needs to be released otherwise no new driver can be acquired.
(the firmware is proprietary and the vendor is reluctant to fix or even acknowledge this bug)
The code now looks like this:
driver = getdriver(devicename)
try:
if isgooddriver(driver):
dowork(driver)
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
That is a lot of boilerplate code that needs to be repeated everytime work needs to be done with a device. A prime candidate for python's context manager also known as with statement. It might look like this:
# note: this code example does not work
#contextlib.contextmanager
def contextgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
And then the code when working with a device is short and sweet:
# note: this code example does not work
with contextgetdriver(devicename) as driver:
dowork(driver)
But this does not work. Because a context manager has to yield. It may not not yield. Not yielding will result in a RuntimeException raised by contextmanager.
So we have to pull out the check from the context manager
#contextlib.contextmanager
def contextgetdriver(devicename):
driver = getdriver(devicename)
try:
yield driver
finally:
release(driver)
and put it in the body of the with statement
with contextgetdriver(devicename) as driver:
if isgooddriver(driver):
dowork(driver)
else:
handledrivererror(geterrordetails(driver))
This is ugly because now we again have some boilerplate that needs to be repeated everytime we want to work with a device.
So we want a context manager that can conditionaly execute the body. But we have none because PEP 377 (suggesting exactly this feature) was rejected.
Instead of not yielding we can raise an Exception ourselves:
#contextlib.contextmanager
def contextexceptgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
raise NoGoodDriverException(geterrordetails(driver))
finally:
release(driver)
but now you need to handle the exception:
try:
with contextexceptgetdriver(devicename) as driver:
dowork(driver)
except NoGoodDriverException as e:
handledrivererror(e.errordetails)
which has almost the same cost of code complexity as the explicit testing for good driver above.
The difference: with an exception we can decide to not handle it here and instead let it bubble up the call stack and handle it elsewhere.
Also difference: by the time we handle the exception the driver has already been released. While with the explicit check the driver has not been released. (the except is outside of the with statement while the else is inside the with statement)
I found that abusing a generator works quite well as a replacement of a context manager which can skip the body
def generatorgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
But then the calling code looks very much like a loop
for driver in generatorgetdriver(devicename):
dowork(driver)
If you can live with this (please don't) then you have a context manager that can conditionaly execute the body.
It seems hat the only way to prevent the boilerplate code is with a callback
def workwithdevice(devicename, callback):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
callback(driver)
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
And the calling code
workwithdevice(devicename, dowork)
This functionality seems to have been rejected. Python developers often prefer the explicit variant:
if need_more_workers():
newcomm = get_new_comm(comm)
# ...
You can also use higher-order functions:
def filter_comm(comm, nworkitems, callback):
if foo:
callback(get_new_comm())
# ...
some_local_var = 5
def do_work_with_newcomm(newcomm):
# we can access the local scope here
filter_comm(comm, nworkitems, do_work_with_newcomm)
How about something like this instead:
#filter_comm(comm, nworkitems)
def _(newcomm): # Name is unimportant - we'll never reference this by name.
... do work with communicator newcomm...
You implement the filter_comm decorator to do whatever work it should with comm and nworkitems, then based on those results decide whether to execute the function it's wrapped around or not, passing in newcomm.
It's not quite as elegant as with, but I think this is a bit more readable and closer to what you want than the other proposals. You could name the inner function something other than _ if you don't like that name, but I went with it since it's the normal name used in Python when the grammar requires a name which you'll never actually use.
I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm