I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm
Related
Long story short, I am writing python code that occasionally causes an underlying module to spit out complaints in the terminal that I want my code to respond to. My question is if there is some way that I can take in all terminal outputs as a string while the program is running so that I might parse it and execute some handler code. Its not errors that crash the program entirely and not a situation where I can simply do a try catch. Thanks for any help!
Edit: Running on Linux
there are several solutions to your need. the easiest would be to use a shared buffer of sort and get all your package output there instead of stdout (with regular print) thus keeping your personal streams under your package control.
since you probably already have some code with print or you want for it to work with minimal change, i suggest use the contextlib.redirect_stdout with a context manager.
give it a shared io.StringIO instance and wrap all your method with it.
you can even create a decorator to do it automatically.
something like:
// decorator
from contextlib import redirect_stdout
import io
import functools
SHARED_BUFFER = io.StringIO()
def std_redirecter(func):
#functools.wrap(func)
def inner(*args, **kwargs):
with redirect_stdout(SHARED_BUFFER) as buffer:
print('foo')
print('bar')
func(*args, **kwargs)
return inner
// your files
#std_redirecter
def writing_to_stdout_func():
print('baz')
// invocation
writing_to_stdout_func()
string = SHARED_BUFFER.getvalue() // 'foo\nbar\nbaz\n'
I've made a map editor in Python2.7.9 for a small project and I'm looking for ways to preserve the data I edit in the event of some unhandled exception. My editor already has a method for saving out data, and my current solution is to have the main loop wrapped in a try..finally block, similar to this example:
import os, datetime #..and others.
if __name__ == '__main__':
DataMgr = DataManager() # initializes the editor.
save_note = None
try:
MainLoop() # unsurprisingly, this calls the main loop.
except Exception as e: # I am of the impression this will catch every type of exception.
save_note = "Exception dump: %s : %s." % (type(e).__name__, e) # A memo appended to the comments in the save file.
finally:
exception_fp = DataMgr.cwd + "dump_%s.kmap" % str(datetime.datetime.now())
DataMgr.saveFile(exception_fp, memo = save_note) # saves out to a dump file using a familiar method with a note outlining what happened.
This seems like the best way to make sure that, no matter what happens, an attempt is made to preserve the editor's current state (to the extent that saveFile() is equipped to do so) in the event that it should crash. But I wonder if encapsulating my entire main loop in a try block is actually safe and efficient and good form. Is it? Are there risks or problems? Is there a better or more conventional way?
Wrapping the main loop in a try...finally block is the accepted pattern when you need something to happen no matter what. In some cases it's logging and continuing, in others it's saving everything possible and quitting.
So you're code is fine.
If your file isn't that big, I would suggest maybe reading the entire input file into memory, closing the file, then doing your data processing on the copy in memory, this will solve any problems you have with not corrupting your data at the cost of potentially slowing down your runtime.
Alternatively, take a look at the atexit python module. This allows you to register a function(s) for a automatic callback function when the program exits.
That being said what you have should work reasonably well.
from contextlib import closing
def init_db():
with closing(connect_db()) as db:
with app.open_resource('schema.sql') as f:
db.cursor().executescript(f.read())
db.commit()
This is from flask tutorial Step 3(http://flask.pocoo.org/docs/tutorial/dbinit/#tutorial-dbinit). And I'm little curious about the line 4 of that.
Must I import and use that 'contextlib.closing()' method?
When I've learned about with statement, many articles said that it closes file automatically after process like below.(same as Finally: thing.close())
with open('filename','w') as f:
f.write(someString);
Even though I don't use that contextlib.closing() like below, What's difference?
It's from version 2.7.6, Thank you.
def init_db():
with connect_db() as db:
with app.open_resource('schema.sql') as f:
db.cursor().executescript(f.read())
db.commit()
Yes, you should be using context.closing(); your own version does something different entirely.
The with statement lets a context manager know when a block of code is entered and exited; on exit the context manager is also given access to the exception, if one occurred. File objects use this to automatically close the file when the block is exited.
The connect_db() function from the tutorial returns a sqlite3 connection object, which can indeed be used as a context manager. However, the connection.__exit__() method doesn't close the connection, it commits the transaction on a successful completion, or aborts it when there is an exception.
The contextlib.closing() context manager on the other hand, calls the connection.close() method on the connection. This is something entirely different.
So, your second snippet may work, but does something different. The tutorial code closes the connection, your version commits a transaction. You are already calling db.commit(), so the action is actually redundant provided no exceptions are raised.
You could use the connection as a context manager again to have the automatic transaction handling behaviour:
def init_db():
with closing(connect_db()) as db:
with app.open_resource('schema.sql') as f, db:
db.cursor().executescript(f.read())
Note the , db on the second with line, ensuring that the db.__exit__() method is called when the block exits.
The only thing done by the with statement is to call __enter__ method before entering its block and __exit__ method before exiting it.
If those methods are not defined the with statement won't work as you may expect. I don't know what is the return type of connect_db, but I guess that it could be many different things from different third-party libraries. So, your code without closing will probably work in many (all?) cases, but you never know what can be returned by connect_db.
Does anyone have a real world example outside of python's file object implementation of an __enter__ and __exit__ use case? Preferably your own, since what I'm trying to achieve is a better way to conceptualize the cases where it would be used.
I've already read this.
And, here's a link to the python documentation.
There are many uses. Just in the standard library we have:
sqlite3; using the connection as a context manager translates to committing or aborting the transaction.
unittest; using assertRaises as a context manager lets you assert an exception is raised, then test aspects of the exception.
decimal; localcontext manages decimal number precision, rounding, and other aspects.
threading objects such as locks, semaphores and conditions are context managers too; letting you obtain a lock for a set of statements, etc.
the warnings module gives you a context manager to temporarily catch warnings.
many libraries offer closing behaviour, just like the default file object. These include the tarfile and the zipfile modules.
Python's own test.test_support module uses several context managers, to check for specific warnings, capture stdout, ignore specific exceptions and temporarily set environment variables.
Any time you want to detect when a block of code starts and / or ends, you want to use a context manager. Where before you'd use try: with a finally: suite to guarantee a cleanup, use a context manager instead.
I've found it useful to have a contextmanager version of os.chdir(): on exit it chdir()s back to the original directory.
This allows you to emulate a common (Bourne) shell-scripting pattern:
(
cd <some dir>
<do stuff>
)
I.e. you change to a new dir <some dir> inside a subshell (the ( )) so that you are sure to return to your original dir, even if the <do stuff> causes an error.
Compare the context manager and vanilla versions in Python. Vanilla:
original_dir = os.getcwd()
os.chdir(<some dir>)
try:
<do stuff>
finally:
os.chdir(original_dir)
Using a context manager:
with os.chdir(<some dir>):
<do stuff>
The latter is much nicer!
There are a few examples on the Python Wiki.
The canonical answer is locks:
with (acquire some mutex):
# do stuff with mutex
Here's a Stack Overflow question and answer involving locks and the with statement.
I end up having to write and support short python wrapper scripts with the following high-level structure:
try:
code
...
...
except:
raise
finally:
file_handle.close()
db_conn.close()
Notice that all I do in the except block is re-raise the exception to the script caller sans window-dressing; this is not a problem in my particular context. The idea here is that cleanup code should always be executed by means of the finally block, exception or not.
Am I better off using an atexit handler for this purpose? I could do without the extra level of indentation introduced by try.
The atexit module provides a simple interface to register functions to be called when a program closes down normally. Functions registered are automatically executed upon normal interpreter termination.
import atexit
def cleanup():
print 'performimg cleanup'
# multiple functions can be registered here...
atexit.register(cleanup)
The sys module also provides a hook, sys.exitfunc, but only one function can be registered there.
Finally is accompanied by try except block, functionality of finally can also be used for something similar like cleanup, however at finally block sys.exc_info is all-None.
If the finally clause raises another exception, the saved exception is discarded however you can put try except in the function registered with atexit to handle them.
Another pro-con is atexit functions are only executes when program terminates, however you can use finally (with try-except) anywhere in the code and perform the cleanup
In you scenario, where you want to raise an exception from cleanup content, usage of atexit would be helpful, if you are ok for cleanup to happen at the end of the program
Just use contextlib.closing
with closing(resource1) as f1, closing(resource2) as f2:
f1.something()
f2.something()
And they will be automatically closed. Files objects can be used directly as contexts so you don't need the closing call.
If close is not the only method used by your resources, you can create custom functions with the contextlib.contextmanager decorator.
atexit is called upon program termination, so this is not what you are looking for.