I'm writing an MPI-based application (but MPI doesn't matter in my question, I mention it only to expose the rationale) and in some cases, when there is less work items than processes, I need to create a new communicator excluding the processes that have nothing to do. Finally, the new communicator has to be freed by the processes that have work to do (and only by them).
A neat way to do that would be to write:
with filter_comm(comm, nworkitems) as newcomm:
... do work with communicator newcomm...
the body being executed only by the processes that have work to do.
Is there a way in a context manager to avoid executing the body?
I understand that context managers have rightfully been designed to avoid hiding control flows, but I wonder if it is possible to circumvent that, since in my case I think it would be justified for clarity sake.
The ability to conditionally skip context manager body has been proposed but rejected as documented in PEP 377.
I did some research about alternatives. Here are my findings.
First let me explain the background of my code examples. You have a bunch of devices you want to work with. For every device you have to acquire the driver for the device; then work with the device using the driver; and lastly release the driver so others can acquire the driver and work with the device.
Nothing out of the ordinary here. The code looks roughly like this:
driver = getdriver(devicename)
try:
dowork(driver)
finally:
releasedriver(driver)
But once every full moon when the planets are not aligned correctly the acquired driver for a device is bad and no work can be done with the device. This is no big deal. Just skip the device this round and try again next round. Usually the driver is good then. But even a bad driver needs to be released otherwise no new driver can be acquired.
(the firmware is proprietary and the vendor is reluctant to fix or even acknowledge this bug)
The code now looks like this:
driver = getdriver(devicename)
try:
if isgooddriver(driver):
dowork(driver)
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
That is a lot of boilerplate code that needs to be repeated everytime work needs to be done with a device. A prime candidate for python's context manager also known as with statement. It might look like this:
# note: this code example does not work
#contextlib.contextmanager
def contextgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
And then the code when working with a device is short and sweet:
# note: this code example does not work
with contextgetdriver(devicename) as driver:
dowork(driver)
But this does not work. Because a context manager has to yield. It may not not yield. Not yielding will result in a RuntimeException raised by contextmanager.
So we have to pull out the check from the context manager
#contextlib.contextmanager
def contextgetdriver(devicename):
driver = getdriver(devicename)
try:
yield driver
finally:
release(driver)
and put it in the body of the with statement
with contextgetdriver(devicename) as driver:
if isgooddriver(driver):
dowork(driver)
else:
handledrivererror(geterrordetails(driver))
This is ugly because now we again have some boilerplate that needs to be repeated everytime we want to work with a device.
So we want a context manager that can conditionaly execute the body. But we have none because PEP 377 (suggesting exactly this feature) was rejected.
Instead of not yielding we can raise an Exception ourselves:
#contextlib.contextmanager
def contextexceptgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
raise NoGoodDriverException(geterrordetails(driver))
finally:
release(driver)
but now you need to handle the exception:
try:
with contextexceptgetdriver(devicename) as driver:
dowork(driver)
except NoGoodDriverException as e:
handledrivererror(e.errordetails)
which has almost the same cost of code complexity as the explicit testing for good driver above.
The difference: with an exception we can decide to not handle it here and instead let it bubble up the call stack and handle it elsewhere.
Also difference: by the time we handle the exception the driver has already been released. While with the explicit check the driver has not been released. (the except is outside of the with statement while the else is inside the with statement)
I found that abusing a generator works quite well as a replacement of a context manager which can skip the body
def generatorgetdriver(devicename):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
yield driver
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
But then the calling code looks very much like a loop
for driver in generatorgetdriver(devicename):
dowork(driver)
If you can live with this (please don't) then you have a context manager that can conditionaly execute the body.
It seems hat the only way to prevent the boilerplate code is with a callback
def workwithdevice(devicename, callback):
driver = getdriver(devicename)
try:
if isgooddriver(driver):
callback(driver)
else:
handledrivererror(geterrordetails(driver))
finally:
release(driver)
And the calling code
workwithdevice(devicename, dowork)
This functionality seems to have been rejected. Python developers often prefer the explicit variant:
if need_more_workers():
newcomm = get_new_comm(comm)
# ...
You can also use higher-order functions:
def filter_comm(comm, nworkitems, callback):
if foo:
callback(get_new_comm())
# ...
some_local_var = 5
def do_work_with_newcomm(newcomm):
# we can access the local scope here
filter_comm(comm, nworkitems, do_work_with_newcomm)
How about something like this instead:
#filter_comm(comm, nworkitems)
def _(newcomm): # Name is unimportant - we'll never reference this by name.
... do work with communicator newcomm...
You implement the filter_comm decorator to do whatever work it should with comm and nworkitems, then based on those results decide whether to execute the function it's wrapped around or not, passing in newcomm.
It's not quite as elegant as with, but I think this is a bit more readable and closer to what you want than the other proposals. You could name the inner function something other than _ if you don't like that name, but I went with it since it's the normal name used in Python when the grammar requires a name which you'll never actually use.
Related
I am writing test scripts using Selenium Webdriver and I have created a simple framework for tests step-by-step execution. My goal was to improve the readability of the code as well as the logic of test scenarios. To do that I wrote some wrappers that group some logic together.
The main ideas are as follows:
The application(webpage) that is being tested is defined as a class.
Big chunks of code that define a single logical test step are defined as a method of a class. For example a class WebPage1 and possible method is LogIn.
Each method (like described in 2) then is wrapped during the execution phase to handle any possible exceptions that might be raised.
Additionally, I have a waitForElement wrapper function that waits for certain condition on the webpage. If that condition was not met within a pre-defined period of time, it raises a TimeoutException which is being handled internally, within the same function.
So to give some idea of what I am talking about, please consider the below example:
class Application(object):
def __init__(self):
#Some code
def waitForElement(self, elementName, expectedCondition, searchBy):
try:
#conditions
except TimeoutException:
#handling
def logIn(self):
#code
self.waitForElement(...)
#code
app = Application() # initialize an object of class Application
# The below chunk of code attempts to complete the logIn() test step and if ANY
# exceptions are raised during its execution they will be caught by the BaseException
try:
app.logIn()
except BaseException as exception:
#handling
So, you can see that the whole step (logIn()) in this case is wrapped inside a try and there is additional wrapper inside the waitForElement.
Now I want to fail the whole execution if TimeoutException is raised during waitForElement and I want to catch it in the outside wrapper (the BaseException). And I still want to handle it locally (inside waitForElement).
My first idea was to to raise an additional exception after the initial one is handled. So I did something like that:
def waitForElement(self, elementName, expectedCondition, searchBy):
try:
#conditions
except TimeoutException:
#handling
raise SystemException
So my 2 questions are:
Does that approach make sense? It doesn't look very clean to me so maybe I miss something? Another approach perhaps?
Ideally, I still want to get the same TimeoutException rather than SystemExit in the outer exception. If my solution is OK, should I just re-raise the TimeoutException?
P.S. Hope all that iss clear and make sense. Please let me know in the comments if I should refine my description.
from contextlib import closing
def init_db():
with closing(connect_db()) as db:
with app.open_resource('schema.sql') as f:
db.cursor().executescript(f.read())
db.commit()
This is from flask tutorial Step 3(http://flask.pocoo.org/docs/tutorial/dbinit/#tutorial-dbinit). And I'm little curious about the line 4 of that.
Must I import and use that 'contextlib.closing()' method?
When I've learned about with statement, many articles said that it closes file automatically after process like below.(same as Finally: thing.close())
with open('filename','w') as f:
f.write(someString);
Even though I don't use that contextlib.closing() like below, What's difference?
It's from version 2.7.6, Thank you.
def init_db():
with connect_db() as db:
with app.open_resource('schema.sql') as f:
db.cursor().executescript(f.read())
db.commit()
Yes, you should be using context.closing(); your own version does something different entirely.
The with statement lets a context manager know when a block of code is entered and exited; on exit the context manager is also given access to the exception, if one occurred. File objects use this to automatically close the file when the block is exited.
The connect_db() function from the tutorial returns a sqlite3 connection object, which can indeed be used as a context manager. However, the connection.__exit__() method doesn't close the connection, it commits the transaction on a successful completion, or aborts it when there is an exception.
The contextlib.closing() context manager on the other hand, calls the connection.close() method on the connection. This is something entirely different.
So, your second snippet may work, but does something different. The tutorial code closes the connection, your version commits a transaction. You are already calling db.commit(), so the action is actually redundant provided no exceptions are raised.
You could use the connection as a context manager again to have the automatic transaction handling behaviour:
def init_db():
with closing(connect_db()) as db:
with app.open_resource('schema.sql') as f, db:
db.cursor().executescript(f.read())
Note the , db on the second with line, ensuring that the db.__exit__() method is called when the block exits.
The only thing done by the with statement is to call __enter__ method before entering its block and __exit__ method before exiting it.
If those methods are not defined the with statement won't work as you may expect. I don't know what is the return type of connect_db, but I guess that it could be many different things from different third-party libraries. So, your code without closing will probably work in many (all?) cases, but you never know what can be returned by connect_db.
Does anyone have a real world example outside of python's file object implementation of an __enter__ and __exit__ use case? Preferably your own, since what I'm trying to achieve is a better way to conceptualize the cases where it would be used.
I've already read this.
And, here's a link to the python documentation.
There are many uses. Just in the standard library we have:
sqlite3; using the connection as a context manager translates to committing or aborting the transaction.
unittest; using assertRaises as a context manager lets you assert an exception is raised, then test aspects of the exception.
decimal; localcontext manages decimal number precision, rounding, and other aspects.
threading objects such as locks, semaphores and conditions are context managers too; letting you obtain a lock for a set of statements, etc.
the warnings module gives you a context manager to temporarily catch warnings.
many libraries offer closing behaviour, just like the default file object. These include the tarfile and the zipfile modules.
Python's own test.test_support module uses several context managers, to check for specific warnings, capture stdout, ignore specific exceptions and temporarily set environment variables.
Any time you want to detect when a block of code starts and / or ends, you want to use a context manager. Where before you'd use try: with a finally: suite to guarantee a cleanup, use a context manager instead.
I've found it useful to have a contextmanager version of os.chdir(): on exit it chdir()s back to the original directory.
This allows you to emulate a common (Bourne) shell-scripting pattern:
(
cd <some dir>
<do stuff>
)
I.e. you change to a new dir <some dir> inside a subshell (the ( )) so that you are sure to return to your original dir, even if the <do stuff> causes an error.
Compare the context manager and vanilla versions in Python. Vanilla:
original_dir = os.getcwd()
os.chdir(<some dir>)
try:
<do stuff>
finally:
os.chdir(original_dir)
Using a context manager:
with os.chdir(<some dir>):
<do stuff>
The latter is much nicer!
There are a few examples on the Python Wiki.
The canonical answer is locks:
with (acquire some mutex):
# do stuff with mutex
Here's a Stack Overflow question and answer involving locks and the with statement.
My code looks like this:
... # class Site(Resource)
def render_POST(self,request)
otherclass.doAssync(request.args)
print '1'
return "done" #that returns the HTTP response, always the same.
...
def doAssync(self,msg):
d = defer.Deferred()
reactor.callLater(0,self.doStuff,d,msg)
d.addCallback(self.sucess)
def doStuff(self,d,msg):
# do some stuff
time.sleep(2) #just for example
d.callback('ok')
def sucess(msg):
print msg
The output:
1
ok
So far, so good, but, the HTTP response (return 'done'), only happens after the delay (time.sleep(2)).
I can tell this, because the browser keeps 'loading' for 2 seconds.
What am I doing wrong?
What you are doing wrong is running a blocking call (time.sleep(2)), while Twisted expects you to only perform non-blocking operations. Things that don't wait. Because you have that time.sleep(2) in there, Twisted can't do anything else while that function is sleeping. So it can't send any data to the browser, either.
In the case of time.sleep(2), you would replace that with another reactor.callLater call. Assuming you actually meant for the time.sleep(2) call to be some other blocking operation, how to fix it depends on the operation. If you can do the operation in a non-blocking way, do that. For many such operations (like database interaction) Twisted already comes with non-blocking alternatives. If the thing you're doing has no non-blocking interface and Twisted doesn't have an alternative to it, you may have to run the code in a separate thread (using for example twisted.internet.threads.deferToThread), although that requires your code is actually thread-safe.
I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm