I have been looking into Python's contextmanager (more specifically, Python 3's contextlib or its back ported contextlib2) as of late, and I was wondering what were the advantages/disadvantages writing them as a class vs. a function?
They both seem to function the same way and handle exceptions the same way. There's a lot of cool utilities like ExitStack(), but those utilities seem to be implementable in context managers written as classes or functions. Thus, I'm struggling to find a good reason as to why one would want to write context managers verbosely as a class when they can be written as a function and just slap on the contextmanager decorator.
Here's a trivial example I wrote to showcase both doing the same thing:
# !/usr/bin/python -u
# encoding: utf-8
from contextlib import contextmanager
# Function-based
#contextmanager
def func_custom_open(filename, mode):
try:
f = open(filename, mode)
yield f
except Exception as e:
print(e)
finally:
f.close()
# Class-based
class class_custom_open(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
def __enter__(self):
return self.f
def __exit__(self, type, value, traceback):
self.f.close()
if __name__ == '__main__':
# Function-based
with func_custom_open('holafile_func.txt', 'w') as func_f:
func_f.write('hola func!')
# Class-based
with class_custom_open('holafile_class.txt', 'w') as class_f:
class_f.write('hola class!')
if you don't need the "verbose" class using syntax, you don't need it, its that simple.
The reason both are present is that the way using classes is the actual way context managers work in the language. Any object having an __enter__ and an __exit__ method in its class can be used as a context manager.
The way using #contextmanager and allowing one to declare context managers as functions is just a practical utility in Python's standard library. What the decorator yields is an object that have both methods nonetheless.
One case in which writting a context manager as a class may be more compact is when the object which is used as a context manager is also a class under your control, and then you can better integrate running __enter__ and __exit__ in its life cycle. For example, it is not uncommon to see objects that can be used either as decorators or as context managers (unittest.mock.patch comes to my mind). It sounds just "magic" to the user, but the implementation is quite clear and well defined: in the class of such an object, the logic for it to behave as a context manager is on __enter__/__exit__, and the logic implementing the decorator behavior is on the __call__ method.
Related
I am new to Python. I come from C++.
In some code reviews, I've had several peers wanting me to move things from init and del to a start and stop method. Most of them time, this goes against the RAII that was beaten into my head with decades of C++.
https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization
Is RAII not a thing in Python?
Shouldn't it be?
After all, we can throw exceptions and we'd want to release resources when we do, no?
If it isn't. Can someone give some insight as to why things are done differently? Is there a language feature that I don't understand?
if I have:
class Poop:
def __init__:
# Get some Windows Resource
def __del__:
#Release some Windows Resource
def foo():
poop = Poop()
raise Exception("Poop happens")
The Windows Resource is released, right?
RAII works in C++ because destruction is deterministic.
In garbage collected languages like Python, your object could theoretically never be destroyed, even if you call del on it.
Anyway, the idiomatic way to handle resources in Python is not with RAII, nor with start/stop, but with context managers.
The simplest example is with a file object:
with open('this_file.txt') as f:
# ... do stuff with f ...
# ... back to code that doesn't touch f ...
The with statement is, more or less, a try-finally block that creates a resource and ensures that the resource is cleaned up when the block ends; something like this:
try:
f = open('this_file.txt')
# ... do stuff with f ...
finally:
f.close()
# ... back to code that doesn't touch f ...
I don't know Java, but I believe that the JVM also uses garbage collection, and similarly try-finally is an idiom for resource management in Java.
Anyway, the with statement takes a context manager, which is an instance of a class defining the __enter__ and __exit__ methods (see the docs).
For completeness, there may be cases where you want a context manager, but don't want to define a whole class just for that. In that case, contextlib may help.
A worked example; say you have a resource:
class Resource:
def method(self):
pass
get_resource = Resource
release_resource = lambda x: None
A RAII-like class might look something like this:
class RAIILike:
def __init__(self):
self.resource = get_resource()
def __del__(self):
release_resource(self.resource)
def do_complex_thing(self):
# do something complex with resource
pass
raii_thingy = RAIILike()
And you would use the resource like this:
raii_thingy.resource.method()
On the other hand, a context managed resource could look like this...
class ContextManagedResource:
def __enter__(self):
self._resource = get_resource()
return self._resource
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is not None:
# handle exception here
pass
else:
pass
release_resource(self._resource)
return True
...and be used like this:
with ContextManagedResource() as res:
res.method()
Once the with block ends, the resource will be automatically released, regardless of whether the object that obtained it has been garbage collected.
Your own reference to wikipedia says:
Perl, Python (in the CPython implementation), and PHP manage
object lifetime by reference counting, which makes it possible to use
RAII. Objects that are no longer referenced are immediately destroyed
or finalized and released, so a destructor or finalizer can release
the resource at that time. However, it is not always idiomatic in such
languages, and is specifically discouraged in Python (in favor of
context managers and finalizers from the weakref package).
You can do RAII in python, or get pretty close. However, unlike C++ where you do the work in the constuctor and destructor, in python you need to use the dunder functions of enter and exit. This post has a excellent write up of how to write the functions and how they will behave in the presence of exceptions: https://preshing.com/20110920/the-python-with-statement-by-example/
I have the following class that implements the context manager protocol:
class Indenter:
def __init__(self):
self.level = 0
def __enter__(self):
self.level += 1
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.level -= 1
def print(self, text):
print('\t' * self.level + text)
The following code:
with Indenter() as indent:
indent.print('bye!')
with indent:
indent.print('goodbye')
with indent:
indent.print('au revoir')
indent.print('bye bye')
produces the following output:
bye!
goodbye
au revoir
bye bye
Now, I want to produce the same functionality, but instead of implementing a class, I want to use the contextmanager decorator. I have the following code so far:
class Indenter:
def __init__(self):
self.level = 0
def print(self, text):
print('\t' * self.level + text)
#contextmanager
def indenter():
try:
i = Indenter()
i.level += 1
yield i
finally:
i.level -= 1
However, I cannot produce the same output when I call:
with indenter() as indent:
indent.print('hi!')
with indent:
indent.print('hello')
with indent:
indent.print('bonjour')
indent.print('hey')
What am I doing wrong? Is it possible to achieve what I am doing with the class that implements the context manager with the function decorated by the contextmanager decorator?
Main question:
Is it possible to convert any class that implements the context manager protocol into a function that uses the contextmanager decorator? What are the limitations of each option? Are there cases in which one is better than the other?
You can't do what you're trying to do, at least not directly.
Your Indenter.__enter__ returns an Indenter object. Then, your nested with indent: uses that Indenter object as a context manager—which is fine, because it is one.
Your indenter function yields an Indenter object. Then, your nested with indent: uses that Indenter object as a context manager—which fails, because it isn't one.
You'd need to change things so that what you return is not an Indenter object, but another call to indenter. While this is possible (any class can be rewritten as a closure), it's probably not what you want here.
If you're willing to change the API slightly, you can do this:
#contextmanager
def indenter():
level=0
#contextmanager
def _indenter():
nonlocal level
try:
level += 1
yield
finally:
level -= 1
def _print(text):
print('\t' * level + text)
_indenter.print = _print
yield _indenter
Now, indenter doesn't create a context manager, but it does create a function that returns a context manager. That's inherent in what the #contextmanager decorator does—just as you have to do with indenter() as indent:, not with indenter as indent:, you'll have to do with indent():, not with indent.
Otherwise, it's all pretty straightforward. Instead of using recursion, I just created a new function that stores level in a closure. And then we can contextmanager it and tack the print method on. And now:
>>> with indenter() as indent:
... indent.print('hi!')
... with indent():
... indent.print('hello')
... with indent():
... indent.print('bonjour')
... indent.print('hey')
hi!
hello
bonjour
hey
If you're wondering why we can't just yield _indenter() (well, we'd have to call _indenter(), then tack the print onto the result, then yield that, but that's not the main issue), the problem is that contextmanager demands a generator function that yields once, and gives you a single-use context manager every time you call it. If you read the contextlib source, you can see how you could write something like contextmanager that instead takes a function that yields alternating enters and exits forever and gives you a context manager that does a next for each __enter__ and __exit__. Or you could write a class that that creates the generator on __enter__ instead of on __init__ so it can do the _recreate_cm thing properly the same way it does when used as a decorator instead of a context manager. But at that point, to avoid writing a class, you're writing two classes and a decorator, which seems a bit silly.
If you're interested in more, you should check out contextlib2, a third-party module written by Nick Coghlan and the other authors of the stdlib contextlib. It's used both for backporting contextlib features to older versions of Python, and for experimenting with new features for future versions of Python. IIRC, at one point, they had a version of #contextmanager that was reusable, but removed it because of a bug that couldn't be worked around cleanly.
What is the main difference between the two? I have been studying Python and came across them. A decorator is essentially a function that wraps another function and you can do anything before and after a particular function executes.
def my_decorator(some_function):
def wrapper(*args, **kwargs):
print("Do something before the function is called")
some_function(*args, **kwargs)
print("Do something after the function is called")
return wrapper
#my_decorator
def addition(a, b):
result = a+b
print("Addition of {} and {} is {}".format(a,b,result))
But after studying Context Manager, I couldn't help but notice that it too has a enter and exit where you could do most similar operations.
from contextlib import contextmanager
#contextmanager
def open_file(path, mode):
the_file = open(path, mode)
yield the_file
the_file.close()
files = []
for x in range(100000):
with open_file('foo.txt', 'w') as infile:
files.append(infile)
for f in files:
if not f.closed:
print('not closed')
Everything before yield is taken as part of the "enter" and everything after a part of "exit".
Although both Context Managers and Decorators are syntactically different, their behaviors can be looked upon as similar. So what is the difference? What are the different scenarios when one should use either of them?
They are completely separate concepts and should not be seen in the same light.
A decorator lets you augment or replace a function or a class when it is defined. This is far broader than just executing things before or after a function call. Sure, your specific decorator lets you do something just before and after a function call, provided no exception is raised, or you explicitly handle exceptions. But you could also use a decorator to add an attribute to the function object, or to update some kind of registry. Or to return something entirely different and ignore the original function. Or to produce a wrapper that manipulates the arguments passed in, or the return value of the original function. A context manager can't do any of those things.
A context manager on the other hand lets you abstract away try: ... finally: constructs, in that no matter how the block exits, you get to execute some more code at the end of the block. Even if the block raises an exception, or uses return to exit a function, the context manager __exit__ method is still going to be called, regardless. A context manager can even suppress any exceptions raised in the block.
The two concepts are otherwise not related at all. Use decorators when you require to do something to or with functions or classes when they are defined. Use context managers when you want to clean up or take other actions after a block ends.
they are completely different concepts.
context managers are objects to be used with the python with keyword. It runs code when entering the block and exiting the block.
decorators are modifications to a function or class definition. It runs code that replaces the function as it is being defined.
#D
def Y(...):
...
is just another way of writing
def Y(...):
....
Y = D(Y)
Good thinking, indeed the concepts have many similiarities, though there are important differences, so it is safer to think of them as totally different concepts
Any context manager created with contextlib.contextmanager is also a decorator, as described here: https://docs.python.org/3/library/contextlib.html#using-a-context-manager-as-a-function-decorator
Context managers can be used to wrap code with setup and teardown steps. Decorators are a more general construct with allow us to modify functions many ways, including by wrapping them with setup/teardown logic. So it seems pretty natural to ask: why can't we use a context manager as a decorator?
We can, and in fact contextlib has already done it for you. If we write a context manager like so:
from contextlib import contextmanager
#contextmanager
def my_context():
print("setup")
yield
print("teardown")
We can use it as a context manager in a with block or we can use it as a decorator:
def foo():
with my_context():
print("foo ran")
#my_context()
def bar():
print("bar ran")
>>> foo()
setup
foo ran
teardown
>>> bar()
setup
bar ran
teardown
Which should you use?
Use a with block when your enclosed code needs access to the object returned by the context manager, e.g. file handling:
with open("my_file.txt") as file:
file.read() # needs access to the file object
Use as a decorator when an entire function needs to be wrapped in a context and doesn't need any context variables:
#contextmanager
def suppress_all_exceptions():
try: yield
except: pass
#suppress_all_exceptions()
def div_by_zero():
print("hi")
x = 1 / 0 # exception suppressed
Note: the same functionality can also be achieved by subclassing contextlib.ContextDecorator:
class MyContext(contextlib.ContextDecorator):
def __enter__(): ...
def __exit__(*errs): ...
If I have a class that wraps a resource, e.g., an sqlite database connection or a file, is there a way I can use the with statement to close the resource when my object goes out of scope or is gcollected?
To clarify what I mean, I want to avoid this:
class x:
def __init__(self):
# open resource
def close(self): # or __del__, even worst
# close resource
but make it in such a way that the resource is always freed as in
with open('foo') as f:
# use resource
You need to provide __enter__ and __exit__ methods. See PEP 343.
This PEP adds a new statement "with" to the Python language to make it
possible to factor out standard uses of try/finally statements.
In this PEP, context managers provide __enter__() and __exit__()
methods that are invoked on entry to and exit from the body of the
with statement.
Use contextlib.closing:
with contextlib.closing(thing) as thing:
do_stuff_with(thing)
# Thing is closed now.
You can always put any cleanup code you need into a class's __del__ method:
class x:
def __init__(self):
self.thing = get_thing()
def __del__(self):
self.thing.close()
But you shouldn't.
This is a bad idea, for a few reasons. If you're using CPython, having custom __del__ methods means the GC can't break reference cycles. If you're using most other Python implementations, __del__ methods aren't called at predictable times.
This is why you usually put cleanup in explicit close methods. That's the best you can do within the class itself. It's always up to the user of your class to make sure the close method gets called, not the class itself.
So, there's no way you can use a with statement, or anything equivalent, inside your class. But you can make it easier for users of your class to use a with statement, by making your class into a context manager, as described in roippi's answer, or just by suggesting they use contextlib.closing in your documentation.
I'm a noob with Python, but I've written an auto-close function like this..
#contextmanager
def AutoClose(obj):
try:
yield obj
finally:
obj.Close()
I have three classes that have a Close() method that this function can be used with. Is this the most Pythonic solution? Should I be doing something in the classes themselves instead?
Most pythonic solution is to define methods __enter__ and __exit__ methods in your class:
class Foo(object):
def __init__(self, filename):
self.filename = filename
def __enter__(self):
self.fd = open(self.filename)
def __exit__(self, exc_type, exc_value, traceback):
self.fd.close()
And using:
with Foo('/path/to/file') as foo:
# do something with foo
Methods __enter__ and __exit__ will be implicitly called when entering and leaving blocks with. Also note that __exit__ allows you to catch exception which raised inside the block with.
Function contextlib.closing is typically used for those classes that do not explicitly define the methods __enter__ and __exit__ (but have a method close). If you define your own classes, much better way is to define these methods.
What you're doing looks totally fine and Pythonic. Although, the contextlib standard library already has something similar, but you'll have to rename your Close methods to close.
import contextlib
with contextlib.closing(thing):
print thing
I would recommend using this instead. After all, the recommended naming convention for Python methods is all_lowercase_with_underscores.