What is the main difference between the two? I have been studying Python and came across them. A decorator is essentially a function that wraps another function and you can do anything before and after a particular function executes.
def my_decorator(some_function):
def wrapper(*args, **kwargs):
print("Do something before the function is called")
some_function(*args, **kwargs)
print("Do something after the function is called")
return wrapper
#my_decorator
def addition(a, b):
result = a+b
print("Addition of {} and {} is {}".format(a,b,result))
But after studying Context Manager, I couldn't help but notice that it too has a enter and exit where you could do most similar operations.
from contextlib import contextmanager
#contextmanager
def open_file(path, mode):
the_file = open(path, mode)
yield the_file
the_file.close()
files = []
for x in range(100000):
with open_file('foo.txt', 'w') as infile:
files.append(infile)
for f in files:
if not f.closed:
print('not closed')
Everything before yield is taken as part of the "enter" and everything after a part of "exit".
Although both Context Managers and Decorators are syntactically different, their behaviors can be looked upon as similar. So what is the difference? What are the different scenarios when one should use either of them?
They are completely separate concepts and should not be seen in the same light.
A decorator lets you augment or replace a function or a class when it is defined. This is far broader than just executing things before or after a function call. Sure, your specific decorator lets you do something just before and after a function call, provided no exception is raised, or you explicitly handle exceptions. But you could also use a decorator to add an attribute to the function object, or to update some kind of registry. Or to return something entirely different and ignore the original function. Or to produce a wrapper that manipulates the arguments passed in, or the return value of the original function. A context manager can't do any of those things.
A context manager on the other hand lets you abstract away try: ... finally: constructs, in that no matter how the block exits, you get to execute some more code at the end of the block. Even if the block raises an exception, or uses return to exit a function, the context manager __exit__ method is still going to be called, regardless. A context manager can even suppress any exceptions raised in the block.
The two concepts are otherwise not related at all. Use decorators when you require to do something to or with functions or classes when they are defined. Use context managers when you want to clean up or take other actions after a block ends.
they are completely different concepts.
context managers are objects to be used with the python with keyword. It runs code when entering the block and exiting the block.
decorators are modifications to a function or class definition. It runs code that replaces the function as it is being defined.
#D
def Y(...):
...
is just another way of writing
def Y(...):
....
Y = D(Y)
Good thinking, indeed the concepts have many similiarities, though there are important differences, so it is safer to think of them as totally different concepts
Any context manager created with contextlib.contextmanager is also a decorator, as described here: https://docs.python.org/3/library/contextlib.html#using-a-context-manager-as-a-function-decorator
Context managers can be used to wrap code with setup and teardown steps. Decorators are a more general construct with allow us to modify functions many ways, including by wrapping them with setup/teardown logic. So it seems pretty natural to ask: why can't we use a context manager as a decorator?
We can, and in fact contextlib has already done it for you. If we write a context manager like so:
from contextlib import contextmanager
#contextmanager
def my_context():
print("setup")
yield
print("teardown")
We can use it as a context manager in a with block or we can use it as a decorator:
def foo():
with my_context():
print("foo ran")
#my_context()
def bar():
print("bar ran")
>>> foo()
setup
foo ran
teardown
>>> bar()
setup
bar ran
teardown
Which should you use?
Use a with block when your enclosed code needs access to the object returned by the context manager, e.g. file handling:
with open("my_file.txt") as file:
file.read() # needs access to the file object
Use as a decorator when an entire function needs to be wrapped in a context and doesn't need any context variables:
#contextmanager
def suppress_all_exceptions():
try: yield
except: pass
#suppress_all_exceptions()
def div_by_zero():
print("hi")
x = 1 / 0 # exception suppressed
Note: the same functionality can also be achieved by subclassing contextlib.ContextDecorator:
class MyContext(contextlib.ContextDecorator):
def __enter__(): ...
def __exit__(*errs): ...
Related
I'm learning to use decorators at the moment but am struggling to wrap my head around their purpose and utility.
I initially thought they provided the convenient option to add extra functionality to an existing function (e.g. func()) without changing its source code, but if the additional functionality is executed whenever func() is called thereafter, then why wouldn't you just save the time/space/complexity and add the new functionality to func() directly?
E.g. Say I wanted to make a function print whenever it is executed, then wouldn't this:
def add(*args):
out = sum(args)
print("Function 'add' executed.")
return out
create the exact same function as below, with far less code/complexity?
def log(func):
def wrapper(*args, **kwargs):
out = func(*args, **kwargs)
print(f"Function '{func.__name__}' executed.")
return out
return wrapper
#log
def add(*args):
return sum(args)
Off the top of my head, the only cases I can think of where the latter could potentially be preferable is if you're importing a generalised decorator to modify a function defined in a separate script, or are applying the same func to many functions so are saving space by just putting it in its own function (in which case it would seem more reasonable to just write a regular old function to simply execute it normally inside others).
EDIT
Perhaps a better way to formulate this question:
Which of the following is preferable, and why?
def logger():
# New functionality here
return
def func1(*args):
# func1 functionality
logger()
return
def func2(*args):
# func2 functionality
logger()
return
Or
def logger(func):
def wrapper(*args, **kwargs):
out = func(*args, **kwargs)
# New functionality here
return out
return wrapper
#logger
def func1(*args):
# func1 functionality
return
#logger
def func2(*args):
# func2 functionality
return
It promotes code reuse and separation of concerns.
To take your argument to the logical extreme, why use functions at all? Why not just have one giant main? A decorator is just a higher-order function and provides a lot of the same benefits that "traditional" functions do; they just solve a slightly different set of problems.
In your example, what if you wanted to change your log implementation to use the logging package instead of print? You would have to find every single function where you copy-pasted the logging behavior to change each implementation. Changing a single decorator's implementation would save you a lot of time making changes (and fixing bugs that arise from making those changes).
Decorators are typically used for behavior (in the decorator function) that wraps or modifies another set of behavior (in the decorated function). Some concrete examples could include:
Start a timer when the decorated function starts, stop it when the function returns, and log the total runtime.
Inspect the function's arguments, mutate some inputs, inject new arguments, or mutate the function's return value (see functools.cache).
Catch and handle certain types of exceptions raised from inside the decorated function.
Register the current function with some other object as a callback (see Flask).
Run the decorated function within a temporary working directory, and clean up the directory when the function returns.
As many others have stated, you can do all of the above without decorators. But all of these cases could be made cleaner and easier to maintain with the help of decorators.
One benefit is minimizing clutter and repetition in your function implementations. Consider:
#log
def add(*args):
return sum(args)
#log
def mult(*args):
return math.product(*args)
vs:
def add(*args):
out = sum(args)
print("Function add executed.")
return out
def mult(*args):
out = math.product(*args)
print("Function mult executed.")
return out
and imagine that repetition over, say, a hundred functions in a large codebase.
If you kept the log function and used it without decorator syntax you might have something like:
def _add(*args):
return sum(args)
def _mult(*args):
return math.product(*args)
add = log(_add)
mult = log(_mult)
which isn't the worst thing in the world, but it'd be annoying for a reader having to bounce through a level of indirection each time they try to look up a function's implementation.
The most important factor to the benefit of decorators is the DRY code principle
"Do Not Repeat Yourself" as an principle lends itself to creating easy-to-understand, easy-to-write code. Python's decorators are a fantastic example of features that minimise unnecessary code repetition:
Consider the #dataclass decorator. In short, it allows for classes which store only instance attributes to be written easier, as shown in the following example:
class Person:
def __init__(self, name, age, gender):
self.name = name
self.age = age
self.gender = gender
versus
#dataclass
class Person:
name: str
age: int
gender: str
The important idea about decorators to realise, however, is that creating the #dataclass decorator and writing the second (better) implementation of Person DO INDEED take more time than just writing the first Person implementation.
However, very critically, the difference emerges when you write a second, or third data-oriented class! At this point, by writing the #dataclass decorator, the creation of every single class can be sped up, by removing the boilerplate.
This example generalises to all decorators: writing a #log decorator is slow for one function, but worth it to log 100 different functions.
I am new to Python. I come from C++.
In some code reviews, I've had several peers wanting me to move things from init and del to a start and stop method. Most of them time, this goes against the RAII that was beaten into my head with decades of C++.
https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization
Is RAII not a thing in Python?
Shouldn't it be?
After all, we can throw exceptions and we'd want to release resources when we do, no?
If it isn't. Can someone give some insight as to why things are done differently? Is there a language feature that I don't understand?
if I have:
class Poop:
def __init__:
# Get some Windows Resource
def __del__:
#Release some Windows Resource
def foo():
poop = Poop()
raise Exception("Poop happens")
The Windows Resource is released, right?
RAII works in C++ because destruction is deterministic.
In garbage collected languages like Python, your object could theoretically never be destroyed, even if you call del on it.
Anyway, the idiomatic way to handle resources in Python is not with RAII, nor with start/stop, but with context managers.
The simplest example is with a file object:
with open('this_file.txt') as f:
# ... do stuff with f ...
# ... back to code that doesn't touch f ...
The with statement is, more or less, a try-finally block that creates a resource and ensures that the resource is cleaned up when the block ends; something like this:
try:
f = open('this_file.txt')
# ... do stuff with f ...
finally:
f.close()
# ... back to code that doesn't touch f ...
I don't know Java, but I believe that the JVM also uses garbage collection, and similarly try-finally is an idiom for resource management in Java.
Anyway, the with statement takes a context manager, which is an instance of a class defining the __enter__ and __exit__ methods (see the docs).
For completeness, there may be cases where you want a context manager, but don't want to define a whole class just for that. In that case, contextlib may help.
A worked example; say you have a resource:
class Resource:
def method(self):
pass
get_resource = Resource
release_resource = lambda x: None
A RAII-like class might look something like this:
class RAIILike:
def __init__(self):
self.resource = get_resource()
def __del__(self):
release_resource(self.resource)
def do_complex_thing(self):
# do something complex with resource
pass
raii_thingy = RAIILike()
And you would use the resource like this:
raii_thingy.resource.method()
On the other hand, a context managed resource could look like this...
class ContextManagedResource:
def __enter__(self):
self._resource = get_resource()
return self._resource
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is not None:
# handle exception here
pass
else:
pass
release_resource(self._resource)
return True
...and be used like this:
with ContextManagedResource() as res:
res.method()
Once the with block ends, the resource will be automatically released, regardless of whether the object that obtained it has been garbage collected.
Your own reference to wikipedia says:
Perl, Python (in the CPython implementation), and PHP manage
object lifetime by reference counting, which makes it possible to use
RAII. Objects that are no longer referenced are immediately destroyed
or finalized and released, so a destructor or finalizer can release
the resource at that time. However, it is not always idiomatic in such
languages, and is specifically discouraged in Python (in favor of
context managers and finalizers from the weakref package).
You can do RAII in python, or get pretty close. However, unlike C++ where you do the work in the constuctor and destructor, in python you need to use the dunder functions of enter and exit. This post has a excellent write up of how to write the functions and how they will behave in the presence of exceptions: https://preshing.com/20110920/the-python-with-statement-by-example/
I have the following class that implements the context manager protocol:
class Indenter:
def __init__(self):
self.level = 0
def __enter__(self):
self.level += 1
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.level -= 1
def print(self, text):
print('\t' * self.level + text)
The following code:
with Indenter() as indent:
indent.print('bye!')
with indent:
indent.print('goodbye')
with indent:
indent.print('au revoir')
indent.print('bye bye')
produces the following output:
bye!
goodbye
au revoir
bye bye
Now, I want to produce the same functionality, but instead of implementing a class, I want to use the contextmanager decorator. I have the following code so far:
class Indenter:
def __init__(self):
self.level = 0
def print(self, text):
print('\t' * self.level + text)
#contextmanager
def indenter():
try:
i = Indenter()
i.level += 1
yield i
finally:
i.level -= 1
However, I cannot produce the same output when I call:
with indenter() as indent:
indent.print('hi!')
with indent:
indent.print('hello')
with indent:
indent.print('bonjour')
indent.print('hey')
What am I doing wrong? Is it possible to achieve what I am doing with the class that implements the context manager with the function decorated by the contextmanager decorator?
Main question:
Is it possible to convert any class that implements the context manager protocol into a function that uses the contextmanager decorator? What are the limitations of each option? Are there cases in which one is better than the other?
You can't do what you're trying to do, at least not directly.
Your Indenter.__enter__ returns an Indenter object. Then, your nested with indent: uses that Indenter object as a context manager—which is fine, because it is one.
Your indenter function yields an Indenter object. Then, your nested with indent: uses that Indenter object as a context manager—which fails, because it isn't one.
You'd need to change things so that what you return is not an Indenter object, but another call to indenter. While this is possible (any class can be rewritten as a closure), it's probably not what you want here.
If you're willing to change the API slightly, you can do this:
#contextmanager
def indenter():
level=0
#contextmanager
def _indenter():
nonlocal level
try:
level += 1
yield
finally:
level -= 1
def _print(text):
print('\t' * level + text)
_indenter.print = _print
yield _indenter
Now, indenter doesn't create a context manager, but it does create a function that returns a context manager. That's inherent in what the #contextmanager decorator does—just as you have to do with indenter() as indent:, not with indenter as indent:, you'll have to do with indent():, not with indent.
Otherwise, it's all pretty straightforward. Instead of using recursion, I just created a new function that stores level in a closure. And then we can contextmanager it and tack the print method on. And now:
>>> with indenter() as indent:
... indent.print('hi!')
... with indent():
... indent.print('hello')
... with indent():
... indent.print('bonjour')
... indent.print('hey')
hi!
hello
bonjour
hey
If you're wondering why we can't just yield _indenter() (well, we'd have to call _indenter(), then tack the print onto the result, then yield that, but that's not the main issue), the problem is that contextmanager demands a generator function that yields once, and gives you a single-use context manager every time you call it. If you read the contextlib source, you can see how you could write something like contextmanager that instead takes a function that yields alternating enters and exits forever and gives you a context manager that does a next for each __enter__ and __exit__. Or you could write a class that that creates the generator on __enter__ instead of on __init__ so it can do the _recreate_cm thing properly the same way it does when used as a decorator instead of a context manager. But at that point, to avoid writing a class, you're writing two classes and a decorator, which seems a bit silly.
If you're interested in more, you should check out contextlib2, a third-party module written by Nick Coghlan and the other authors of the stdlib contextlib. It's used both for backporting contextlib features to older versions of Python, and for experimenting with new features for future versions of Python. IIRC, at one point, they had a version of #contextmanager that was reusable, but removed it because of a bug that couldn't be worked around cleanly.
I have been looking into Python's contextmanager (more specifically, Python 3's contextlib or its back ported contextlib2) as of late, and I was wondering what were the advantages/disadvantages writing them as a class vs. a function?
They both seem to function the same way and handle exceptions the same way. There's a lot of cool utilities like ExitStack(), but those utilities seem to be implementable in context managers written as classes or functions. Thus, I'm struggling to find a good reason as to why one would want to write context managers verbosely as a class when they can be written as a function and just slap on the contextmanager decorator.
Here's a trivial example I wrote to showcase both doing the same thing:
# !/usr/bin/python -u
# encoding: utf-8
from contextlib import contextmanager
# Function-based
#contextmanager
def func_custom_open(filename, mode):
try:
f = open(filename, mode)
yield f
except Exception as e:
print(e)
finally:
f.close()
# Class-based
class class_custom_open(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
def __enter__(self):
return self.f
def __exit__(self, type, value, traceback):
self.f.close()
if __name__ == '__main__':
# Function-based
with func_custom_open('holafile_func.txt', 'w') as func_f:
func_f.write('hola func!')
# Class-based
with class_custom_open('holafile_class.txt', 'w') as class_f:
class_f.write('hola class!')
if you don't need the "verbose" class using syntax, you don't need it, its that simple.
The reason both are present is that the way using classes is the actual way context managers work in the language. Any object having an __enter__ and an __exit__ method in its class can be used as a context manager.
The way using #contextmanager and allowing one to declare context managers as functions is just a practical utility in Python's standard library. What the decorator yields is an object that have both methods nonetheless.
One case in which writting a context manager as a class may be more compact is when the object which is used as a context manager is also a class under your control, and then you can better integrate running __enter__ and __exit__ in its life cycle. For example, it is not uncommon to see objects that can be used either as decorators or as context managers (unittest.mock.patch comes to my mind). It sounds just "magic" to the user, but the implementation is quite clear and well defined: in the class of such an object, the logic for it to behave as a context manager is on __enter__/__exit__, and the logic implementing the decorator behavior is on the __call__ method.
I'm looking to add a decorator that runs a video recorder on certain tests like so:
#decorators.video(self)
def test_1234(self):
...
I'm having trouble passing the self variable into the decorator as it is need for some attributes. How can I do this?
theodox answer is generally good, but for decorators you should use functools.wraps function, like in an example below:
from functools import wraps
def enable_video(fn)
'''Decorate the function to start video, call the function, stop video.'''
#wraps(fn)
def inner(*args, **kwargs):
# could be just `def inner(self):` if only intended to use
# with methods without arguments and keyword arguments
do_stuff_before()
fn(*args, **kwargs)
do_stuff_after()
return inner
It will persist original docstrings, original function name (and more). You can read more about it in Python docs.
Then, assuming that previous code is in decorators module, you should use it as follows:
class MyTestCase(unittests.TestCase);
#decorators.enable_video
def testSomeVideoFunction(self):
do_test_stuff()
Note that in the code example it's just #decorators.enable_video, not #decorators.enable_video(self). As like in jonrsharpe's comment to your question, reference to a self is not present at a decoration time.
Are you sure you need the self reference at all?
More commonly you'd do something like this
def enable_video(fn):
'''decorate the test function so it starts the video, runs the test, and stop the video'''
def video_aware_test(self_refrence):
start_video_recorder()
try:
fn()
finally:
stop_video_recorder()
return video_aware_test
And you'd apply it like this:
#enable_video
def test_something(self)
If for some reason the decorator actually needed the self reference, you can see where you'd grab it. This version doesn't include configuring the video recorder in any way, to that you'd use a class rather than a function decorator and pass the configuration as arguments.