I'm required to use "with" on a method of an object and not on the object itself.
Here is what I've already tried:
class LSTM:
...
def run(self):
def __enter__(self):
do something
return self
def __exit__(self, type, value, tb):
return self
An example of I want to use the function in main:
lstm = LSTM(...)
with lstm.run():
...
The error I get:
AttributeError: __enter__
The object returned by your method must be a context manager. Write your method as a generator and apply the contextlib.contextmanager decorator to automatically create the proper helper object:
from contextlib import contextmanager
class LSTM:
#contextmanager
def run(self):
# prepare
yield self
# clean up
The object created by the decorator uses anything before the yield as __enter__, and anything after it as __exit__. Whatever is provided by yield is available for use in the as clause of the with statement. If an error terminates the context, it is raised at yield.
When you write:
with soemthing:
Then the object soemthing needs to have those __enter__ & __exit__ methods.
So for:
with lstm.run():
The object returned by lstm.run() needs to have the __enter__ and __exit__ methods - the fact that lstm has those methods is of no consequence.
What you should do will depend on what you're trying to achieve. But this:
with lstm:
Will call the __enter__ & __exit__ methods you have defined.
The context expression lstm.run() of your with statement does not evaluate to a context manager but to None, since there is no return statement in your run method.
Cf. https://docs.python.org/3/reference/compound_stmts.html#the-with-statement
What is the main difference between the two? I have been studying Python and came across them. A decorator is essentially a function that wraps another function and you can do anything before and after a particular function executes.
def my_decorator(some_function):
def wrapper(*args, **kwargs):
print("Do something before the function is called")
some_function(*args, **kwargs)
print("Do something after the function is called")
return wrapper
#my_decorator
def addition(a, b):
result = a+b
print("Addition of {} and {} is {}".format(a,b,result))
But after studying Context Manager, I couldn't help but notice that it too has a enter and exit where you could do most similar operations.
from contextlib import contextmanager
#contextmanager
def open_file(path, mode):
the_file = open(path, mode)
yield the_file
the_file.close()
files = []
for x in range(100000):
with open_file('foo.txt', 'w') as infile:
files.append(infile)
for f in files:
if not f.closed:
print('not closed')
Everything before yield is taken as part of the "enter" and everything after a part of "exit".
Although both Context Managers and Decorators are syntactically different, their behaviors can be looked upon as similar. So what is the difference? What are the different scenarios when one should use either of them?
They are completely separate concepts and should not be seen in the same light.
A decorator lets you augment or replace a function or a class when it is defined. This is far broader than just executing things before or after a function call. Sure, your specific decorator lets you do something just before and after a function call, provided no exception is raised, or you explicitly handle exceptions. But you could also use a decorator to add an attribute to the function object, or to update some kind of registry. Or to return something entirely different and ignore the original function. Or to produce a wrapper that manipulates the arguments passed in, or the return value of the original function. A context manager can't do any of those things.
A context manager on the other hand lets you abstract away try: ... finally: constructs, in that no matter how the block exits, you get to execute some more code at the end of the block. Even if the block raises an exception, or uses return to exit a function, the context manager __exit__ method is still going to be called, regardless. A context manager can even suppress any exceptions raised in the block.
The two concepts are otherwise not related at all. Use decorators when you require to do something to or with functions or classes when they are defined. Use context managers when you want to clean up or take other actions after a block ends.
they are completely different concepts.
context managers are objects to be used with the python with keyword. It runs code when entering the block and exiting the block.
decorators are modifications to a function or class definition. It runs code that replaces the function as it is being defined.
#D
def Y(...):
...
is just another way of writing
def Y(...):
....
Y = D(Y)
Good thinking, indeed the concepts have many similiarities, though there are important differences, so it is safer to think of them as totally different concepts
Any context manager created with contextlib.contextmanager is also a decorator, as described here: https://docs.python.org/3/library/contextlib.html#using-a-context-manager-as-a-function-decorator
Context managers can be used to wrap code with setup and teardown steps. Decorators are a more general construct with allow us to modify functions many ways, including by wrapping them with setup/teardown logic. So it seems pretty natural to ask: why can't we use a context manager as a decorator?
We can, and in fact contextlib has already done it for you. If we write a context manager like so:
from contextlib import contextmanager
#contextmanager
def my_context():
print("setup")
yield
print("teardown")
We can use it as a context manager in a with block or we can use it as a decorator:
def foo():
with my_context():
print("foo ran")
#my_context()
def bar():
print("bar ran")
>>> foo()
setup
foo ran
teardown
>>> bar()
setup
bar ran
teardown
Which should you use?
Use a with block when your enclosed code needs access to the object returned by the context manager, e.g. file handling:
with open("my_file.txt") as file:
file.read() # needs access to the file object
Use as a decorator when an entire function needs to be wrapped in a context and doesn't need any context variables:
#contextmanager
def suppress_all_exceptions():
try: yield
except: pass
#suppress_all_exceptions()
def div_by_zero():
print("hi")
x = 1 / 0 # exception suppressed
Note: the same functionality can also be achieved by subclassing contextlib.ContextDecorator:
class MyContext(contextlib.ContextDecorator):
def __enter__(): ...
def __exit__(*errs): ...
I have been looking into Python's contextmanager (more specifically, Python 3's contextlib or its back ported contextlib2) as of late, and I was wondering what were the advantages/disadvantages writing them as a class vs. a function?
They both seem to function the same way and handle exceptions the same way. There's a lot of cool utilities like ExitStack(), but those utilities seem to be implementable in context managers written as classes or functions. Thus, I'm struggling to find a good reason as to why one would want to write context managers verbosely as a class when they can be written as a function and just slap on the contextmanager decorator.
Here's a trivial example I wrote to showcase both doing the same thing:
# !/usr/bin/python -u
# encoding: utf-8
from contextlib import contextmanager
# Function-based
#contextmanager
def func_custom_open(filename, mode):
try:
f = open(filename, mode)
yield f
except Exception as e:
print(e)
finally:
f.close()
# Class-based
class class_custom_open(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
def __enter__(self):
return self.f
def __exit__(self, type, value, traceback):
self.f.close()
if __name__ == '__main__':
# Function-based
with func_custom_open('holafile_func.txt', 'w') as func_f:
func_f.write('hola func!')
# Class-based
with class_custom_open('holafile_class.txt', 'w') as class_f:
class_f.write('hola class!')
if you don't need the "verbose" class using syntax, you don't need it, its that simple.
The reason both are present is that the way using classes is the actual way context managers work in the language. Any object having an __enter__ and an __exit__ method in its class can be used as a context manager.
The way using #contextmanager and allowing one to declare context managers as functions is just a practical utility in Python's standard library. What the decorator yields is an object that have both methods nonetheless.
One case in which writting a context manager as a class may be more compact is when the object which is used as a context manager is also a class under your control, and then you can better integrate running __enter__ and __exit__ in its life cycle. For example, it is not uncommon to see objects that can be used either as decorators or as context managers (unittest.mock.patch comes to my mind). It sounds just "magic" to the user, but the implementation is quite clear and well defined: in the class of such an object, the logic for it to behave as a context manager is on __enter__/__exit__, and the logic implementing the decorator behavior is on the __call__ method.
It is not clear to me why the Python 2.7 implementation of Pool does not have the __exit__() method that is present in the Python 3 version of the same class. Is it safe to add the __exit__() method (together with __enter__(), of course) (I just want to use with Pool(n) as p: ) or is there a special reason to avoid it?
Doesn't seem like there's any reason to avoid it. Looking at it and testing it real quick didn't bring up any odd behavior. This was implemented in Issue 15064, it just seems it wasn't added in 2.7 (probably because only bug-fixes were considered).
Returning self from __enter__ and calling terminate from __exit__ as implemented in Python 3.3 should be the way to go. Instead of altering the source though (if that was your intention), just create a custom subclass:
from multiprocessing.pool import Pool as PoolCls
class CMPool(PoolCls):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
return self.terminate()
class Package:
def __init__(self):
self.files = []
# ...
def __del__(self):
for file in self.files:
os.unlink(file)
__del__(self) above fails with an AttributeError exception. I understand Python doesn't guarantee the existence of "global variables" (member data in this context?) when __del__() is invoked. If that is the case and this is the reason for the exception, how do I make sure the object destructs properly?
I'd recommend using Python's with statement for managing resources that need to be cleaned up. The problem with using an explicit close() statement is that you have to worry about people forgetting to call it at all or forgetting to place it in a finally block to prevent a resource leak when an exception occurs.
To use the with statement, create a class with the following methods:
def __enter__(self)
def __exit__(self, exc_type, exc_value, traceback)
In your example above, you'd use
class Package:
def __init__(self):
self.files = []
def __enter__(self):
return self
# ...
def __exit__(self, exc_type, exc_value, traceback):
for file in self.files:
os.unlink(file)
Then, when someone wanted to use your class, they'd do the following:
with Package() as package_obj:
# use package_obj
The variable package_obj will be an instance of type Package (it's the value returned by the __enter__ method). Its __exit__ method will automatically be called, regardless of whether or not an exception occurs.
You could even take this approach a step further. In the example above, someone could still instantiate Package using its constructor without using the with clause. You don't want that to happen. You can fix this by creating a PackageResource class that defines the __enter__ and __exit__ methods. Then, the Package class would be defined strictly inside the __enter__ method and returned. That way, the caller never could instantiate the Package class without using a with statement:
class PackageResource:
def __enter__(self):
class Package:
...
self.package_obj = Package()
return self.package_obj
def __exit__(self, exc_type, exc_value, traceback):
self.package_obj.cleanup()
You'd use this as follows:
with PackageResource() as package_obj:
# use package_obj
The standard way is to use atexit.register:
# package.py
import atexit
import os
class Package:
def __init__(self):
self.files = []
atexit.register(self.cleanup)
def cleanup(self):
print("Running cleanup...")
for file in self.files:
print("Unlinking file: {}".format(file))
# os.unlink(file)
But you should keep in mind that this will persist all created instances of Package until Python is terminated.
Demo using the code above saved as package.py:
$ python
>>> from package import *
>>> p = Package()
>>> q = Package()
>>> q.files = ['a', 'b', 'c']
>>> quit()
Running cleanup...
Unlinking file: a
Unlinking file: b
Unlinking file: c
Running cleanup...
A better alternative is to use weakref.finalize. See the examples at Finalizer Objects and Comparing finalizers with __del__() methods.
As an appendix to Clint's answer, you can simplify PackageResource using contextlib.contextmanager:
#contextlib.contextmanager
def packageResource():
class Package:
...
package = Package()
yield package
package.cleanup()
Alternatively, though probably not as Pythonic, you can override Package.__new__:
class Package(object):
def __new__(cls, *args, **kwargs):
#contextlib.contextmanager
def packageResource():
# adapt arguments if superclass takes some!
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
yield package
package.cleanup()
def __init__(self, *args, **kwargs):
...
and simply use with Package(...) as package.
To get things shorter, name your cleanup function close and use contextlib.closing, in which case you can either use the unmodified Package class via with contextlib.closing(Package(...)) or override its __new__ to the simpler
class Package(object):
def __new__(cls, *args, **kwargs):
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
return contextlib.closing(package)
And this constructor is inherited, so you can simply inherit, e.g.
class SubPackage(Package):
def close(self):
pass
Here is a minimal working skeleton:
class SkeletonFixture:
def __init__(self):
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
pass
def method(self):
pass
with SkeletonFixture() as fixture:
fixture.method()
Important: return self
If you're like me, and overlook the return self part (of Clint Miller's correct answer), you will be staring at this nonsense:
Traceback (most recent call last):
File "tests/simplestpossible.py", line 17, in <module>
fixture.method()
AttributeError: 'NoneType' object has no attribute 'method'
Hope it helps the next person.
I don't think that it's possible for instance members to be removed before __del__ is called. My guess would be that the reason for your particular AttributeError is somewhere else (maybe you mistakenly remove self.file elsewhere).
However, as the others pointed out, you should avoid using __del__. The main reason for this is that instances with __del__ will not be garbage collected (they will only be freed when their refcount reaches 0). Therefore, if your instances are involved in circular references, they will live in memory for as long as the application run. (I may be mistaken about all this though, I'd have to read the gc docs again, but I'm rather sure it works like this).
I think the problem could be in __init__ if there is more code than shown?
__del__ will be called even when __init__ has not been executed properly or threw an exception.
Source
Just wrap your destructor with a try/except statement and it will not throw an exception if your globals are already disposed of.
Edit
Try this:
from weakref import proxy
class MyList(list): pass
class Package:
def __init__(self):
self.__del__.im_func.files = MyList([1,2,3,4])
self.files = proxy(self.__del__.im_func.files)
def __del__(self):
print self.__del__.im_func.files
It will stuff the file list in the del function that is guaranteed to exist at the time of call. The weakref proxy is to prevent Python, or yourself from deleting the self.files variable somehow (if it is deleted, then it will not affect the original file list). If it is not the case that this is being deleted even though there are more references to the variable, then you can remove the proxy encapsulation.
It seems that the idiomatic way to do this is to provide a close() method (or similar), and call it explicitely.
A good idea is to combine both approaches.
To implement a context manager for explicit life-cycle handling. As well as handle cleanup in case the user forgets it or it is not convenient to use a with statement. This is best done by weakref.finalize.
This is how many libraries actually do it. And depending on the severity, you could issue a warning.
It is guaranteed to be called exactly once, so it is safe to call it at any time before.
import os
from typing import List
import weakref
class Package:
def __init__(self):
self.files = []
self._finalizer = weakref.finalize(self, self._cleanup_files, self.files)
#staticmethod
def _cleanup_files(files: List):
for file in files:
os.unlink(file)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
self._finalizer()
weakref.finalize returns a callable finalizer object which will be called when obj is garbage collected. Unlike an ordinary weak reference, a finalizer will always survive until the reference object is collected, greatly simplifying lifecycle management."
Unlike atexit.register the object is not held in memory until the interpreter is shut down.
And unlike object.__del__, weakref.finalize is guaranteed to be called at interpreter shutdown. So it is much more safe.
atexit.register is the standard way as has already been mentioned in ostrakach's answer.
However, it must be noted that the order in which objects might get deleted cannot be relied upon as shown in example below.
import atexit
class A(object):
def __init__(self, val):
self.val = val
atexit.register(self.hello)
def hello(self):
print(self.val)
def hello2():
a = A(10)
hello2()
a = A(20)
Here, order seems legitimate in terms of reverse of the order in which objects were created as program gives output as :
20
10
However when, in a larger program, python's garbage collection kicks in object which is out of it's lifetime would get destructed first.