I am having a deeper look at the Session class inside the Sqlalchemy library in sqlalchemy.orm.session.py (link here) and i see this block inside the Session class at line 1170:
#util.contextmanager
def _maker_context_manager(self):
with self:
with self.begin():
yield self
I don't understand the syntax and what it does. Why is there a with self: at the start? Can we use with and any class? Can someone please explain this and how it is useful in the context of sqlalchemy session?
with self: points to __enter__ and __exit__ methods. These methods act as context managers. Here's an excerpt from Python's doc on context managers:
Python’s with statement supports the concept of a runtime context defined by a context manager. This is implemented using a pair of methods that allow user-defined classes to define a runtime context that is entered before the statement body is executed and exited when the statement ends:
contextmanager.__enter__(): Enter the runtime context and return either this object or another object related to the runtime context [...]
contextmanager.__exit__(exc_type, exc_val, exc_tb): Exit the runtime
context [...].
From Session (here):
def __enter__(self: _S) -> _S:
return self
def __exit__(self, type_: Any, value: Any, traceback: Any) -> None:
self.close()
Here, __enter__ returns the instance of the class. And __exit__ helps with graceful termination (closing the session when you exit the scope). Acting as internal context managers.
Related
I'm required to use "with" on a method of an object and not on the object itself.
Here is what I've already tried:
class LSTM:
...
def run(self):
def __enter__(self):
do something
return self
def __exit__(self, type, value, tb):
return self
An example of I want to use the function in main:
lstm = LSTM(...)
with lstm.run():
...
The error I get:
AttributeError: __enter__
The object returned by your method must be a context manager. Write your method as a generator and apply the contextlib.contextmanager decorator to automatically create the proper helper object:
from contextlib import contextmanager
class LSTM:
#contextmanager
def run(self):
# prepare
yield self
# clean up
The object created by the decorator uses anything before the yield as __enter__, and anything after it as __exit__. Whatever is provided by yield is available for use in the as clause of the with statement. If an error terminates the context, it is raised at yield.
When you write:
with soemthing:
Then the object soemthing needs to have those __enter__ & __exit__ methods.
So for:
with lstm.run():
The object returned by lstm.run() needs to have the __enter__ and __exit__ methods - the fact that lstm has those methods is of no consequence.
What you should do will depend on what you're trying to achieve. But this:
with lstm:
Will call the __enter__ & __exit__ methods you have defined.
The context expression lstm.run() of your with statement does not evaluate to a context manager but to None, since there is no return statement in your run method.
Cf. https://docs.python.org/3/reference/compound_stmts.html#the-with-statement
I have been looking into Python's contextmanager (more specifically, Python 3's contextlib or its back ported contextlib2) as of late, and I was wondering what were the advantages/disadvantages writing them as a class vs. a function?
They both seem to function the same way and handle exceptions the same way. There's a lot of cool utilities like ExitStack(), but those utilities seem to be implementable in context managers written as classes or functions. Thus, I'm struggling to find a good reason as to why one would want to write context managers verbosely as a class when they can be written as a function and just slap on the contextmanager decorator.
Here's a trivial example I wrote to showcase both doing the same thing:
# !/usr/bin/python -u
# encoding: utf-8
from contextlib import contextmanager
# Function-based
#contextmanager
def func_custom_open(filename, mode):
try:
f = open(filename, mode)
yield f
except Exception as e:
print(e)
finally:
f.close()
# Class-based
class class_custom_open(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
def __enter__(self):
return self.f
def __exit__(self, type, value, traceback):
self.f.close()
if __name__ == '__main__':
# Function-based
with func_custom_open('holafile_func.txt', 'w') as func_f:
func_f.write('hola func!')
# Class-based
with class_custom_open('holafile_class.txt', 'w') as class_f:
class_f.write('hola class!')
if you don't need the "verbose" class using syntax, you don't need it, its that simple.
The reason both are present is that the way using classes is the actual way context managers work in the language. Any object having an __enter__ and an __exit__ method in its class can be used as a context manager.
The way using #contextmanager and allowing one to declare context managers as functions is just a practical utility in Python's standard library. What the decorator yields is an object that have both methods nonetheless.
One case in which writting a context manager as a class may be more compact is when the object which is used as a context manager is also a class under your control, and then you can better integrate running __enter__ and __exit__ in its life cycle. For example, it is not uncommon to see objects that can be used either as decorators or as context managers (unittest.mock.patch comes to my mind). It sounds just "magic" to the user, but the implementation is quite clear and well defined: in the class of such an object, the logic for it to behave as a context manager is on __enter__/__exit__, and the logic implementing the decorator behavior is on the __call__ method.
I often see code that uses self to manage a context. For example
with self:
self.x = 4
self.y = 6
What's going on here? What does using self as a context allow?
Code that uses with self: suggests that whatever class you're using provides __enter__ and __exit__ methods. These methods create context. They can be convenient for setup / teardown in testing, etc.
What's going on here? What does using self as a context allow?
As long as the class has implemented the necessary "hooks" that a context manager should, Python allows it to be used like a normal context manager. Here is an excerpt from the docs which helps clear things up here:
Python’s with statement supports the concept of a runtime context defined by a context manager. This is implemented using a pair of methods that allow user-defined classes to define a runtime context that is entered before the statement body is executed and exited when the statement ends:
contextmanager.__enter__()
Enter the runtime context and return either this object or another object related to the runtime context. The value returned by this method is bound to the identifier in the as clause of with statements using this context manager.
[...]
contextmanager.__exit__(exc_type, exc_val, exc_tb)
Exit the runtime context and return a Boolean flag indicating if any exception that occurred should be suppressed. If an exception occurred while executing the body of the with statement, the arguments contain the exception type, value and traceback information. Otherwise, all three arguments are None.
[...]
As stated above, when you implement the necessary __enter__ and __exit__ magic methods for your class, Python allows you to treat it as a context manager.
If self is a context manager (i.e. has __enter__ and __exit__ methods) this will simply invoke that functionality, the same as it would if the instance were used in a with block outside the class.
There's nothing special happening here. self behaves the same way in a with block that anything else would. It calls __enter__ when you enter the scope and __exit__ when you leave the scope through any means. I can't imagine what using self here would accomplish, but if you can come up with some examples of where you've seen that, we might be able to provide a better answer.
I have a class that needs to run a TensorFlow session for each instance of the class, as long as that instance exists.
TensorFlow sessions use context managers, but I don't want to force anyone who uses my class to put my class into a context manager.
Is there any way to auto-close the session once the instance is no longer in use without using a context manager?
Can I just put in an __exit__ method and not an __enter__ method and start the session without the context manager and just close the session in the exit?
Is there any way to auto-close the session once the instance is no longer in use without using a context manager?
Not really, how would an object figure out when it’s no longer being used? If there was a safe way to do this, there wouldn’t be a need for context managers in the first place.
So you have to use context managers and the with statement to get this kind of feedback. But just because you have to use context managers, that does not mean that you actually need to have some separate “thing” you open. You can return anything in the __enter__ method, including the current object.
So the simplest context manager implementation that closes itself when the context is closed looks like this:
class MyClass:
def __enter__ (self):
return self
def __exit__ (self, *exc):
self.close()
def close (self):
# actually close the object
In fact, this pattern is so common, that there is a built-in recipe for this context manager: contextlib.closing. Using that, you do not actually need to modify your class at all, you can just wrap it in a closing() call and have it call close when the context is exited:
with closing(my_object):
my_object.do_something()
# my_object.close() is automatically called
You must define an __enter__ method, but you can just define it as:
def __enter__(self):
return self
and have the session defined in the init. Then, define __exit__ like so:
def __exit__(self, *exc):
self.close()
Then, define a close method that closes whatever resources were opened in __init__. (In my case, it's a TensorFlow session.)
This way, if the user decides to use the context manager, it will close it for them, and if they don't, they'll have to close it on their own.
Some background: I work in a large bank and I'm trying to re-use some Python modules, which I cannot change, only import. I also don't have the option of installing any new utilities/functions etc (running Python 2.6 on Linux).
I've got this at present:
In my module:
from common.databaseHelper import BacktestingDatabaseHelper
class mfReportProcess(testingResource):
def __init__(self):
self.db = BacktestingDatabaseHelper.fromConfig('db_name')
One of the methods called within the 'testingResource' class has this:
with self.db as handler:
which falls over with this:
with self.db as handler:
AttributeError: 'BacktestingDatabaseHelper' object has no attribute '__exit__'
and, indeed, there is no __exit__ method in the 'BacktestingDatabaseHelper' class, a class which I cannot change.
However, this code I'm trying to re-use works perfectly well for other apps - does anyone know why I get this error and no-one else?
Is there some way of defining __exit__ locally?
Many thanks in advance.
EDITED to add:
I've tried to add my own class to setup DB access but can't get it to work - added this to my module:
class myDB(BacktestingDatabaseHelper):
def __enter__(self):
self.db = fromConfig('db_name')
def __exit__(self):
self.db.close()
and added:
self.db = myDB
into my init attribute for my main class but I get this error:
with self.db as handler:
TypeError: unbound method __enter__() must be called with myDB instance as first argument (got nothing instead)
Any suggestions as to how to do this properly?
Using the with protocol assumes that the object used in with implements the context manager protocol.
Basically this means that the class definition should have __enter__() and __exit__() methods defined. If you use an object without these, python will throw an AttributeError complaining about the missing __exit__ attribute.
The error means that BacktestingDatabaseHelper is not designed to be used in a with statement. Sounds like the classes testingResource and BacktestingDatabaseHelper are not compatible with each other (perhaps your version of common.databaseHelper is out of date).
As you cannot change the with statement, you must add a class deriving from BacktestingDatabaseHelper which adds appropriate __enter__() and __exit__() functions and use this instead.
Here is an example which tries to be as close to the original as possible:
class myDB(BacktestingDatabaseHelper):
def __enter__(self):
return self
def __exit__(self):
self.db.close()
def fromConfig(self, name):
x = super(myDB, self).fromConfig(name)
assert isinstance(x, BacktestingDatabaseHelper)
x.__class__ = myDB # not sure if that really works
[...]
self.db=myDB.fromConfig('tpbp')
The problem is, however, that I am not sure what the __enter__ is supposed to return. If you take MySQLdb, for example, the context manager of the connection creates a cursor representing one transaction. If that's the case here as well, wou have to work out something else...
You might want to try the contextlib.contextmanager decorator to wrap your object so that it supports the context manager protocol.
The 'with' keyword is basically a shortcut for writing out:
try:
// Do something
finally:
hander.__exit__()
which is useful if your handler object is using up resources (like, for example, an open file stream). It makes sure that no matter what happens in the 'do something' part, the resource is released cleanly.
In your case, your handler object doesn't have an __exit__ method, and so with fails. I would assume that other people can use BacktestingDatabaseHelper because they're not using with.
As for what you can do now, I would suggest forgetting with and using try ... finally instead, rather than trying to add your own version of __exit__ to the object. You'll just have to make sure you release the handler properly (how you do this will depend on how BacktestingDatabaseHelper is supposed to be used), e.g.
try:
handler = self.db
// do stuff
finally:
handler.close()
Edit:
Since you can't change it, you should do something like #Daniel Roseman suggests to wrap BacktestingDatabaseHelper. Depending on how best to clean up BacktestingDatabaseHelper (as above), you can write something like:
from contextlib import contextmanager
#contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
and use this as:
class mfReportProcess(testingResource):
def __init__(self):
self.db = closing(BacktestingDatabaseHelper.fromConfig('db_name'))
(this is directly from the documentation).