i'm writing a python class that create connection to remote server by using fabric2 library.
I'm wondering which is the right writing design for my class.
for example:
class SshToServer:
"""ssh connection to server."""
def __init__(self, ip, user, ssh_key_path):
self.ssh_client = Connection(
host=ip,
user=user,
port=22,
connect_kwargs={"key_filename": ssh_key_path},
inline_ssh_env=True,
)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.ssh_client.close()
First it would be great to get general info before:
I should use __enter__ and __exit__ methods only if i plan to create the object with "with" statement? there is a situation that i will choose to use those methods without the "with" statement creation?
What is the meaning of using both __init__ and __enter__ inside the class? what's the difference between them? because they both run when i create object. using one of them is enough ?
Same for using __exit__ and __del__ class methods.
back to my code:
now i'm going to write my module and basically i want to create the connection object based on some parameter, for example base on environment.
if it's environment X connect to server X - same for environment Y:
with SshToServer(ip=BLA_1, user=BLA_1, ssh_key_path=BLA_1) as conn:
do_stuff...
with SshToServer(ip=BLA_2, user=BLA+2, ssh_key_path=BLA_2) as conn:
do_stuff...
and it's the same methods in both connection, it's just different remote server.
so i'm guessing that using "with" is not the smart way in that case (cause i duplicate the code).
i want to change the object creation way to:
conn = SshToServer(...)
do_stuff...
conn.__exit__ / conn.__del__
But i'm confusing between those 4 methods and when i should use them.
__enter__ and __exit__ methods are use to perform setup and teardown actions.
It will automatically close the connection for you. __exit__ is called when you exit the with scope.
In this case you are creating two new connections so you need to create two context managers.
I don't know about the Connection class. if it lets you configure user separately then you can do is
with SshToServer(ip=BLA_1, ssh_key_path=BLA_1) as conn:
conn.user = BLA_1
// do some stuff
conn.user = BLA_2
// do some stuff
Yes, they are only intended for usage with with.
In your sample code, __init__ is called first to create (actually only initialize) an object (as usual), then __enter__ is called to use the object as context manager (only in with). Both methods have their own uses.
Calling special methods directly is usually a bad idea. Exception: Calling in a special method the same method of the super class.
Main advantage of with is that __exit__ is called under nearly all circumstances and can't be forgotten.
In your code you could e.g. write the two with as one like
conn = SshToServer(...) # Somehow select which object to create
with conn:
do_stuff...
This is much cleaner and safer than calling special methods directly.
Related
I have a class that needs to run a TensorFlow session for each instance of the class, as long as that instance exists.
TensorFlow sessions use context managers, but I don't want to force anyone who uses my class to put my class into a context manager.
Is there any way to auto-close the session once the instance is no longer in use without using a context manager?
Can I just put in an __exit__ method and not an __enter__ method and start the session without the context manager and just close the session in the exit?
Is there any way to auto-close the session once the instance is no longer in use without using a context manager?
Not really, how would an object figure out when it’s no longer being used? If there was a safe way to do this, there wouldn’t be a need for context managers in the first place.
So you have to use context managers and the with statement to get this kind of feedback. But just because you have to use context managers, that does not mean that you actually need to have some separate “thing” you open. You can return anything in the __enter__ method, including the current object.
So the simplest context manager implementation that closes itself when the context is closed looks like this:
class MyClass:
def __enter__ (self):
return self
def __exit__ (self, *exc):
self.close()
def close (self):
# actually close the object
In fact, this pattern is so common, that there is a built-in recipe for this context manager: contextlib.closing. Using that, you do not actually need to modify your class at all, you can just wrap it in a closing() call and have it call close when the context is exited:
with closing(my_object):
my_object.do_something()
# my_object.close() is automatically called
You must define an __enter__ method, but you can just define it as:
def __enter__(self):
return self
and have the session defined in the init. Then, define __exit__ like so:
def __exit__(self, *exc):
self.close()
Then, define a close method that closes whatever resources were opened in __init__. (In my case, it's a TensorFlow session.)
This way, if the user decides to use the context manager, it will close it for them, and if they don't, they'll have to close it on their own.
If I have a class that wraps a resource, e.g., an sqlite database connection or a file, is there a way I can use the with statement to close the resource when my object goes out of scope or is gcollected?
To clarify what I mean, I want to avoid this:
class x:
def __init__(self):
# open resource
def close(self): # or __del__, even worst
# close resource
but make it in such a way that the resource is always freed as in
with open('foo') as f:
# use resource
You need to provide __enter__ and __exit__ methods. See PEP 343.
This PEP adds a new statement "with" to the Python language to make it
possible to factor out standard uses of try/finally statements.
In this PEP, context managers provide __enter__() and __exit__()
methods that are invoked on entry to and exit from the body of the
with statement.
Use contextlib.closing:
with contextlib.closing(thing) as thing:
do_stuff_with(thing)
# Thing is closed now.
You can always put any cleanup code you need into a class's __del__ method:
class x:
def __init__(self):
self.thing = get_thing()
def __del__(self):
self.thing.close()
But you shouldn't.
This is a bad idea, for a few reasons. If you're using CPython, having custom __del__ methods means the GC can't break reference cycles. If you're using most other Python implementations, __del__ methods aren't called at predictable times.
This is why you usually put cleanup in explicit close methods. That's the best you can do within the class itself. It's always up to the user of your class to make sure the close method gets called, not the class itself.
So, there's no way you can use a with statement, or anything equivalent, inside your class. But you can make it easier for users of your class to use a with statement, by making your class into a context manager, as described in roippi's answer, or just by suggesting they use contextlib.closing in your documentation.
Some background: I work in a large bank and I'm trying to re-use some Python modules, which I cannot change, only import. I also don't have the option of installing any new utilities/functions etc (running Python 2.6 on Linux).
I've got this at present:
In my module:
from common.databaseHelper import BacktestingDatabaseHelper
class mfReportProcess(testingResource):
def __init__(self):
self.db = BacktestingDatabaseHelper.fromConfig('db_name')
One of the methods called within the 'testingResource' class has this:
with self.db as handler:
which falls over with this:
with self.db as handler:
AttributeError: 'BacktestingDatabaseHelper' object has no attribute '__exit__'
and, indeed, there is no __exit__ method in the 'BacktestingDatabaseHelper' class, a class which I cannot change.
However, this code I'm trying to re-use works perfectly well for other apps - does anyone know why I get this error and no-one else?
Is there some way of defining __exit__ locally?
Many thanks in advance.
EDITED to add:
I've tried to add my own class to setup DB access but can't get it to work - added this to my module:
class myDB(BacktestingDatabaseHelper):
def __enter__(self):
self.db = fromConfig('db_name')
def __exit__(self):
self.db.close()
and added:
self.db = myDB
into my init attribute for my main class but I get this error:
with self.db as handler:
TypeError: unbound method __enter__() must be called with myDB instance as first argument (got nothing instead)
Any suggestions as to how to do this properly?
Using the with protocol assumes that the object used in with implements the context manager protocol.
Basically this means that the class definition should have __enter__() and __exit__() methods defined. If you use an object without these, python will throw an AttributeError complaining about the missing __exit__ attribute.
The error means that BacktestingDatabaseHelper is not designed to be used in a with statement. Sounds like the classes testingResource and BacktestingDatabaseHelper are not compatible with each other (perhaps your version of common.databaseHelper is out of date).
As you cannot change the with statement, you must add a class deriving from BacktestingDatabaseHelper which adds appropriate __enter__() and __exit__() functions and use this instead.
Here is an example which tries to be as close to the original as possible:
class myDB(BacktestingDatabaseHelper):
def __enter__(self):
return self
def __exit__(self):
self.db.close()
def fromConfig(self, name):
x = super(myDB, self).fromConfig(name)
assert isinstance(x, BacktestingDatabaseHelper)
x.__class__ = myDB # not sure if that really works
[...]
self.db=myDB.fromConfig('tpbp')
The problem is, however, that I am not sure what the __enter__ is supposed to return. If you take MySQLdb, for example, the context manager of the connection creates a cursor representing one transaction. If that's the case here as well, wou have to work out something else...
You might want to try the contextlib.contextmanager decorator to wrap your object so that it supports the context manager protocol.
The 'with' keyword is basically a shortcut for writing out:
try:
// Do something
finally:
hander.__exit__()
which is useful if your handler object is using up resources (like, for example, an open file stream). It makes sure that no matter what happens in the 'do something' part, the resource is released cleanly.
In your case, your handler object doesn't have an __exit__ method, and so with fails. I would assume that other people can use BacktestingDatabaseHelper because they're not using with.
As for what you can do now, I would suggest forgetting with and using try ... finally instead, rather than trying to add your own version of __exit__ to the object. You'll just have to make sure you release the handler properly (how you do this will depend on how BacktestingDatabaseHelper is supposed to be used), e.g.
try:
handler = self.db
// do stuff
finally:
handler.close()
Edit:
Since you can't change it, you should do something like #Daniel Roseman suggests to wrap BacktestingDatabaseHelper. Depending on how best to clean up BacktestingDatabaseHelper (as above), you can write something like:
from contextlib import contextmanager
#contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
and use this as:
class mfReportProcess(testingResource):
def __init__(self):
self.db = closing(BacktestingDatabaseHelper.fromConfig('db_name'))
(this is directly from the documentation).
Using SQLAlchemy to connect to MySQL and I've gotten tired of writing things like this:
with closing(engine) as connection:
do_sql_stuff(connection)
This pattern is repeated throughout many areas of my code and it seems with the availability of __del__ this is not necessary. Why not just implement a class to wrap the connection creation and closing:
class MyConnectionManager(object):
def __init__(self, db_uri):
self.__db_engine = sqlalchemy.create_engine(db_uri)
self.__db_conn = self.__db_engine.connect()
def __del__(self):
self.__db_conn.close()
Is this simply two different styles/preferences, or are there more important reasons that using with closing() is a better way to go that using __del__ (or vice versa)?
There is no guarantee about when __del__ is actually called (or if it is called at all in the case of circular references). with closing(...) as ...: guarantees that the cleanup code is called whenever you exit the with clause.
I have an object with an internal database connection that's active throughout its lifetime. At the end of the program's run, the connection has to be committed and closed. So far I've used an explicit close method, but this is somewhat cumbersome, especially when exceptions can happen in the calling code.
I'm considering using the __del__ method for closing, but after some reading online I have concerns. Is this a valid usage pattern? Can I be sure that the internal resources will be freed in __del__ correctly?
This discussion raised a similar question but found no satisfactory answer. I don't want to have an explicit close method, and using with isn't an option, because my object isn't used as simply as open-play-close, but is kept as a member of another, larger object, that uses it while running in a GUI.
C++ has perfectly working destructors where one can free resources safely, so I would imagine Python has something agreed-upon too. For some reason it seems not to be the case, and many in the community vow against __del__. What's the alternative, then?
Read up on the with statement. You're describing its use case.
You'll need to wrap your connection in a "Context Manager" class that handles the __enter__ and __exit__ methods used by the with statement.
See PEP 343 for more information.
Edit
"my object isn't used as simply as open-play-close, but is kept as a member of another, larger object"
class AnObjectWhichMustBeClosed( object ):
def __enter__( self ):
# acquire
def __exit__( self, type, value, traceback ):
# release
def open( self, dbConnectionInfo ):
# open the connection, updating the state for __exit__ to handle.
class ALargerObject( object ):
def __init__( self ):
pass
def injectTheObjectThatMustBeClosed( self, anObject ):
self.useThis = anObject
class MyGuiApp( self ):
def run( self ):
# build GUI objects
large = ALargeObject()
with AnObjectWhichMustBeClosed() as x:
large.injectTheObjectThatMustBeClosed( x )
mainLoop()
Some folks call this "Dependency Injection" and "Inversion of Control". Other folks call this the Strategy pattern. The "ObjectThatMustBeClosed" is a strategy, plugged into some larger object. The assembly is created at a top-level of the GUI app, since that's usually where resources like databases are acquired.
You can make a connection module, since modules keep the same object in the whole application, and register a function to close it with the atexit module
# db.py:
import sqlite3
import atexit
con = None
def get_connection():
global con
if not con:
con = sqlite3.connect('somedb.sqlite')
atexit.register(close_connection, con)
return con
def close_connection(some_con):
some_con.commit()
some_con.close()
# your_program.py
import db
con = db.get_connection()
cur = con.cursor()
cur.execute("SELECT ...")
This sugestion is based on the assumption that the connection in your application seems like a single instance (singleton) which a module global provides well.
If that's not the case, then you can use a destructor.
However destructors don't go well with garbage collectors and circular references (you must remove the circular reference yourself before the destructor is called) and if that's not the case (you need multiple connections) then you can go for a destructor. Just don't keep circular references around or you'll have to break them yourself.
Also, what you said about C++ is wrong. If you use destructors in C++ they are called either when the block that defines the object finishes (like python's with) or when you use the delete keyword (that deallocates an object created with new). Outside that you must use an explicit close() that is not the destructor. So it is just like python - python is even "better" because it has a garbage collector.