Using SQLAlchemy to connect to MySQL and I've gotten tired of writing things like this:
with closing(engine) as connection:
do_sql_stuff(connection)
This pattern is repeated throughout many areas of my code and it seems with the availability of __del__ this is not necessary. Why not just implement a class to wrap the connection creation and closing:
class MyConnectionManager(object):
def __init__(self, db_uri):
self.__db_engine = sqlalchemy.create_engine(db_uri)
self.__db_conn = self.__db_engine.connect()
def __del__(self):
self.__db_conn.close()
Is this simply two different styles/preferences, or are there more important reasons that using with closing() is a better way to go that using __del__ (or vice versa)?
There is no guarantee about when __del__ is actually called (or if it is called at all in the case of circular references). with closing(...) as ...: guarantees that the cleanup code is called whenever you exit the with clause.
Related
As you may know, Python objects have a __del__ method. This method may or may not be called once your process stops. As such you must not use it to free resources.
Most people would tell you to use with-statements (or try-finally). I use with-statements all the time and I love them, I even create objects with __enter__/__exit__ all the time, but in my very specific case, it won't do.
Is there a way for an instance of an object to detect when its process ends?
import my_module as db
class Connection:
def __init__(self, config: db.config.DatabaseConfiguration):
self.connection = db.connect(config)
def __del__(self):
self.connection.disconnect()
# def __finally__(self) ? Something to trigger when root context ends ?
My bet is that it could be done with some good old Python meta-programming.
Context:
I work on a module that will be used by others. I can't go in their code and add with-statements.
i'm writing a python class that create connection to remote server by using fabric2 library.
I'm wondering which is the right writing design for my class.
for example:
class SshToServer:
"""ssh connection to server."""
def __init__(self, ip, user, ssh_key_path):
self.ssh_client = Connection(
host=ip,
user=user,
port=22,
connect_kwargs={"key_filename": ssh_key_path},
inline_ssh_env=True,
)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.ssh_client.close()
First it would be great to get general info before:
I should use __enter__ and __exit__ methods only if i plan to create the object with "with" statement? there is a situation that i will choose to use those methods without the "with" statement creation?
What is the meaning of using both __init__ and __enter__ inside the class? what's the difference between them? because they both run when i create object. using one of them is enough ?
Same for using __exit__ and __del__ class methods.
back to my code:
now i'm going to write my module and basically i want to create the connection object based on some parameter, for example base on environment.
if it's environment X connect to server X - same for environment Y:
with SshToServer(ip=BLA_1, user=BLA_1, ssh_key_path=BLA_1) as conn:
do_stuff...
with SshToServer(ip=BLA_2, user=BLA+2, ssh_key_path=BLA_2) as conn:
do_stuff...
and it's the same methods in both connection, it's just different remote server.
so i'm guessing that using "with" is not the smart way in that case (cause i duplicate the code).
i want to change the object creation way to:
conn = SshToServer(...)
do_stuff...
conn.__exit__ / conn.__del__
But i'm confusing between those 4 methods and when i should use them.
__enter__ and __exit__ methods are use to perform setup and teardown actions.
It will automatically close the connection for you. __exit__ is called when you exit the with scope.
In this case you are creating two new connections so you need to create two context managers.
I don't know about the Connection class. if it lets you configure user separately then you can do is
with SshToServer(ip=BLA_1, ssh_key_path=BLA_1) as conn:
conn.user = BLA_1
// do some stuff
conn.user = BLA_2
// do some stuff
Yes, they are only intended for usage with with.
In your sample code, __init__ is called first to create (actually only initialize) an object (as usual), then __enter__ is called to use the object as context manager (only in with). Both methods have their own uses.
Calling special methods directly is usually a bad idea. Exception: Calling in a special method the same method of the super class.
Main advantage of with is that __exit__ is called under nearly all circumstances and can't be forgotten.
In your code you could e.g. write the two with as one like
conn = SshToServer(...) # Somehow select which object to create
with conn:
do_stuff...
This is much cleaner and safer than calling special methods directly.
Suppose I have the following code:
import pandas as pd
import cx_Oracle
class Foo:
instances = []
def __init__(self, id, name):
self.id = id
self.name = name
dsn_tns = cx_Oracle.makedsn('***', '***', service_name='***')
conn = cx_Oracle.connect(user=r'***', password='***', dsn=***)
self.data = pd.read_sql(***, conn)
conn.close()
Foo.instances.append(self)
def method1(...):
...
def method2(...):
...
one = Foo(1, 'one')
Where I create a class and initialize one (or potenitally more) instances of that class.
In this case I am importing some data from an oracle server which takes a long time for the SQL query to run.
Say I run method1 on my object:
one.method1...
after running this say I decide I want to use method2 on one:
one.method2...
Now I will have to run the code again which will have to reinitialize one as an instance of the class Foo, which will force the SQL query to rerun which takes time. It will also rerun method1 on one which may also take a long time.
I am a beginner so I am wondering what is the best way around this?
Something like storing one in memory so when I rerun the code it doesn't have to initialize one again and applies the method directly to the object in memory?
Also is there a way to store potential outputs of method1 so when I run
one.method1
one.method2
It doesn't have to re-do one.method1 if it has been run previously.
Obviously this is not a problem on things that run fast but on code with many methods and many objects in a class and many classes etc. I can imagine this would become overwhelming.
I am sure a solution and best practices do exist but I am struggling to find help on this issue.
I suspect this is a general thing with all OOP and not just python.
Thank you.
If you avoid changing your class instace's data when calling one of the methods, you simple can run either of the methods, nothing will change for one. This is obvious.
If the methods you are calling in fact do make changes to some member variables of your class, you can make a copy of the instance:
import copy
...
one = Foo(1, 'one')
one_bak = copy.copy(one)
one.method1
one = one_bak
one.method2
In principle, memory is connected to your Python program (see this answer for example and this thread for possible solutions).
Easiest would be to pickle your object and unpickle it if file is available. See Python documentation for more information.
If I have a class that wraps a resource, e.g., an sqlite database connection or a file, is there a way I can use the with statement to close the resource when my object goes out of scope or is gcollected?
To clarify what I mean, I want to avoid this:
class x:
def __init__(self):
# open resource
def close(self): # or __del__, even worst
# close resource
but make it in such a way that the resource is always freed as in
with open('foo') as f:
# use resource
You need to provide __enter__ and __exit__ methods. See PEP 343.
This PEP adds a new statement "with" to the Python language to make it
possible to factor out standard uses of try/finally statements.
In this PEP, context managers provide __enter__() and __exit__()
methods that are invoked on entry to and exit from the body of the
with statement.
Use contextlib.closing:
with contextlib.closing(thing) as thing:
do_stuff_with(thing)
# Thing is closed now.
You can always put any cleanup code you need into a class's __del__ method:
class x:
def __init__(self):
self.thing = get_thing()
def __del__(self):
self.thing.close()
But you shouldn't.
This is a bad idea, for a few reasons. If you're using CPython, having custom __del__ methods means the GC can't break reference cycles. If you're using most other Python implementations, __del__ methods aren't called at predictable times.
This is why you usually put cleanup in explicit close methods. That's the best you can do within the class itself. It's always up to the user of your class to make sure the close method gets called, not the class itself.
So, there's no way you can use a with statement, or anything equivalent, inside your class. But you can make it easier for users of your class to use a with statement, by making your class into a context manager, as described in roippi's answer, or just by suggesting they use contextlib.closing in your documentation.
I have an object with an internal database connection that's active throughout its lifetime. At the end of the program's run, the connection has to be committed and closed. So far I've used an explicit close method, but this is somewhat cumbersome, especially when exceptions can happen in the calling code.
I'm considering using the __del__ method for closing, but after some reading online I have concerns. Is this a valid usage pattern? Can I be sure that the internal resources will be freed in __del__ correctly?
This discussion raised a similar question but found no satisfactory answer. I don't want to have an explicit close method, and using with isn't an option, because my object isn't used as simply as open-play-close, but is kept as a member of another, larger object, that uses it while running in a GUI.
C++ has perfectly working destructors where one can free resources safely, so I would imagine Python has something agreed-upon too. For some reason it seems not to be the case, and many in the community vow against __del__. What's the alternative, then?
Read up on the with statement. You're describing its use case.
You'll need to wrap your connection in a "Context Manager" class that handles the __enter__ and __exit__ methods used by the with statement.
See PEP 343 for more information.
Edit
"my object isn't used as simply as open-play-close, but is kept as a member of another, larger object"
class AnObjectWhichMustBeClosed( object ):
def __enter__( self ):
# acquire
def __exit__( self, type, value, traceback ):
# release
def open( self, dbConnectionInfo ):
# open the connection, updating the state for __exit__ to handle.
class ALargerObject( object ):
def __init__( self ):
pass
def injectTheObjectThatMustBeClosed( self, anObject ):
self.useThis = anObject
class MyGuiApp( self ):
def run( self ):
# build GUI objects
large = ALargeObject()
with AnObjectWhichMustBeClosed() as x:
large.injectTheObjectThatMustBeClosed( x )
mainLoop()
Some folks call this "Dependency Injection" and "Inversion of Control". Other folks call this the Strategy pattern. The "ObjectThatMustBeClosed" is a strategy, plugged into some larger object. The assembly is created at a top-level of the GUI app, since that's usually where resources like databases are acquired.
You can make a connection module, since modules keep the same object in the whole application, and register a function to close it with the atexit module
# db.py:
import sqlite3
import atexit
con = None
def get_connection():
global con
if not con:
con = sqlite3.connect('somedb.sqlite')
atexit.register(close_connection, con)
return con
def close_connection(some_con):
some_con.commit()
some_con.close()
# your_program.py
import db
con = db.get_connection()
cur = con.cursor()
cur.execute("SELECT ...")
This sugestion is based on the assumption that the connection in your application seems like a single instance (singleton) which a module global provides well.
If that's not the case, then you can use a destructor.
However destructors don't go well with garbage collectors and circular references (you must remove the circular reference yourself before the destructor is called) and if that's not the case (you need multiple connections) then you can go for a destructor. Just don't keep circular references around or you'll have to break them yourself.
Also, what you said about C++ is wrong. If you use destructors in C++ they are called either when the block that defines the object finishes (like python's with) or when you use the delete keyword (that deallocates an object created with new). Outside that you must use an explicit close() that is not the destructor. So it is just like python - python is even "better" because it has a garbage collector.