As you may know, Python objects have a __del__ method. This method may or may not be called once your process stops. As such you must not use it to free resources.
Most people would tell you to use with-statements (or try-finally). I use with-statements all the time and I love them, I even create objects with __enter__/__exit__ all the time, but in my very specific case, it won't do.
Is there a way for an instance of an object to detect when its process ends?
import my_module as db
class Connection:
def __init__(self, config: db.config.DatabaseConfiguration):
self.connection = db.connect(config)
def __del__(self):
self.connection.disconnect()
# def __finally__(self) ? Something to trigger when root context ends ?
My bet is that it could be done with some good old Python meta-programming.
Context:
I work on a module that will be used by others. I can't go in their code and add with-statements.
Related
Suppose I have the following code:
import pandas as pd
import cx_Oracle
class Foo:
instances = []
def __init__(self, id, name):
self.id = id
self.name = name
dsn_tns = cx_Oracle.makedsn('***', '***', service_name='***')
conn = cx_Oracle.connect(user=r'***', password='***', dsn=***)
self.data = pd.read_sql(***, conn)
conn.close()
Foo.instances.append(self)
def method1(...):
...
def method2(...):
...
one = Foo(1, 'one')
Where I create a class and initialize one (or potenitally more) instances of that class.
In this case I am importing some data from an oracle server which takes a long time for the SQL query to run.
Say I run method1 on my object:
one.method1...
after running this say I decide I want to use method2 on one:
one.method2...
Now I will have to run the code again which will have to reinitialize one as an instance of the class Foo, which will force the SQL query to rerun which takes time. It will also rerun method1 on one which may also take a long time.
I am a beginner so I am wondering what is the best way around this?
Something like storing one in memory so when I rerun the code it doesn't have to initialize one again and applies the method directly to the object in memory?
Also is there a way to store potential outputs of method1 so when I run
one.method1
one.method2
It doesn't have to re-do one.method1 if it has been run previously.
Obviously this is not a problem on things that run fast but on code with many methods and many objects in a class and many classes etc. I can imagine this would become overwhelming.
I am sure a solution and best practices do exist but I am struggling to find help on this issue.
I suspect this is a general thing with all OOP and not just python.
Thank you.
If you avoid changing your class instace's data when calling one of the methods, you simple can run either of the methods, nothing will change for one. This is obvious.
If the methods you are calling in fact do make changes to some member variables of your class, you can make a copy of the instance:
import copy
...
one = Foo(1, 'one')
one_bak = copy.copy(one)
one.method1
one = one_bak
one.method2
In principle, memory is connected to your Python program (see this answer for example and this thread for possible solutions).
Easiest would be to pickle your object and unpickle it if file is available. See Python documentation for more information.
I use RaspberryPi3 with Python to Remote control GPIO of other RPIs.
I created a class to initialize connections and pin for all Pis:
class relay(pigpio.pi):
def __init__(self,ip_addr):
pigpio.pi.__init__(self)
self.GPIO=[4,5,6,12]
self.rpi=pigpio.pi(ip_addr)
for t in range(len(self.GPIO)):
self.rpi.write(self.GPIO[t],0)
def switch_state(self,i,state):
self.rpi.write(self.GPIO[i],state)
pi_1=relay('192.168.2.112') # creating first Rpi link
pi_2=relay('192.168.2.113') # creating second Rpi link
x=0
pi_1.switch_state(x,0)
how can I inherit pigpio module's attributes into relay ? in order not to create switch_state as I did, but to use read, write and more that belong to pigpio
If I'm right you want to extend a module to a class by inheritance.
If that is true, you can not perform that without hacky things but anyway, you should not do that.
Module and classes are not designed to be used in that way.
You better keep using module's functions as expected, I see no good reasons to map module's function within a class.
You could just map it by hand like:
import spam
class Foo():
def egg(self, *args, **kwargs):
return spam.egg(*args, **kwargs)
But again, I'm not sure there is a valid reason to do that - but there is plenty of valid reasons to not..
Using SQLAlchemy to connect to MySQL and I've gotten tired of writing things like this:
with closing(engine) as connection:
do_sql_stuff(connection)
This pattern is repeated throughout many areas of my code and it seems with the availability of __del__ this is not necessary. Why not just implement a class to wrap the connection creation and closing:
class MyConnectionManager(object):
def __init__(self, db_uri):
self.__db_engine = sqlalchemy.create_engine(db_uri)
self.__db_conn = self.__db_engine.connect()
def __del__(self):
self.__db_conn.close()
Is this simply two different styles/preferences, or are there more important reasons that using with closing() is a better way to go that using __del__ (or vice versa)?
There is no guarantee about when __del__ is actually called (or if it is called at all in the case of circular references). with closing(...) as ...: guarantees that the cleanup code is called whenever you exit the with clause.
I have an object with an internal database connection that's active throughout its lifetime. At the end of the program's run, the connection has to be committed and closed. So far I've used an explicit close method, but this is somewhat cumbersome, especially when exceptions can happen in the calling code.
I'm considering using the __del__ method for closing, but after some reading online I have concerns. Is this a valid usage pattern? Can I be sure that the internal resources will be freed in __del__ correctly?
This discussion raised a similar question but found no satisfactory answer. I don't want to have an explicit close method, and using with isn't an option, because my object isn't used as simply as open-play-close, but is kept as a member of another, larger object, that uses it while running in a GUI.
C++ has perfectly working destructors where one can free resources safely, so I would imagine Python has something agreed-upon too. For some reason it seems not to be the case, and many in the community vow against __del__. What's the alternative, then?
Read up on the with statement. You're describing its use case.
You'll need to wrap your connection in a "Context Manager" class that handles the __enter__ and __exit__ methods used by the with statement.
See PEP 343 for more information.
Edit
"my object isn't used as simply as open-play-close, but is kept as a member of another, larger object"
class AnObjectWhichMustBeClosed( object ):
def __enter__( self ):
# acquire
def __exit__( self, type, value, traceback ):
# release
def open( self, dbConnectionInfo ):
# open the connection, updating the state for __exit__ to handle.
class ALargerObject( object ):
def __init__( self ):
pass
def injectTheObjectThatMustBeClosed( self, anObject ):
self.useThis = anObject
class MyGuiApp( self ):
def run( self ):
# build GUI objects
large = ALargeObject()
with AnObjectWhichMustBeClosed() as x:
large.injectTheObjectThatMustBeClosed( x )
mainLoop()
Some folks call this "Dependency Injection" and "Inversion of Control". Other folks call this the Strategy pattern. The "ObjectThatMustBeClosed" is a strategy, plugged into some larger object. The assembly is created at a top-level of the GUI app, since that's usually where resources like databases are acquired.
You can make a connection module, since modules keep the same object in the whole application, and register a function to close it with the atexit module
# db.py:
import sqlite3
import atexit
con = None
def get_connection():
global con
if not con:
con = sqlite3.connect('somedb.sqlite')
atexit.register(close_connection, con)
return con
def close_connection(some_con):
some_con.commit()
some_con.close()
# your_program.py
import db
con = db.get_connection()
cur = con.cursor()
cur.execute("SELECT ...")
This sugestion is based on the assumption that the connection in your application seems like a single instance (singleton) which a module global provides well.
If that's not the case, then you can use a destructor.
However destructors don't go well with garbage collectors and circular references (you must remove the circular reference yourself before the destructor is called) and if that's not the case (you need multiple connections) then you can go for a destructor. Just don't keep circular references around or you'll have to break them yourself.
Also, what you said about C++ is wrong. If you use destructors in C++ they are called either when the block that defines the object finishes (like python's with) or when you use the delete keyword (that deallocates an object created with new). Outside that you must use an explicit close() that is not the destructor. So it is just like python - python is even "better" because it has a garbage collector.
The standard way of doing singletons in Python is
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs)
return cls._instance
However, this doesn't work on App Engine, since there are may be many servers and we would get one instance per server. So how would we do it for an app engine entity?
Something like:
class MySingleton(db.models):
def __init__(self):
all = MySingleton.all()
if all.count() > 0:
return all.fetch(1).get()
super(MySingleton, self).__init__ (*args, **kwargs)
This leads to a recusion error, since get() calls __init__.
How we're going to use it:
We just want to represent a configuration file, ie:
{ 'sitename': "My site", 'footer': "This page owned by X"}
Singletons are usually a bad idea, and I'd be interested to see what makes this an exception. Typically they're just globals in disguise, and apart from all the old problems with globals (eg. see http://c2.com/cgi/wiki?GlobalVariablesAreBad, in particular the bit at the top talking about non-locality, implicit coupling, concurrency issues, and testing and confinement), in the modern world you get additional problems caused by distributed and concurrent systems. If your app is potentially running across multiple servers, can you meaningfully have both instances of your application operate on the same singleton instance both safely and correctly?
If the object has no state of its, then the answer is yes, but you don't need a singleton, just a namespace.
But if the object does have some state, you need to worry about how the two application instances are going to keep the details synchronised. If two instances try reading and then writing to the same instance concurrently then your results are likely to be wrong. (eg. A HitCounter singleton that reads the current value, adds 1, and writes the current value, can miss hits this way - and that's about the least damaging example I can think of.)
I am largely unfamiliar with it, so perhaps Google App Engine has some transactional logic to handle all this for you, but that presumably means you'll have to add some extra stuff in to deal with rollbacks and the like.
So my basic advice would be to see if you can rewrite the algorithm or system without resorting to using a singleton.
If you aren't going to store the data in the datastore, why don't you just create a module with variables instead of a db.Model?
Name your file mysettings.py and inside it write:
sitename = "My site"
footer = "This page owned by X"
Then the python module effectively becomes a "singleton". You can even add functions, if needed. To use it, you do something like this:
import mysettings
print mysettings.sitename
That's how django deals with this with their DJANGO_SETTINGS_MODULE
Update:
It sounds like you really want to use a db.Model, but use memcached so you only retrieve one object once. But you'll have to come up with a way to flush it when you change data, or have it have a timeout so that it gets get'd occasionally. I'd probably go with the timeout version and do something like this in mysettings.py:
from google.appengine.api import memcache
class MySettings(db.Model):
# properties...
def Settings():
key = "mysettings"
obj = memcache.get(key)
if obj is None:
obj = MySettings.all().get() # assume there is only one
if obj:
memcache.add(key, zone, 360)
else:
logging.error("no MySettings found, create one!")
return obj
Or, if you don't want to use memcache, then just store the object in a module level variable and always use the Settings() function to reference it. But then you'll have to implement a way to flush it until the interpreter instance is recycled. I would normally use memcached for this sort of functionality.
__init__ cannot usefully return anything: just like in the first example, override __new__ instead!
I don't think there's a real "singleton" object you can hold in a distributed environment with multiple instances running. The closest you can come to this is using memcache.
Perhaps it's better to think less in terms of singletons and more in terms of data consistency. For this App Engine provides transactions, which allow you to trap any changes in an entity that might happen while you're working with that entity.