Develop Reference

One connection to DB for app, or a connection on every execution?

I'm using psycopg2 library to connection to my postgresql database.
Every time I want to execute any query, I make a make a new connection like this:
import psycopg2
def run_query(query):
with psycopg2.connect("dbname=test user=postgres") as connection:
cursor = connection.cursor()
cursor.execute(query)
cursor.close()
But I think it's faster to make one connection for whole app execution like this:
import psycopg2
connection = psycopg2.connect("dbname=test user=postgres")
def run_query(query):
cursor = connection.cursor()
cursor.execute(query)
cursor.close()
So which is better way to connect my database during all execution time on my app?
I've tried both ways and both worked, but I want to know which is better and why.

You should strongly consider using a connection pool, as other answers have suggested, this will be less costly than creating a connection every time you query, as well as deal with workloads that one connection alone couldn't deal with.
Create a file called something like mydb.py, and include the following:
import psycopg2
import psycopg2.pool
from contextlib import contextmanager
dbpool = psycopg2.pool.ThreadedConnectionPool(host=<<YourHost>>,
port=<<YourPort>>,
dbname=<<YourDB>>,
user=<<YourUser>>,
password=<<YourPassword>>,
)
#contextmanager
def db_cursor():
conn = dbpool.getconn()
try:
with conn.cursor() as cur:
yield cur
conn.commit()
"""
You can have multiple exception types here.
For example, if you wanted to specifically check for the
23503 "FOREIGN KEY VIOLATION" error type, you could do:
except psycopg2.Error as e:
conn.rollback()
if e.pgcode = '23503':
raise KeyError(e.diag.message_primary)
else
raise Exception(e.pgcode)
"""
except:
conn.rollback()
raise
finally:
dbpool.putconn(conn)
This will allow you run queries as so:
import mydb
def myfunction():
with mydb.db_cursor() as cur:
cur.execute("""Select * from blahblahblah...""")

Both ways are bad. The fist one is particularly bad, because opening a database connection is quite expensive. The second is bad, because you will end up with a single connection (which is too few) one connection per process or thread (which is usually too many).
Use a connection pool.

With psycopg2 how to avoid using the connection context manager

With psycopg2, connection and querying the database works like so
conn = psycopg2.connect('connection string')
with conn:
cur=conn.cursor()
cur.execute("SELECT * FROM pg_stat_activity") #simple query
rows = cur.fetchall()
for row in rows:
print (row)
After trial and error, I found out that with conn is absolutely necessary or you will get many unexplained locks.
My question is: is there a way to setup the connection to avoid the need to use it?

From https://www.psycopg.org/docs/usage.html,
Warning
Unlike file objects or other resources, exiting the connection’s with
block doesn’t close the connection, but only the transaction
associated to it. If you want to make sure the connection is closed
after a certain point, you should still use a try-catch block:
conn = psycopg2.connect(DSN)
try:
# connection usage
finally:
conn.close()

In psycopg, the Context manager has been implemented in such a way that the with statement will only terminate the transaction and not close the connection for you. The connection needs to be closed by you separately.
In case of an error with your transaction, you have the option to rollback and raise the error.
One way to do this is to write the connection closing logic yourself.
def with_connection(func):
"""
Function decorator for passing connections
"""
def connection(*args, **kwargs):
# Here, you may even use a connection pool
conn = psycopg.connect(DSN)
try:
rv = func(conn, *args, **kwargs)
except Exception as e:
conn.rollback()
raise e
else:
# Can decide to see if you need to commit the transaction or not
conn.commit()
finally:
conn.close()
return rv
return connection
#with_connection
def run_sql(conn, arg1):
cur = conn.cursor()
cur.execute(SQL, (arg1))

Since Version 2.5, psycopg2 should support the with statement like you expect it to behave.
Docs
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()

What to do with sqlite connection if an exception occurs before you can close the connection and datasbe is locked?

import sqlite3
location = 'data'
table_name = 'table_name'
try:
conn = sqlite3.connect(location)
c = conn.cursor()
sql = 'insert into ' + table_name + ' (id) values (%d)' % (1)
c.execute(sql) # Some exception occurs here
conn.commit() # This line is not reached, so database is locked.
except Exception as e:
pass:
So if we run the above code and an exception occurs, the database is locked. So what should we do ?
EDIT: I am aware that this is vulnerable to sql injection. I've posted this example simply so that I can quickly ask the question.

This is a great use case for with:
from contextlib import closing
conn = sqlite3.connect(location)
with closing(conn.cursor()) as c:
# Do anything that may raise an exception here
With works on context objects. These objects define behavior for cleaning up in the event of an exception. In this case, the cursor will be released and the database will be unlocked if an exception occurs.
Note that we need closing from contextlib. This is because sqlite3 doesn't make its cursors contexts (even though it should, and other db's like mysql and postgres do this for their db bindings). closing allows the with to know what to do with the cursor when you leave the with's block.
A similar pattern is the correct way to handle files in Python:
with open('my_file.txt', 'r') as f:
# Do anything you want here, even something that may
# raise an exception
# the with will automatically f.close() whenever control
# leaves the with statement (whether by natural flow or
# exception)
As #IljaEverilä mentions in the comments, the connections themselves are also context managers and you should definitely use them this way to ensure that connections are properly cleaned up:
with sqlite3.connect(location) as conn:
# Do whatever you need with conn

How can SQLAlchemy be taught to recover from a disconnect?

According to http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html#disconnect-handling-pessimistic, SQLAlchemy can be instrumented to reconnect if an entry in the connection pool is no longer valid. I create the following test case to test this:
import subprocess
from sqlalchemy import create_engine, event
from sqlalchemy import exc
from sqlalchemy.pool import Pool
#event.listens_for(Pool, "checkout")
def ping_connection(dbapi_connection, connection_record, connection_proxy):
cursor = dbapi_connection.cursor()
try:
print "pinging server"
cursor.execute("SELECT 1")
except:
print "raising disconnect error"
raise exc.DisconnectionError()
cursor.close()
engine = create_engine('postgresql://postgres#localhost/test')
connection = engine.connect()
subprocess.check_call(['psql', str(engine.url), '-c',
"select pg_terminate_backend(pid) from pg_stat_activity " +
"where pid <> pg_backend_pid() " +
"and datname='%s';" % engine.url.database],
stdout=subprocess.PIPE)
result = connection.execute("select 'OK'")
for row in result:
print "Success!", " ".join(row)
But instead of recovering I receive this exception:
sqlalchemy.exc.OperationalError: (OperationalError) terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Since "pinging server" is printed on the terminal it seems safe to conclude that the event listener is attached. How can SQLAlchemy be taught to recover from a disconnect?

It looks like the checkout method is only called when you first get a connection from the pool (eg your connection = engine.connect() line)
If you subsequently lose your connection, you will have to explicitly replace it, so you could just grab a new one, and retry your sql:
try:
result = connection.execute("select 'OK'")
except sqlalchemy.exc.OperationalError: # may need more exceptions here
connection = engine.connect() # grab a new connection
result = connection.execute("select 'OK'") # and retry
This would be a pain to do around every bit of sql, so you could wrap database queries using something like:
def db_execute(conn, query):
try:
result = conn.execute(query)
except sqlalchemy.exc.OperationalError: # may need more exceptions here (or trap all)
conn = engine.connect() # replace your connection
result = conn.execute(query) # and retry
return result
The following:
result = db_execute(connection, "select 'OK'")
Should now succeed.
Another option would be to also listen for the invalidate method, and take some action at that time to replace your connection.

How to tell if Python SQLite database connection or cursor is closed?

Let's say that you have the following code:
import sqlite3
conn = sqlite3.connect('mydb')
cur = conn.cursor()
# some database actions
cur.close()
conn.close()
# more code below
If I try to use the conn or cur objects later on, how could I tell that they are closed? I cannot find a .isclosed() method or anything like it.

You could wrap in a try, except statement:
>>> conn = sqlite3.connect('mydb')
>>> conn.close()
>>> try:
... one_row = conn.execute("SELECT * FROM my_table LIMIT 1;")
... except sqlite3.ProgrammingError as e:
... print(e)
Cannot operate on a closed database.
This relies on a shortcut specific to sqlite3.

How about making sure that the connection and cursor are never closed?
You could have a state based program that you can guarantee only calls close() at the right time.
Or wrap them in other objects that have a pass implementation of close(). Or add an _isclosed member set by close() and accessed by isclosed().