I write a program that, with the help of pyodbc, connects to the base several times and performs selects.
Unfortunately, I have to reestablish the connection before calling any of my methods.
Why doesn't a single connection in each method work?
# create object (connect to DB)
conn = db.db_connect()
# Call method with my select
weak_password_list = db.Find_LoginsWithWeakPassword(conn)
# I need to connect again
conn = db.db_connect()
# Call method with my select
logins_with_expired_password = db.LoginsWithExpiredPassword(conn)
# And again...
conn = db.db_connect()
# Call method with my select
logins_with_expiring_password = db.Find_LoginsWithExpiringPassword(conn)
######################################################
def db_connect(self):
try:
conn = pyodbc.connect('Driver={SQL Server};'
'Server='+self.server_name+''
'Database='+self.database_name+';'
'Trusted_Connection='+self.trusted_connection+'')
except Exception as e:
conn = ""
self.print_error("Failed to connect to the database.", e)
return conn
############################
def Find_LoginsWithWeakPassword(self, conn):
try:
cursor = conn.cursor()
query_result = cursor.execute('''SELECT * FROM table_name''')
except Exception as e:
query_result=""
self.print_error("Select failed in Find_LoginsWithWeakPassword", e)
return query_result
If I only connect once, the second and subsequent methods with select has no effect.
Why?
When you call
weak_password_list = db.Find_LoginsWithWeakPassword(conn)
the function returns the pyodbc Cursor object returned by .execute():
<pyodbc.Cursor object at 0x012F4F60>
You are not calling .fetchall() (or similar) on it, so the connection has an open cursor with unconsumed results. If you do your next call
logins_with_expired_password = db.LoginsWithExpiredPassword(conn)
without first (implicitly) closing the existing connection by clobbering it, then .execute() will fail with
('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)')
TL;DR: Consume you result sets before calling another function, either by having the functions themselves call .fetchall() or by calling .fetchall() on the cursor objects that they return.
Related
I followed the answer here to create a connection using psycopg2. It works on the first call on the endpoint. The second try gives this error psycopg2.InterfaceError: connection already closed. Below is a snippet of my code:
from config import conn
with conn:
with conn.cursor() as cursor:
cursor.execute("""
select ...
"""
)
pos = cursor.fetchone()
cursor.execute("""
select ...'
"""
)
neg = cursor.fetchone()
conn.close()
Since you're closing the connection in the last line of your code, it cannot be used further (when calling the endpoint for the second time) without reconnecting.
You can either delete the last line (which would result in never closing connection) and move conn.close() to app shutdown event or perform psycopg2.connect each time you need it.
With psycopg2, connection and querying the database works like so
conn = psycopg2.connect('connection string')
with conn:
cur=conn.cursor()
cur.execute("SELECT * FROM pg_stat_activity") #simple query
rows = cur.fetchall()
for row in rows:
print (row)
After trial and error, I found out that with conn is absolutely necessary or you will get many unexplained locks.
My question is: is there a way to setup the connection to avoid the need to use it?
From https://www.psycopg.org/docs/usage.html,
Warning
Unlike file objects or other resources, exiting the connection’s with
block doesn’t close the connection, but only the transaction
associated to it. If you want to make sure the connection is closed
after a certain point, you should still use a try-catch block:
conn = psycopg2.connect(DSN)
try:
# connection usage
finally:
conn.close()
In psycopg, the Context manager has been implemented in such a way that the with statement will only terminate the transaction and not close the connection for you. The connection needs to be closed by you separately.
In case of an error with your transaction, you have the option to rollback and raise the error.
One way to do this is to write the connection closing logic yourself.
def with_connection(func):
"""
Function decorator for passing connections
"""
def connection(*args, **kwargs):
# Here, you may even use a connection pool
conn = psycopg.connect(DSN)
try:
rv = func(conn, *args, **kwargs)
except Exception as e:
conn.rollback()
raise e
else:
# Can decide to see if you need to commit the transaction or not
conn.commit()
finally:
conn.close()
return rv
return connection
#with_connection
def run_sql(conn, arg1):
cur = conn.cursor()
cur.execute(SQL, (arg1))
Since Version 2.5, psycopg2 should support the with statement like you expect it to behave.
Docs
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
Which one is more efficient than others in Python. My requirement is to have one connection until we close the application.
I have two classes one is to make and get connection/cursor and using which I am able to get and fetch the data in my Service. Following MVC in python :)
One DBConnection class
import pyodbc
class Connection:
def getconnection(self):
conn = pyodbc.connect('Driver={SQL Server};Server=.\SQLEXPRESS;Database=School;Trusted_Connection=yes;')
print("Connection Established")
#cursor = conn.cursor()
return conn
def getcursor(self):
conn = pyodbc.connect('Driver={SQL Server};Server=.\SQLEXPRESS;Database=School;Trusted_Connection=yes;')
print("Connection Established")
cursor = conn.cursor()
return cursor
and one Service Class
import Connection
import pyodbc
class StudentDataService:
connection = Connection.Connection().getconnection()
cursor = Connection.Connection().getcursor()
def getstudentdata(self):
print("In method getStudentdata()")
try:
row = self.connection.execute('select * from StudentGrade')
studentList = list(row)
return studentList
except pyodbc.DatabaseError as err:
print("Error Occurred while fetching Student Records", err)
return None
finally:
self.connection.close()
def getcursorstudentdata(self):
print("In method getcursorstudentdata()")
try:
row = self.cursor.execute('select * from StudentGrade')
studentList = list(row)
return studentList
except pyodbc.DatabaseError as err:
print("Error Occurred while fetching Student Records", err)
return None
finally:
self.cursor.close()
stu = StudentDataService()
print(stu.getstudentdata())
print("++++++++++++++++++++++++++++++++")
print(stu.getcursorstudentdata())
And both are giving me results
Connection Established
Connection Established
In method getStudentdata()
[(1, 2021, 2, Decimal('4.00')), (2, 2030, 2, Decimal('3.50')),... ]
++++++++++++++++++++++++++++++++
In method getcursorstudentdata()
[(1, 2021, 2, Decimal('4.00')), (2, 2030, 2, Decimal('3.50')),... ]
So my confusion is, which one to use?
In pyodbc, connection.execute is just a convenience for creating a cursor and performing cursor.execute. This is covered in the documentation at:
https://github.com/mkleehammer/pyodbc/wiki/Connection#execute
execute()
This function is not part of the Python DB API.
Creates a new Cursor object, calls its execute method, and returns the
new cursor.
num_products = cnxn.execute("SELECT COUNT(*) FROM product")
See Cursor.execute() for more details. This is a convenience method
that is not part of the DB API. Since a new Cursor is allocated by
each call, this should not be used if more than one SQL statement
needs to be executed on the connection.
If in doubt, just use Cursor.execute.
Both of these work:
sel = select([self.tbl])
rec = self.engine.execute(sel)
and
sel = select([self.tbl])
conn = self.engine.connect()
rec = conn.execute(sel)
What is the underlying difference?
According to the docs:
About connect()
The engine can be used directly to issue SQL to the database. The most generic way is first procure a connection resource, which you get via the Engine.connect() method:
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print("username:", row['username'])
connection.close()
The connection is an instance of Connection, which is a proxy object
for an actual DBAPI connection. The DBAPI connection is retrieved from
the connection pool at the point at which Connection is created.
About execute()
The above procedure can be performed in a shorthand way by using the execute() method of Engine itself:
result = engine.execute("select username from users") for row in result:
print("username:", row['username'])
Where above, the execute() method acquires a new Connection on its
own, executes the statement
with that object, and returns the ResultProxy. In this case, the
ResultProxy contains a special flag known as close_with_result, which
indicates that when its underlying DBAPI cursor is closed, the
Connection object itself is also closed, which again returns the DBAPI
connection to the connection pool, releasing transactional resources.
Let's say that you have the following code:
import sqlite3
conn = sqlite3.connect('mydb')
cur = conn.cursor()
# some database actions
cur.close()
conn.close()
# more code below
If I try to use the conn or cur objects later on, how could I tell that they are closed? I cannot find a .isclosed() method or anything like it.
You could wrap in a try, except statement:
>>> conn = sqlite3.connect('mydb')
>>> conn.close()
>>> try:
... one_row = conn.execute("SELECT * FROM my_table LIMIT 1;")
... except sqlite3.ProgrammingError as e:
... print(e)
Cannot operate on a closed database.
This relies on a shortcut specific to sqlite3.
How about making sure that the connection and cursor are never closed?
You could have a state based program that you can guarantee only calls close() at the right time.
Or wrap them in other objects that have a pass implementation of close(). Or add an _isclosed member set by close() and accessed by isclosed().