I have written a function for connecting to a database using pymysql. Here is my code:
def SQLreadrep(sql):
connection=pymysql.connect(host=############,
user=#######,
password=########,
db=#########)
with connection.cursor() as cursor:
cursor.execute(sql)
rows=cursor.fetchall()
connection.commit()
connection.close()
return rows
I pass the SQL into this function and return the rows. However, I am doing quick queries to the database. (Something like "SELECT sku WHERE object='2J4423K').
What is a way to avoid so many connections?
Should I be avoiding this many connections to begin with?
Could I crash a server using this many connections and queries?
Let me answer your last question first. Your function is acquiring a connection but it is closing it prior to returning. So, I see no reason why unless your were multithreading or multiprocessing you would ever be using more than one connection at a time and you should not be crashing the server.
The way to avoid the overhead of creating and closing so many connections would be to "cache" the connection. One way to do that would be to replace your function by a class:
import pymysql
class DB(object):
def __init__(self, datasource, db_user, db_password):
self.conn = pymysql.connect(db=datasource, user=db_user, password=db_password)
def __del__(self):
self.conn.close()
def query(self, sql):
with self.conn.cursor() as cursor:
cursor.execute(sql)
self.conn.commit()
return cursor.fetchall()
Then you instantiate an instance of the DB class and invoke its query method. When the DB instance is grabage collected, the connection will be automatically closed.
Related
I'm using psycopg2 library to connection to my postgresql database.
Every time I want to execute any query, I make a make a new connection like this:
import psycopg2
def run_query(query):
with psycopg2.connect("dbname=test user=postgres") as connection:
cursor = connection.cursor()
cursor.execute(query)
cursor.close()
But I think it's faster to make one connection for whole app execution like this:
import psycopg2
connection = psycopg2.connect("dbname=test user=postgres")
def run_query(query):
cursor = connection.cursor()
cursor.execute(query)
cursor.close()
So which is better way to connect my database during all execution time on my app?
I've tried both ways and both worked, but I want to know which is better and why.
You should strongly consider using a connection pool, as other answers have suggested, this will be less costly than creating a connection every time you query, as well as deal with workloads that one connection alone couldn't deal with.
Create a file called something like mydb.py, and include the following:
import psycopg2
import psycopg2.pool
from contextlib import contextmanager
dbpool = psycopg2.pool.ThreadedConnectionPool(host=<<YourHost>>,
port=<<YourPort>>,
dbname=<<YourDB>>,
user=<<YourUser>>,
password=<<YourPassword>>,
)
#contextmanager
def db_cursor():
conn = dbpool.getconn()
try:
with conn.cursor() as cur:
yield cur
conn.commit()
"""
You can have multiple exception types here.
For example, if you wanted to specifically check for the
23503 "FOREIGN KEY VIOLATION" error type, you could do:
except psycopg2.Error as e:
conn.rollback()
if e.pgcode = '23503':
raise KeyError(e.diag.message_primary)
else
raise Exception(e.pgcode)
"""
except:
conn.rollback()
raise
finally:
dbpool.putconn(conn)
This will allow you run queries as so:
import mydb
def myfunction():
with mydb.db_cursor() as cur:
cur.execute("""Select * from blahblahblah...""")
Both ways are bad. The fist one is particularly bad, because opening a database connection is quite expensive. The second is bad, because you will end up with a single connection (which is too few) one connection per process or thread (which is usually too many).
Use a connection pool.
I have two programs: One that fill and updates a database and another that selects info from the database every 10 seconds.
I use Pymysql.
When I update the database I commit the data, I can see the results in the database with command lines, but the other program has the same output and doesn't get the new data!
Do I need to make a special query other than SELECT?
Do I need to close the connection and reopen it before all query?
I create the GetData class when starting the program and get_data is called every 10 seconds.
class GetData:
def __init__(self):
self.conn = pymysql.connect(host='localhost', user='root', password='', db='mydb', charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor)
def get_data(self, data):
with self.conn.cursor() as cursor:
self.sql = "SELECT id_data, somedata FROM mytable WHERE (%s = 'example');"
cursor.execute(self.sql, (data,))
return cursor.fetchall()
def close_conn(self):
self.conn.close()
The program that fills the database:
class FillDb:
def __init__(self):
self.conn = pymysql.connect(host='localhost', user='root', password='', db='mydb', charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor)
#added this line but doesen't help!
self.conn.autocommit(True)
def add_in_db(self, data):
with self.conn.cursor() as cursor:
self.sql = "INSERT INTO mytable (somedata) VALUES (%s);"
cursor.execute(self.sql, (data,))
self.conn.commit()
Why you did not see the updates:
The cause of the behavior is InnoDB's default isolation level REPEATABLE READ. With REPEATABLE READ, the first nonlocking SELECT establishes a snapshot representing the data at that point in time. All consecutive nonlocking SELECTs read from that same snapshot. Updates to the DB from other transactions are not reflected in that snapshot, thus remaining transparent.
Committing the transaction (or closing it and creating a new one) will cause a new snapshot to be created with the next query, representing the data in the DB at that point in time. This is how MySQL implements Consistent Nonlocking Reads as part of their ACID compliance strategy.
Why with self.conn works and what it does:
In PyMySQL, there's two (relevant) contextmanager implementations, one on the Cursor (more or less 'documented') and one on the Connection (can be found in the code :D).
When you used with self.conn.cursor() as cursor: it was the cursor's implementation that was in effect. Entering the context returned self (the cursor object returned from the cursor() method on self.conn); leaving the context ultimately closed that cursor. It has no effect on the transaction.
When using with self.conn as cursor it is the connection's implementation that is in effect. Entering the context returns the cursor from calling self.cursor(); leaving the context does a commit or rollback on the transaction. The cursor is closed implicitly as well.
So, the implicit call to self.commit when leaving the context of the connection's implementation 'expires' the existing snapshot in your transaction and forces the creation of a new one in the next iteration of your loop, which potentially contains your inserts, as long as their commit has completed before the creation of said new snapshot.
I have resolved the same issue with adding self.conn.commit() after
cursor.fetchall()
My experience with classes is somewhat limited, so I apologize. I have a Database class. I create an instance of it and attempt to run SQL functions through it, which work fine and dandy, until I need to save an INSERT INTO with conn.commit(). my code is as follows:-
Class DatabaseClass():
def __init__(self):
self.open_database()
def open_database(self):
conn = pyodbc.connect(
r"Driver={SQL Server};"
r"Server=ServerName;"
r"Database=DatabaseName;"
r"Trusted_Connection=yes;"
)
self.cursor = conn.cursor()
def NewRecord()
self.cursor.execute ("INSERT INTO ....")
self.conn.commit()
db = DatabaseClass()
db.NewRecord()
But I get the error
"DatabaseClass object has no attribute 'conn'".
What do I have to do so that the NewRecord function knows what connection it's dealing with?
Thanks!
I am opening mysql connection in the main function and use that connection in multiple functions called by the main function.
Is there anything wrong with passing cursor from main function instead of passing the connection?
I.e.:
Pass in cursor from main function
def main():
conn = pymysql.connect(...)
with conn as cursor:
func1(cursor)
func2(cursor)
conn.close()
def func1(cursor):
cursor.execute('select ...')
def func2(cursor):
cursor.execute('insert ...')
Pass in connection from main function
def main():
conn = pymysql.connect(...)
func1(conn)
func2(conn)
conn.close()
def func1(conn):
with conn as cursor:
cursor.execute('select ...')
def func2(conn):
with conn as cursor:
cursor.execute('insert ...')
The answer comes from Law of Demeter: Pass cursor.
This also leads to a slightly shorter code. In this case, it's pretty trivial, but sometimes it may be a lot (e.g., passing a database name vs. passing a cursor).
I'm using a few apps running Tornado Web server which all connect to a MySql DB using mysqldb. When I spin up the server, it instantiates a DB class (below) which opens a connection to the DB. All transactions are made using this same connection - which I'm not sure is a good idea.
class RDSdb(object):
def __init__(self):
self.connect()
def connect(self):
self.connection = MySQLdb.connect(cursorclass = MySQLdb.cursors.SSDictCursor, host=self.RDS_HOST,
user=self.RDS_USER, passwd=self.RDS_PASS, db=self.RDS_DB)
def get_cursor(self):
try:
cursor = self.connection.cursor()
except (AttributeError, MySQLdb.OperationalError):
self.connect()
cursor = self.connection.cursor()
return cursor
def fetch_by_query(self, query):
cursor = self.get_cursor()
cursor.execute(query)
result = cursor.fetchall()
cursor.close()
return result
I'm pretty sure I shouldn't open/close a new connection for every transaction, but then, when should I?
I noticed something else that's a bit off, which I'm certain is related : when I need to update one of my db table's schema (ex : alter table), the whole table in question gets locked and unresponsive - until I kill my 3 apps with open connections to the DB - I realize that one of those connections was holding up this update.
Best practices when it comes to this? Ideas?
thanks.