I'm looking for some input on how to handle my connection to AWS-RDS. Should I open and close the connection each time I execute a query? Should I use a lambda function, and why?
I currently have it setup so the connection remains open and executions are handled from there. I have no connection closes or timeouts.
conn = pymysql.connect(db=dbname, host=host, port=port, user=user,
password=password)
cur = conn.cursor()
I then have query executions throughout the code like such.
cur.execute("SELECT product, amount, total " +
"FROM " + table +
" WHERE po_date BETWEEN %s AND %s",
(cur_month, next_month))
This depends on your application preferences.
Global Connection- If you create the connection at the global level, you save on the cost of opening the connection at each time you need to access the database, but you are using more memory on the database as it maintains the open connection. If the application does not close the connection on exit, the database must manually timeout this idle connection and kill it. You will need to add retry logic to the database to ensure the connection is still alive.
Connect Each Time - Added overhead of creating and closing the connection. Uses extra cpu on the client and db side to open and close the connection, but will keep the connection count lower.
As for using lambda, that completely depends on the application design. But, I would say yes, use it when you can!
If you want to use lambda to connect to a database, you will need to build a deployment package or a lambda layer to include the SQL client. Here are some links with step by step instructions to create these for python with pymysql. If needed, you can substitute the pymysql library with another SQL client using these same instructions.
https://geektopia.tech/post.php?blogpost=Create_Lambda_Package_Python
https://geektopia.tech/post.php?blogpost=Create_Lambda_Layer_Python
Related
I have an RDS database that a program I created using python and Mysql connect to, in order to keep track of usage of the program. Anytime the program is used, it adds 1 to a counter on the RDS database. Just this week the program has started throwing an error connecting to the RDS SQL database after about an hour of use. Previous to this, I could leave the software running for days without ever timing out. Closing the software and re-opening it, to re-establish the connection allows me to connect for approx another hour or so before it times out again.
I am connecting using the following parameters:
awsConn = mysql.connector.connect(host='myDatabase.randomStringofChars.us-east-1.rds.amazonaws.com', database='myDatabase', port=3306, user='username', password='password')
Did something recently change with AWS/RDS, do I just need to pass a different parameter into the connection string, or do I just need to add somewhere into my program to attempt to re-establish the connection every so often?
Thanks
I am writing a server program for many clients and i used threads.
Every client can make an action that requires writing or reading from sqlite database.
Do I need to open and close connection for every action or to open the database once for all the clients to share one connection?
Example for my code:
if command == "s":
conn = open_database() #connect to the database
cursor = conn.cursor()
cursor.execute('''SELECT s FROM users WHERE username=?''', (username,))
s= cursor.fetchone()[0]
conn.close()
if not s:
s= "Empty!"
clientsock.send(str(s))
I also used insert command to the database.
One connection has exactly one transaction, so your program is likely to blow up when multiple threads try to share the same connection without locking around all transactions.
Use one connection per thread.
(If you need high concurrency, SQLite might not be the best choice.)
I have an infinite script written in Python which connects to Postgresql and inserts there a record when the person appears in front of the camera connected to my computer.
I would like to know what is the best way to connect (and store connection) to the database, if it is necessary to connect and close every time when the person appears or if I can somehow store connection. Because when I create a connection before the infinite loop and there is no activity in front of the camera, the connection stays idle and when the script wants to insert a new row after some time, the connection is closed. When I connect every time I want to insert a new row, there is no problem, but this is slower.
Thank you for any suggestions.
A connection pool works well for this kind of thing. I have not worked with it in production (using mainly Django or SQLAlchemy), but psycopg2.pool includes a few different implementations (SimpleConnectionPool or PersistentConnectionPool) that would probably fit your need. Generally speaking, a pool not only helps with managing connections as a shared resource, but also testing and re-initializing the connection when it's needed.
from psycopg2 import pool
conn_pool = pool.PersistentConnectionPool(minconn, maxconn, **dbopts)
def work_method():
conn = conn_pool.getconn()
with conn.cursor() as stmt:
stmt.execute(sql)
conn_pool.putconn(conn)
The putconn is extremely important, so that an exception doesn't leave the pool thinking the connection is still in use. Would be good to handle it as a context manager:
import contextlib
#contextlib.contextmanager
def get_db_connection():
conn = conn_pool.getconn()
yield conn
conn_pool.putconn(conn)
def work_method():
with get_db_connection() as conn:
with conn.cursor() as stmt:
stmt.execute(sql)
Hope that helps.
I am writing a script in python to listen to the twitter streaming api which will track specific keywords and insert them in mysql database using MySQLdb. I am not sure which way to choose:
For each incoming tweet, open a db connection, insert to db, then close the connection.
Open a db connection and execute insert command for incoming tweets and not closing the connection at all.
I think the script will receive 1-10 tweets per second.
It kind of depends on how your script is supposed to be run, but it should close the connection at some point - at least once when the process dies. Assuming it's a long-running process (daemon etc), the simplest strategy would be to use a "with" block to ensure the connection is closed, ie:
with MySQLdb.connect(**kw) as db:
while some_condition():
do_stuff_with(db)
but you'll probably need something a bit more involved since MySQL tends to close idle connections by itself
I have been logging temperatures at home to a MySQL database (read 10 sensors in total every 5 minutes), and have been using Python, but I am wondering something...
Currently when I first run my program, I run the normal connect to MySQL, which is only run once.
db = MySQLdb.connect(mysql_server, mysql_username, mysql_passwd, mysql_db)
cursor = db.cursor()
Then I collect the data and publish it to the database successfully. The script then sleeps for 5 minutes, then starts again and collects and publishes the data again and so on. However, I only connect once, and I don't ever disconnect; it just keeps going in a loop. I only disconnect if I terminate the program.
Is this the best practice? That is, keeping the connection open all the time to the MySQL server, or should I disconnect after I have done a insert/commit?
The reason I ask: every now and then, I have to restart the script because maybe my MySQL server has gone offline or some other issue. Should I:
Keep doing what I am doing and just handle any MySQL database disconnections with a reconnect,
Put it in the crontab to collect data every five minutes and have no loop and no sleep, or
Something else?
MySQL servers are configured to handle a fixed limited number of connections. It's not a good practice to tie up a connection that you are not using constantly. So typically you should close the connection as soon as you are done with it, and reconnect only when you need it again. MySQLdb's connections are context mangagers, so you could use the with-statement syntax to make closing the connection automatic.
connection = MySQLdb.connect(
host=config.HOST, user=config.USER,
passwd=config.PASS, db=config.MYDB, )
with connection as cursor:
print(cursor)
# the connection is closed for you automatically
# when Python leaves the `with-suite`.
For robustness, you might want to use try..except to handle the case when (even on the first run) connect fails to make a connection.
Having said that, I would just put it in a crontab entry and dispense with sleeping.