I have two programs: One that fill and updates a database and another that selects info from the database every 10 seconds.
I use Pymysql.
When I update the database I commit the data, I can see the results in the database with command lines, but the other program has the same output and doesn't get the new data!
Do I need to make a special query other than SELECT?
Do I need to close the connection and reopen it before all query?
I create the GetData class when starting the program and get_data is called every 10 seconds.
class GetData:
def __init__(self):
self.conn = pymysql.connect(host='localhost', user='root', password='', db='mydb', charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor)
def get_data(self, data):
with self.conn.cursor() as cursor:
self.sql = "SELECT id_data, somedata FROM mytable WHERE (%s = 'example');"
cursor.execute(self.sql, (data,))
return cursor.fetchall()
def close_conn(self):
self.conn.close()
The program that fills the database:
class FillDb:
def __init__(self):
self.conn = pymysql.connect(host='localhost', user='root', password='', db='mydb', charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor)
#added this line but doesen't help!
self.conn.autocommit(True)
def add_in_db(self, data):
with self.conn.cursor() as cursor:
self.sql = "INSERT INTO mytable (somedata) VALUES (%s);"
cursor.execute(self.sql, (data,))
self.conn.commit()
Why you did not see the updates:
The cause of the behavior is InnoDB's default isolation level REPEATABLE READ. With REPEATABLE READ, the first nonlocking SELECT establishes a snapshot representing the data at that point in time. All consecutive nonlocking SELECTs read from that same snapshot. Updates to the DB from other transactions are not reflected in that snapshot, thus remaining transparent.
Committing the transaction (or closing it and creating a new one) will cause a new snapshot to be created with the next query, representing the data in the DB at that point in time. This is how MySQL implements Consistent Nonlocking Reads as part of their ACID compliance strategy.
Why with self.conn works and what it does:
In PyMySQL, there's two (relevant) contextmanager implementations, one on the Cursor (more or less 'documented') and one on the Connection (can be found in the code :D).
When you used with self.conn.cursor() as cursor: it was the cursor's implementation that was in effect. Entering the context returned self (the cursor object returned from the cursor() method on self.conn); leaving the context ultimately closed that cursor. It has no effect on the transaction.
When using with self.conn as cursor it is the connection's implementation that is in effect. Entering the context returns the cursor from calling self.cursor(); leaving the context does a commit or rollback on the transaction. The cursor is closed implicitly as well.
So, the implicit call to self.commit when leaving the context of the connection's implementation 'expires' the existing snapshot in your transaction and forces the creation of a new one in the next iteration of your loop, which potentially contains your inserts, as long as their commit has completed before the creation of said new snapshot.
I have resolved the same issue with adding self.conn.commit() after
cursor.fetchall()
Related
Is there any way I can convert a weak variable to strong one in Python?
# utils.py
def connect_db():
cnx = mysql.connector.connect(user="root", database="test_db")
cursor = cnx.cursor()
return cursor
# main.py
from utils import connect_db
def main():
cursor = connect_db()
cursor.execute("some commands")
I got this error
ReferenceError: weakly-referenced object no longer exists
First, let's see why the error is happening. You open a connection and bind it to a function-local name cnx. Then you create a cursor, which has a weak reference to the connection. As soon as you return the cursor, the connection no longer has strong references, and it up for garbage collection. By the time you attempt to execute a query, the connection has been garbage collected.
As you noted, making the reference from cursor to connection a strong one would solve your immediate problem. At the same time, there is a reason that the API was designed this way. You don't want to have too many connections lingering around, all because some cursors did not get garbage collected.
Instead, the right answer is to handle the connection explicitly, instead of burying it in a function that returns a cursor. Among other things, it should be done in an enclosing with block to make the shut-down explicit in case of error. Since the existing implementation does not appear to support context management, you will have to write out the try-catch explicitly.
For example, you could return both the connection and the cursor:
def connect_db():
cnx = mysql.connector.connect(user="root", database="test_db")
cursor = cnx.cursor()
return cnx, cursor
def main():
cnx = None
try:
cnx, cursor = connect_db()
cursor.execute("some commands")
finally:
if cnx is not None:
cnx.close()
A more elegant solution might be to make your own context manager for the database instead of returning two separate objects (similar to https://stackoverflow.com/a/67645694/2988730, but more encapsulated):
class connect_db:
def __enter__(self):
self.cnx = mysql.connector.connect(user="root", database="test_db")
self.cursor = self.cnx.cursor()
return cursor
def __exit__(self, *args):
self.cursor.close()
self.cnx.close()
def main():
with connect_db() as cursor:
cursor.execute("some commands")
I have written a function for connecting to a database using pymysql. Here is my code:
def SQLreadrep(sql):
connection=pymysql.connect(host=############,
user=#######,
password=########,
db=#########)
with connection.cursor() as cursor:
cursor.execute(sql)
rows=cursor.fetchall()
connection.commit()
connection.close()
return rows
I pass the SQL into this function and return the rows. However, I am doing quick queries to the database. (Something like "SELECT sku WHERE object='2J4423K').
What is a way to avoid so many connections?
Should I be avoiding this many connections to begin with?
Could I crash a server using this many connections and queries?
Let me answer your last question first. Your function is acquiring a connection but it is closing it prior to returning. So, I see no reason why unless your were multithreading or multiprocessing you would ever be using more than one connection at a time and you should not be crashing the server.
The way to avoid the overhead of creating and closing so many connections would be to "cache" the connection. One way to do that would be to replace your function by a class:
import pymysql
class DB(object):
def __init__(self, datasource, db_user, db_password):
self.conn = pymysql.connect(db=datasource, user=db_user, password=db_password)
def __del__(self):
self.conn.close()
def query(self, sql):
with self.conn.cursor() as cursor:
cursor.execute(sql)
self.conn.commit()
return cursor.fetchall()
Then you instantiate an instance of the DB class and invoke its query method. When the DB instance is grabage collected, the connection will be automatically closed.
I am trying to create a login function. But it only works ones. Ex- When i give a wrong userid and password I got correct error massage that "Could't login" after canceling that message and giving correct userid and password then I get "pymysql.err.Error: Already closed" below are the sample code.
import pymysql
# Connect to the database
connection = pymysql.connect(host='localhost',
user='root',
password='',
db='python_code',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
class LoginModel:
def check_user(self, data):
try:
with connection.cursor() as cursor:
# Read a single record
sql = "SELECT `username` FROM `users` WHERE `username`=%s"
cursor.execute(sql, (data.username))
user = cursor.fetchone()
print(user)
if user:
if (user, data.password):
return user
else:
return False
else:
return False
finally:
connection.close()
You have a mismatch with respect to the number of times you're creating the connection (once) and the number of times you're closing the connection (once per login attempt).
One fix would be to move your:
connection = pymysql.connect(host='localhost',
user='root',
password='',
db='python_code',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
into your def check__user(). It would work because you'd create and close the connection on each invocation (as others have pointed out, the finally clause always gets executed.)
That's not a great design because getting database connections tends to be relatively expensive. So keeping the connection creating outside of the method is preferred.... which means you must remove the connection.close() within the method.
I think you're mixing up connection.close() with cursor.close(). You want to do the latter, not the former. In your example you don't have to explicitly close the cursor because that happens automatically with your with connection.cursor() as cursor: line.
Change finally to except, or remove the try block completely.
This is the culprit code:
finally:
connection.close()
Per the docs:
"A finally clause is always executed before leaving the try statement, whether an exception has occurred or not"
From: https://docs.python.org/2/tutorial/errors.html
You didn't describe alternative behavior for what you would like to see happen instead of this, but my answer addresses the crux of your question.
Had the same issue. The "Finally clause is needed for Postgres with the psycopg2 driver, if used with context manager (with clause), it close the cursor but not the connection. The same does not apply with Pymysql.
I am opening mysql connection in the main function and use that connection in multiple functions called by the main function.
Is there anything wrong with passing cursor from main function instead of passing the connection?
I.e.:
Pass in cursor from main function
def main():
conn = pymysql.connect(...)
with conn as cursor:
func1(cursor)
func2(cursor)
conn.close()
def func1(cursor):
cursor.execute('select ...')
def func2(cursor):
cursor.execute('insert ...')
Pass in connection from main function
def main():
conn = pymysql.connect(...)
func1(conn)
func2(conn)
conn.close()
def func1(conn):
with conn as cursor:
cursor.execute('select ...')
def func2(conn):
with conn as cursor:
cursor.execute('insert ...')
The answer comes from Law of Demeter: Pass cursor.
This also leads to a slightly shorter code. In this case, it's pretty trivial, but sometimes it may be a lot (e.g., passing a database name vs. passing a cursor).
I periodically query a MySQL table and check data in the same row.
I use MySQLdb to do the job, querying the same table and row every 15 seconds.
Actually, the row data changes every 3 seconds, but the cursor always return the same value.
The strange thing is: after I close the MySQL connection and reconnect, using a new cursor to execute the same select command, the new value is returned.
The code that I suspect to be wrong is begins after the comment:
config = SafeConfigParser()
config.read("../test/settings_test.conf")
settings = {}
settings["mysql_host"] = config.get("mysql","mysql_host")
settings["mysql_port"] = int(config.get("mysql","mysql_port"))
settings["mysql_user"] = config.get("mysql","mysql_user")
settings["mysql_password"] = config.get("mysql","mysql_password")
settings["mysql_charset"] = config.get("mysql","mysql_charset")
#suspected wrong code
conn = mysql_from_settings(settings)
cur = conn.cursor()
cur.execute('use database_a;')
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()[0]
print result
#during 15 second, I manually update the row and commit from mysql workbench
time.sleep(15)
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()
print result
conn.close()
The output is:
94
94
If I change the code so that it closes the connection and re-connects, it returns the latest value instead of repeating the same value:
conn = mysql_from_settings(settings)
cur = conn.cursor()
cur.execute('use database_a;')
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()[0]
print result
conn.close()
time.sleep(15)
#during that period, I manually update the row and commit from mysql workbench
conn = mysql_from_settings(settings)
cur = conn.cursor()
cur.execute('use database_a;')
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()[0]
print result
conn.close()
The output is:
94
104
Why this difference in behavior?
Here is the definition of mysql_from_settings:
def mysql_from_settings(settings):
try:
host = settings.get('mysql_host')
port = settings.get('mysql_port')
user = settings.get('mysql_user')
password = settings.get('mysql_password')
charset = settings.get('mysql_charset')
conn=MySQLdb.connect(host=host,user=user,passwd=password,port=port,\
charset=charset)
return conn
except MySQLdb.Error,e:
print "Mysql Error %d: %s" % (e.args[0], e.args[1])
This is almost certainly the result of transaction isolation. I'm going to assume, since you haven't stated otherwise, that you're using the default storage engine (InnoDB) and isolation level (REPEATABLE READ):
REPEATABLE READ
The default isolation level for InnoDB. It prevents any rows that are queried from being changed by other
transactions, thus blocking non-repeatable reads but not
phantom reads. It uses a moderately strict locking strategy so that all queries within a transaction see data from the
same snapshot, that is, the data as it was at the time the transaction
started.
For more details, see Consistent Nonlocking Reads in the MySQL docs.
In plain English, this means that when you SELECT from a table within a transaction, the values you read from the table will not change for the duration of the transaction; you'll continue to see the state of the table at the time the transaction opened, plus any changes made in the same transaction.
In your case, the changes every 3 seconds are being made in some other session and transaction. In order to "see" these changes, you need to leave the transaction that began when you issued the first SELECT and start a new transaction, which will then "see" a new snapshot of the table.
You can manage transactions explicitly with START TRANSACTION, COMMIT and ROLLBACK in SQL or by calling Connection.commit() and Connection.rollback(). An even better approach here might be to take advantage of context managers; for example:
conn = mysql_from_settings(settings)
with conn as cur:
cur.execute('use database_a;')
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()[0]
print result
#during 15 second, I manually update the row and commit from mysql workbench
time.sleep(15)
cur.execute('select pages from database_a_monitor where id=1;')
result = cur.fetchone()
print result
conn.close()
The with statement, when used with MySQLdb's Connection object, gives you back a cursor. When you leave the with block, Connection.__exit__ is called:
def __exit__(self, exc, value, tb):
if exc:
self.rollback()
else:
self.commit()
Since all you've done is read data, there's nothing to roll back or commit; when writing data, remember that leaving the block via an exception will cause your changes to be rolled back, while leaving normally will cause your changes to be committed.
Note that this didn't close the cursor, it only managed the transaction context. I go into more detail on this subject in my answer to When to close cursors using MySQLdb but the short story is, you don't generally have to worry about closing cursors when using MySQLdb.
You can also make your life a little easier by passing the database as a parameter to MySQLdb.connect instead of issuing a USE statement.
This answer to a very similar question offers two other approaches—you could change the isolation level to READ COMMITTED, or turn on autocommit.
The problem you face not related with Python but MySql setting:
After changing bellow in mysql databse you would fix that:
Login mysql as root
mysql> set global transaction isolation level read committed;
For permanent: (Even after restart mysql)
mysql> set global transaction isolation level read committed;
Query OK, 0 rows affected (0.00 sec)
mysql> show session variables like '%isolation%';
+-----------------------+----------------+
| Variable_name | Value |
+-----------------------+----------------+
| transaction_isolation | READ-COMMITTED |
+-----------------------+----------------+
1 row in set (0.01 sec)