I have an issue which is related to connection pool but I don't understand it.
Below is my code and this is the behavior:
Starting with empty table, I do SELECT query for non-existing value (no results)
Then I do INSERT query, it successfully inserts the value
HOWEVER, after inserting a new value, if I try to do more SELECT statements it only works 2 out of 3 times, always fails exactly every 3rd try (with pool size=3. ie with pool size=10 it will work exactly 9 out of 10 times)
finally, if i restart the script, with the initial SELECT commented out (but the value is in table before script ones) I get the inserted value and it works every time.
Why does this code seem to 'get stuck returning empty result for the connection that had no result' until restarting the script?
(note that it keep opening and closing connections from connection pool because this is taken from a web application where each connect/close is a different web request. Here i cut the whole 'web' aspect out of it)
#!/usr/bin/python
import mysql.connector
dbvars = {'host':'h','user':'u','passwd':'p','db':'d'}
# db has 1 empty table 'test' with one varchar field 'id'
con = mysql.connector.connect(pool_name="mypool", pool_size=3, pool_reset_session=False, **dbvars)
cur = con.cursor()
cur.execute("SELECT id FROM test WHERE id = '123';")
result = cur.fetchall()
cur.close()
con.close()
con = mysql.connector.connect(pool_name="mypool")
cur = con.cursor()
cur.execute("INSERT INTO test VALUES ('123');")
con.commit()
cur.close()
con.close()
for i in range(12):
con = mysql.connector.connect(pool_name="mypool")
cur = con.cursor()
cur.execute("SELECT id FROM test WHERE id = '123';")
result = cur.fetchall()
cur.close()
con.close()
print result
The output of the above is:
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
Again, if I don't do the initial SELECT before the insert, then all of them return 123 (if it's already in db). It seems the initial SELECT 'corrupts' one of the connections of the connection pool. Further, if I do 2 SELECTs for empty results before the INSERT, then 2 of the 3 connections are 'corrupt'. Finally if I do 3 SELECTs before the insert, it still works 1 of 3 times, because it seems the INSERT 'fixes' the connection (presumably by having 'results').
Ubuntu 18.04
Python 2.7.17 (released Oct 2019)
mysql-connector-python 8.0.21 (June 2020)
MySql server 5.6.10
It seems to be a rather severe bug in the python driver for MySQL. Perhaps some configuration incompatibility but clearly a bug as no error is shown yet it returns wrong query results.
I filed the bug report with MySQL team and it's status is currently 'verified'.
https://bugs.mysql.com/bug.php?id=102053
Related
I have written Python code to update my MySQL database via a for loop, however when I run the code, it does not insert the data into the table. Here is my code:
connection = mysql.connector.connect(\*\*db) # \*\*
cursor = connection.cursor()
for i in range(len(alumniNames)):
currentName = alumniNames[i]
query = (f'INSERT INTO alumni (name, address, hometown, state, country, home_phone, mobile_phone) VALUES ("{currentName}", "{alumniInfo[currentName][2]}", "{alumniCities[i]}", "{alumniStates[i]}", "{alumniInfo[currentName][5]}", "{alumniInfo[currentName][7]}", "{alumniInfo[currentName][8]}")')
values = (currentName, alumniInfo[currentName][2], alumniCities[i], alumniStates[i],
alumniInfo[currentName][5], alumniInfo[currentName][7], alumniInfo[currentName][8])
cursor.execute(query)
print(f"Query {i + 1} Completed.")
if i % 50 == 0:
time.sleep(1)
results = cursor.fetchall()
cursor.close()
connection.close()
When I run my code, no data gets inserted into the table. Also, the print statement stops at 331 (ex. "Query 331 Completed.").
I tried to googling the issue but I can't come to a conclusion as to why this is happening.
Based on your code, you're probably looking for autocommit configuration. If you're new to python + mysql, I'd recommend checking out SQL Alchemy and Alembic; it will level up your game.
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlconnection-autocommit.html
https://www.sqlalchemy.org/
https://alembic.sqlalchemy.org
Also, just for fun you can reduce the code a bit. I use enumerate all - the - time! :-)
# instead of range+len
for i, currentName in enumerate(alumniNames):
....
I'm using cx_oracle to update record data in oracle from python. It just a simple update, but it takes forever to run and timeout in the end. If I run the same statement directly from Oracle, it works perfectly. Does anyone know why this happened? Thanks!
my code:
con = cx_Oracle.connect()
cur = con.cursor()
stmt = "UPDATE table SET rank = 4 WHERE id like 'SAP_1000141471' and rank = 2"
cur.execute(stmt)
con.commit()
result =cur.fetchall()
I tried inserting the values into the DB through python. However i do not get any error but i do not see it updating in DB. Please advice.
#!/usr/bin/python
import MySQLdb
val = MySQLdb.connect(host='localhost', user='root', passwd='root123',
db='expenses')
def access_db(val):
access = val.cursor()
sql = """Insert into monthly values (2,'Food',1000)"""
access.execute(sql)
val.commit()
val.close()
Output from DB after the script execution:
MariaDB[expenses]> select * from monthly;
SL_no Type Amount
1 Fuel 500
I do not find the second entry in Db.
I dont think you are calling the access_db() function anywhere
I am trying to fetch data from AWS MariaDB:
cursor = self._cnx.cursor()
stmt = ('SELECT * FROM flights')
cursor.execute(stmt)
print(cursor.rowcount)
# prints 2
for z in cursor:
print(z)
# Does not iterate
row = cursor.fetchone()
# row is None
rows = cursor.fetchall()
# throws 'No result set to fetch from.'
I can verify that table contains data using MySQL Workbench. Am I missing some step?
EDIT: re 2 answers:
res = cursor.execute(stmt)
# res is None
EDIT:
I created new Python project with a single file:
import mysql.connector
try:
cnx = mysql.connector.connect(
host='foobar.rds.amazonaws.com',
user='devuser',
password='devpasswd',
database='devdb'
)
cursor = cnx.cursor()
#cursor = cnx.cursor(buffered=True)
cursor.execute('SELECT * FROM flights')
print(cursor.rowcount)
rows = cursor.fetchall()
except Exception as exc:
print(exc)
If I run this code with simple cursor, fetchall raises "No result set to fetch from". If I run with buffered cursor, I can see that _rows property of cursor contains my data, but fetchall() returns empty array.
Your issue is that cursor.execute(stmt) returns an object with results and you're not storing that.
results = cursor.execute(stmt)
print(results.fetchone()) # Prints out and pops first row
For the future googlers with the same Problem I found a workaround which may help in some cases:
I didn't find the source of the problem but a solution which worked for me.
In my case .fetchone() also returned none whatever I did on my local(on my own Computer) Database. I tried the exact same code with the Database on our companies server and somehow it worked. So I copied the complete server Database onto my local Database (by using database dumps) just to get the server settings and afterwards I also could get data from my local SQL-Server with the code which didn't work before.
I am a SQL-newbie but maybe some crazy setting on my local SQL-Server prevented me from fetching data. Maybe some more experienced SQL-user knows this setting and can explain.
I am currently connecting to a Sybase 15.7 server using sybpydb. It seems to connect fine:
import sys
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib')
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/lib')
import sybpydb
conn = sybpydb.connect(user='usr', password='pass', servername='serv')
is working fine. Changing any of my connection details results in a connection error.
I then select a database:
curr = conn.cursor()
curr.execute('use db_1')
however, now when I try to run queries, it always returns None
print curr.execute('select * from table_1')
I have tried running the use and select queries in the same execute, I have tried including go commands after each, I have tried using curr.connection.commit() after each, all with no success. I have confirmed, using dbartisan and isql, that the same queries I am using return entries.
Why am I not getting results from my queries in python?
EDIT:
Just some additional info. In order to get the sybpydb import to work, I had to change two environment variables. I added the lib paths (the same ones that I added to sys.path) to $LD_LIBRARY_PATH, i.e.:
setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib:/dba/sybase/ase/15.7/OCS-15_0/lib
and I had to change the SYBASE path from 12.5 to 15.7. All this was done in csh.
If I print conn.error(), after every curr.execute(), I get:
("Server message: number(5701) severity(10) state(2) line(0)\n\tChanged database context to 'master'.\n\n", 5701)
I completely understand where you might be confused by the documentation. Its doesn't seem to be on par with other db extensions (e.g. psycopg2).
When connecting with most standard db extensions you can specify a database. Then, when you want to get the data back from a SELECT query, you either use fetch (an ok way to do it) or the iterator (the more pythonic way to do it).
import sybpydb as sybase
conn = sybase.connect(user='usr', password='pass', servername='serv')
cur = conn.cursor()
cur.execute("use db_1")
cur.execute("SELECT * FROM table_1")
print "Query Returned %d row(s)" % cur.rowcount
for row in cur:
print row
# Alternate less-pythonic way to read query results
# for row in cur.fetchall():
# print row
Give that a try and let us know if it works.
Python 3.x working solution:
import sybpydb
try:
conn = sybpydb.connect(dsn="Servername=serv;Username=usr;Password=pass")
cur = conn.cursor()
cur.execute('select * from db_1..table_1')
# table header
header = tuple(col[0] for col in cur.description)
print('\t'.join(header))
print('-' * 60)
res = cur.fetchall()
for row in res:
line = '\t'.join(str(col) for col in row)
print(line)
cur.close()
conn.close()
except sybpydb.Error:
for err in cur.connection.messages:
print(f'Error {err[0]}, Value {err[1]}')