I have written Python code to update my MySQL database via a for loop, however when I run the code, it does not insert the data into the table. Here is my code:
connection = mysql.connector.connect(\*\*db) # \*\*
cursor = connection.cursor()
for i in range(len(alumniNames)):
currentName = alumniNames[i]
query = (f'INSERT INTO alumni (name, address, hometown, state, country, home_phone, mobile_phone) VALUES ("{currentName}", "{alumniInfo[currentName][2]}", "{alumniCities[i]}", "{alumniStates[i]}", "{alumniInfo[currentName][5]}", "{alumniInfo[currentName][7]}", "{alumniInfo[currentName][8]}")')
values = (currentName, alumniInfo[currentName][2], alumniCities[i], alumniStates[i],
alumniInfo[currentName][5], alumniInfo[currentName][7], alumniInfo[currentName][8])
cursor.execute(query)
print(f"Query {i + 1} Completed.")
if i % 50 == 0:
time.sleep(1)
results = cursor.fetchall()
cursor.close()
connection.close()
When I run my code, no data gets inserted into the table. Also, the print statement stops at 331 (ex. "Query 331 Completed.").
I tried to googling the issue but I can't come to a conclusion as to why this is happening.
Based on your code, you're probably looking for autocommit configuration. If you're new to python + mysql, I'd recommend checking out SQL Alchemy and Alembic; it will level up your game.
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlconnection-autocommit.html
https://www.sqlalchemy.org/
https://alembic.sqlalchemy.org
Also, just for fun you can reduce the code a bit. I use enumerate all - the - time! :-)
# instead of range+len
for i, currentName in enumerate(alumniNames):
....
Related
I am inserting JSON data into a MySQL database
I am parsing the JSON and then inserting it into a MySQL db using the python connector
Through trial, I can see the error is associated with this piece of code
for steps in result['routes'][0]['legs'][0]['steps']:
query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s')
if steps['travel_mode'] == "pub_tran":
travel_mode = steps['travel_mode']
Orig_lat = steps['var_1']['dep']['lat']
Orig_lng = steps['var_1']['dep']['lng']
Dest_lat = steps['var_1']['arr']['lat']
Dest_lng = steps['var_1']['arr']['lng']
time_stamp = leg['_sent_time_stamp']
if steps['travel_mode'] =="a_pied":
query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s')
travel_mode = steps['travel_mode']
Orig_lat = steps['var_2']['lat']
Orig_lng = steps['var_2']['lng']
Dest_lat = steps['var_2']['lat']
Dest_lng = steps['var_2']['lng']
time_stamp = leg['_sent_time_stamp']
cursor.execute(query,(travel_mode, Orig_lat, Orig_lng, Dest_lat, Dest_lng, time_stamp))
leg_no = cursor.fetchone()[0]
print(leg_no)
I have inserted higher level details and am now searching the database to associate this lower level information with its parent. The only way to find this unique value is to search via the origin and destination coordinates with the time_stamp. I believe the logic is sound and by printing the leg_no immediately after this section, I can see values which appear at first inspection to be correct
However, when added to the rest of the code, it causes subsequent sections where more data is inserted using the cursor to fail with this error -
raise errors.InternalError("Unread result found.")
mysql.connector.errors.InternalError: Unread result found.
The issue seems similar to MySQL Unread Result with Python
Is the query too complex and needs splitting or is there another issue?
If the query is indeed too complex, can anyone advise how best to split this?
EDIT As per #Gord's help, Ive tried to dump any unread results
cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng))
leg_no = cursor.fetchone()[0]
try:
cursor.fetchall()
except mysql.connector.errors.InterfaceError as ie:
if ie.msg == 'No result set to fetch from.':
pass
else:
raise
cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng, time_stamp))
But, I still get
raise errors.InternalError("Unread result found.")
mysql.connector.errors.InternalError: Unread result found.
[Finished in 3.3s with exit code 1]
scratches head
EDIT 2 - when I print the ie.msg, I get -
No result set to fetch from
All that was required was for buffered to be set to true!
cursor = cnx.cursor(buffered=True)
The reason is that without a buffered cursor, the results are "lazily" loaded, meaning that "fetchone" actually only fetches one row from the full result set of the query. When you will use the same cursor again, it will complain that you still have n-1 results (where n is the result set amount) waiting to be fetched. However, when you use a buffered cursor the connector fetches ALL rows behind the scenes and you just take one from the connector so the mysql db won't complain.
I was able to recreate your issue. MySQL Connector/Python apparently doesn't like it if you retrieve multiple rows and don't fetch them all before closing the cursor or using it to retrieve some other stuff. For example
import mysql.connector
cnxn = mysql.connector.connect(
host='127.0.0.1',
user='root',
password='whatever',
database='mydb')
crsr = cnxn.cursor()
crsr.execute("DROP TABLE IF EXISTS pytest")
crsr.execute("""
CREATE TABLE pytest (
id INT(11) NOT NULL AUTO_INCREMENT,
firstname VARCHAR(20),
PRIMARY KEY (id)
)
""")
crsr.execute("INSERT INTO pytest (firstname) VALUES ('Gord')")
crsr.execute("INSERT INTO pytest (firstname) VALUES ('Anne')")
cnxn.commit()
crsr.execute("SELECT firstname FROM pytest")
fname = crsr.fetchone()[0]
print(fname)
crsr.execute("SELECT firstname FROM pytest") # InternalError: Unread result found.
If you only expect (or care about) one row then you can put a LIMIT on your query
crsr.execute("SELECT firstname FROM pytest LIMIT 0, 1")
fname = crsr.fetchone()[0]
print(fname)
crsr.execute("SELECT firstname FROM pytest") # OK now
or you can use fetchall() to get rid of any unread results after you have finished working with the rows you retrieved.
crsr.execute("SELECT firstname FROM pytest")
fname = crsr.fetchone()[0]
print(fname)
try:
crsr.fetchall() # fetch (and discard) remaining rows
except mysql.connector.errors.InterfaceError as ie:
if ie.msg == 'No result set to fetch from.':
# no problem, we were just at the end of the result set
pass
else:
raise
crsr.execute("SELECT firstname FROM pytest") # OK now
cursor.reset() is really what you want.
fetchall() is not good because you may end up moving unnecessary data from the database to your client.
The problem is about the buffer, maybe you disconnected from the previous MySQL connection and now it cannot perform the next statement. There are two ways to give the buffer to the cursor. First, only to the particular cursor using the following command:
import mysql.connector
cnx = mysql.connector.connect()
# Only this particular cursor will buffer results
cursor = cnx.cursor(buffered=True)
Alternatively, you could enable buffer for any cursor you use:
import mysql.connector
# All cursors created from cnx2 will be buffered by default
cnx2 = mysql.connector.connect(buffered=True)
cursor = cnx.cursor()
In case you disconnected from MySQL, the latter works for you.
Enjoy coding
If you want to get only one result from a request, and want after to reuse the same connexion for other requests, limit your sql select request to 1 using "limit 1" at the end of your request.
ex "Select field from table where x=1 limit 1;"
This method is faster using "buffered=True"
Set the consume_results argument on the connect() method to True.
cnx = mysql.connector.connect(
host="localhost",
user="user",
password="password",
database="database",
consume_results=True
)
Now instead of throwing an exception, it basically does fetchall().
Unfortunately this still makes it slow, if you have a lot of unread rows.
There is also a possibility that your connection to MySQL Workbench is disconnected. Establish the connection again. This solved the problem for me.
cursor.reset()
and then create tables and load entries
Would setting the cursor within the for loop, executing it, and then closing it again in the loop help?
Like:
for steps in result['routes'][0]['legs'][0]['steps']:
cursor = cnx.cursor()
....
leg_no = cursor.fetchone()[0]
cursor.close()
print(leg_no)
I have an issue which is related to connection pool but I don't understand it.
Below is my code and this is the behavior:
Starting with empty table, I do SELECT query for non-existing value (no results)
Then I do INSERT query, it successfully inserts the value
HOWEVER, after inserting a new value, if I try to do more SELECT statements it only works 2 out of 3 times, always fails exactly every 3rd try (with pool size=3. ie with pool size=10 it will work exactly 9 out of 10 times)
finally, if i restart the script, with the initial SELECT commented out (but the value is in table before script ones) I get the inserted value and it works every time.
Why does this code seem to 'get stuck returning empty result for the connection that had no result' until restarting the script?
(note that it keep opening and closing connections from connection pool because this is taken from a web application where each connect/close is a different web request. Here i cut the whole 'web' aspect out of it)
#!/usr/bin/python
import mysql.connector
dbvars = {'host':'h','user':'u','passwd':'p','db':'d'}
# db has 1 empty table 'test' with one varchar field 'id'
con = mysql.connector.connect(pool_name="mypool", pool_size=3, pool_reset_session=False, **dbvars)
cur = con.cursor()
cur.execute("SELECT id FROM test WHERE id = '123';")
result = cur.fetchall()
cur.close()
con.close()
con = mysql.connector.connect(pool_name="mypool")
cur = con.cursor()
cur.execute("INSERT INTO test VALUES ('123');")
con.commit()
cur.close()
con.close()
for i in range(12):
con = mysql.connector.connect(pool_name="mypool")
cur = con.cursor()
cur.execute("SELECT id FROM test WHERE id = '123';")
result = cur.fetchall()
cur.close()
con.close()
print result
The output of the above is:
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
[(u'123',)]
[]
[(u'123',)]
Again, if I don't do the initial SELECT before the insert, then all of them return 123 (if it's already in db). It seems the initial SELECT 'corrupts' one of the connections of the connection pool. Further, if I do 2 SELECTs for empty results before the INSERT, then 2 of the 3 connections are 'corrupt'. Finally if I do 3 SELECTs before the insert, it still works 1 of 3 times, because it seems the INSERT 'fixes' the connection (presumably by having 'results').
Ubuntu 18.04
Python 2.7.17 (released Oct 2019)
mysql-connector-python 8.0.21 (June 2020)
MySql server 5.6.10
It seems to be a rather severe bug in the python driver for MySQL. Perhaps some configuration incompatibility but clearly a bug as no error is shown yet it returns wrong query results.
I filed the bug report with MySQL team and it's status is currently 'verified'.
https://bugs.mysql.com/bug.php?id=102053
I am writing a script in python 3.x using mysqlconnector.
What I am trying to achieve right now is to check if there is a record inside my db which may be a duplicate to the one I am analyzing right now.
I came up with such code:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
isDuplicate = cursor.execute(("SELECT destination FROM {0} WHERE destination = '{1}';")
.format(db_name, data['destination']))
print(cursor.statement)
self.commit()
print(isDuplicate is None)
Though I still get isDuplicate as None object. I tried to check via cursor.statement what statement is being passed to my db: it turned out that while in script I get None obj while passed in db that query works fine.
I also tried SELECT COUNT(1) FROM db_name which also gave me different results.
I am out of ideas: maybe you guys can help me out?
Update:
The solution that works for me was:
q = ("SELECT * FROM {0} WHERE destination = %s AND countryCode = %s AND prefix = %s")
.format(db_name)
cursor.execute(q, (data['destination'], data['country_code'], data['prefix']))
self.cnx.commit()
isDoubled = cursor.fetchone()
So at the end of the day it was all about fetching data from the cursor :)
Maybe the reason of your issue is the way you use execute() method.
Try to make some changes and see what is printed out:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
q = 'SELECT count(*) FROM {} WHERE destination = %s'.format(db_name)
duplicate_count = cursor.execute(q, (data['destination'], )).fetchall()
print(duplicate_count)
Why should I provide query parameters this way? (article is on psql, but the core principles are the same as in mysql)
update
If you are still receiving "NoneType" object has no atribute "fetchall", then the error is probably here:
cursor = self.cnx.cursor(buffered=True)
Looks like you are not creating cursor at all. I can take a look at it if you post some code about cnx creation.
I am trying to fetch data from AWS MariaDB:
cursor = self._cnx.cursor()
stmt = ('SELECT * FROM flights')
cursor.execute(stmt)
print(cursor.rowcount)
# prints 2
for z in cursor:
print(z)
# Does not iterate
row = cursor.fetchone()
# row is None
rows = cursor.fetchall()
# throws 'No result set to fetch from.'
I can verify that table contains data using MySQL Workbench. Am I missing some step?
EDIT: re 2 answers:
res = cursor.execute(stmt)
# res is None
EDIT:
I created new Python project with a single file:
import mysql.connector
try:
cnx = mysql.connector.connect(
host='foobar.rds.amazonaws.com',
user='devuser',
password='devpasswd',
database='devdb'
)
cursor = cnx.cursor()
#cursor = cnx.cursor(buffered=True)
cursor.execute('SELECT * FROM flights')
print(cursor.rowcount)
rows = cursor.fetchall()
except Exception as exc:
print(exc)
If I run this code with simple cursor, fetchall raises "No result set to fetch from". If I run with buffered cursor, I can see that _rows property of cursor contains my data, but fetchall() returns empty array.
Your issue is that cursor.execute(stmt) returns an object with results and you're not storing that.
results = cursor.execute(stmt)
print(results.fetchone()) # Prints out and pops first row
For the future googlers with the same Problem I found a workaround which may help in some cases:
I didn't find the source of the problem but a solution which worked for me.
In my case .fetchone() also returned none whatever I did on my local(on my own Computer) Database. I tried the exact same code with the Database on our companies server and somehow it worked. So I copied the complete server Database onto my local Database (by using database dumps) just to get the server settings and afterwards I also could get data from my local SQL-Server with the code which didn't work before.
I am a SQL-newbie but maybe some crazy setting on my local SQL-Server prevented me from fetching data. Maybe some more experienced SQL-user knows this setting and can explain.
Hi im having issues with a sql query it works perfect in console, but when i implement into python it seems to work perfect no errors but when i check the database it hasnt worked, yet with the console it does work the same no errors yet when i check db the data is there... exact same query i use.
Any ideas?
UPDATE ex SET fbsiteurl = stringvarible, fbsitesource = '' WHERE id = 23123;
in python:
cur = con.cursor()
sqlquery = "UPDATE ex SET fbsiteurl = '"+somevarible+"', fbsitesource = '"+somevarible+"' WHERE id = %d;" % recordid
print sqlquery
cur.execute(sqlquery)
query shows up fine in print no issues, if i copy the print out and paste it into a mysql console it works perfect everytime, just come python it acts like it works but dosnt really 0_o
connection.autocommit(), or you need to do connection.commit()
Been there :) you need to close the cursor
This little gotcha continues to this day. Just to clarify, I had to use both of the above answers, as in:
cur = self.db.cursor()
try:
cur.execute(sqlcommand)
self.db.commit()
res = cur.fetchall()
except res is not None:
print(res)
finally:
cur.close()