For Loop with input confirmation after n iterations with pymssql - python

I am updating a database table with a dictionary with this code:
cursor = conn.cursor()
for key in dRt:
x = dRt[key]
sql = 'UPDATE table SET R = %s WHERE %s = ID'
cursor.execute(sql, (x, key))
conn.commit()
conn.close()
My dictionary got several thousand entries. Is it possible to add a part to the code which asks, after writing 1000 rows into the database, if it should continue?
I tried something like this:
cursor = conn.cursor()
counter = 0
for key in dRt:
x = dRt[key]
sql = 'UPDATE table SET R = %s WHERE %s = ID'
if counter == 1000:
break
eingabe = input("Beenden? Enter drücken!\n")
cursor.execute(sql, (x, key))
conn.commit()
conn.close()
But this does not work correctly.


Conditionally initialising an input based on counter should work. After initialising an input you should also reset the counter to 0.
cursor = conn.cursor()
counter = 0
for key in dRt:
counter += 1
x = dRt[key]
sql = 'UPDATE table SET R = %s WHERE %s = ID'
if counter == 1000:
input("Beenden? Enter drücken!\n")
counter = 0
cursor.execute(sql, (x, key))
conn.commit()
conn.close()


It's okay to have an input statement in the loop, but execution will stop until something is entered. You must then check to see what was entered to determine whether you want to continue or not:
cursor = conn.cursor()
sql = 'UPDATE table SET R = %s WHERE %s = ID' # moved out of the loop
counter = 0
for key in dRt:
counter += 1
x = dRt[key]
cursor.execute(sql, (x, key)) # moved higher in the loop
if counter == 1000: # now we have done the next 1000 updates
reply = input("Beenden? Enter drücken!\n")
if reply == '':
break # anything else entered than just enter, for example "geh", and we continue
counter = 0
conn.commit()
conn.close()

Related

How can I speed up updating MySQL

How can I speed up updating data in my database? Do I need to change the loop or build the update in a different way?
try:
g = 1
for i in gate_io().values.tolist():
with connection.cursor() as cursor:
if i[1] != 0 and i[1] != '':
insert_quarry = "UPDATE gate SET symbol = %s, bidPX = %s, askPx = %s WHERE id = %s"
currency = [i[0], i[1], i[2]]
cursor.execute(insert_quarry, (currency[0], currency[1], currency[2], g))
connection.commit()
g = g + 1
else:
continue
finally:
connection.close()
Is it possible to use NumPy for this? Or are there other options?

Don't commit after every UPDATE. Instead commit after each batch of 100 or so. Most of the actual work of updating happens at commit.
Don't forget to commit the last batch.

Why wont this loop?

Newby working my way through python and sql with mariadb.
Why wont this loop? It updates the first record only. Im pretty hack at this...
cursor1.execute("select recordid, mvavgvol, quote_usd_volume_change_24h from pumped")
records = cursor1.fetchall()
for x in records:
rid = (x[0])
m = (x[1])
o = (x[2])
if (m >= o):
result = 0
else:
result = 1
upd_data=(result,rid)
sql1 = ("UPDATE pumped SET mvavgvolcheck = %s WHERE recordid = %s")
cursor2.execute(sql1,upd_data)
conn.commit()

Since you are fetching multiple rows you have to store the fetched values in an array and use cursor's executemany() method instead.
✂
data= []
for x in records:
rid = (x[0])
result= int(x[1] > x[2])
data+= [(result, rid)]
cursor.executemany(UPDATE pumped SET mvavgvolcheck = %s WHERE recordid = %s", data);
✂
When using mariadb python module (MariaDB Connector/Python) this is much more effective since it reduces network traffic: instead of sending n update commands in a loop (where n is the number of rows in table pumped) only one command will be send to the server.

conn = msql.connect(host=Host,port=Port, user=User, password=Password, database=database)
cursor1 = conn.cursor()
cursor2 = conn.cursor()
cursor1.execute("select recordid, mvavgvol, quote_usd_volume_change_24h from pumped")
records = cursor1.fetchall()
for x in records:
rid = (x[0])
m = (x[1])
o = (x[2])
if (m >= o):
result = 0
cursor2.execute("UPDATE pumped SET mvavgvolcheck = %s WHERE recordid = %s",(result, rid))
conn.commit()
else:
result = 1
cursor2.execute("UPDATE pumped SET mvavgvolcheck = %s WHERE recordid = %s",(result, rid))
conn.commit()

MySQL update or insert based on fetchall results in Python

I need to set some user meta in my wordpress through local python script. Hence I can't use the WP update_user_meta for it - it has to be done manually.
import mysql.connector as mysql
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cursor = cnx.cursor()
get_meta = ("SELECT * FROM `ff_usermeta` WHERE `user_id`= 1 AND (`meta_key`='nickname' OR `meta_key`='info' OR `meta_key`='bg' OR `meta_key`='avatar' OR `meta_key`='profile_updated')")
cursor.execute(get_meta)
meta = cursor.fetchall()
#some processing of the result
cursor.execute(q, (...))
cnx.commit()
cursor.close()
cnx.close()
Now I need to check if the result has meta with each of the keys.
If the key already exists for this user, it needs to run UPDATE for this meta.
If this user still has no meta of this key, it has to INSERT new row.
if(there's no 'nickname' in meta_key on either of 5 or less rows):
q = ("INSERT INTO `ff_usermeta` ...")
else:
q = ("UPDATE `ff_usermeta` ...")
...and 4 more times like that?.. Seems like a good place for a cycle, but I don't really like the idea to make it 5x queues, especially since there might be more fields in the future.
I was thinking along the lines of searching the fetchall result for matches in meta_key, and if found, adding required data to one array, if not - to another. And then just running one update and one insert at the end, assuming both are not empty. If I were to write it in semi-php style, it would look roughly like this:
if(in_array("nickname", meta))
for_update .= "`nickname`='"+data[0]+"', "
else:
fields .= "`nickname`, "
vals .= "'"+data[0]+"', "
if(in_array("bg", meta)):
for_update .= "`bg`='"+data[1]+"', "
else:
fields .= "`bg`, "
vals .= "'"+data[1]+"', "
if(for_update):
update = ("UPDATE `ff_usermeta` SET "+for_update+" WHERE 1")
if(fields):
insert = ("INSERT INTO `ff_usermeta`("+fields+") VALUES ("+vals+")")
But absolutely no clue how to translate it correctly to python. I had to google it up to things like "why dot not working to add one string to another". Any advice? Or perhaps there is a better way? Thanks!

It is not complete, you can not update your rows in that way.
But with this you can start to make your query
The frist select gets exactly 1 row, if the user_id exists.
The user_id doesn't seem the right choice for this, but to get what you can do it is enough.
If the query doesn't have an entry, the it will insert some data you get from anywhere
The update as the insert are in that form wrong as you have to insert 5 new orws or update max 5 rows, but that is more for you to programm
import mysql.connector as mysql
HOST = "localhost"
DATABASE = ""
USER = "root"
PASSWORD = "mypassword"
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cursor = cnx.cursor()
user_id = 1
get_meta = ("""SELECT umeta_id, user_id , MAX(IF( `meta_key`='nickname', meta_value,'')) AS 'nickname' , MAX(IF( `meta_key`='info', meta_value,'')) AS 'info' , MAX(IF( `meta_key`='bg', meta_value,'')) AS 'bg' , MAX(IF( `meta_key`='avatar', meta_value,''NULL'')) AS 'avatar' , MAX(IF (`meta_key`='profile_updated', meta_value,'')) AS 'profile_updated' FROM `ff_usermeta` WHERE `user_id`= %s GROUP BY umeta_id, user_id:""")
result = cursor.execute(get_meta,(user_id,))
if result > 0:
data = cursor.fetchone()
for_update = "";
#some processing of the result
if not data["nickname"]:
for_update += "`nickname`='"+data["nickname"]+"', "
if not data["bg"]:
for_update += "`bg`='"+data["bg"]+"', "
query = ("UPDATE `ff_usermeta` SET "+for_update+" WHERE user_id = " + user_id)
else:
#here are no data to be gathered as there is no user_id present add new user
nickname = ""
bg= ""
info = ""
avatar = ""
profile_updated = ""
fields= ""
vals = ""
fields += "`nickname`,`info`, `bg`,`avatar`,`profile_updated`"
vals += "'"+nickname+"', "+"'"+info+"', "+"'"+bg+"', "+"'"+avatar+"', "+"'"+profile_updatedfo+"'"
query = ("INSERT INTO `ff_usermeta`("+fields+") VALUES ("+vals+")")
cursor.execute(query)
cnx.commit()
cursor.close()
cnx.close()

I tried my best to adapt the suggestion above, but couldn't figure out how to make it work. Eventually I went another way, and it seems to work somehow, so I'll post the full code in case anyone would find it useful.
What it does: checks the queue in table with validation request, then parses a page (separate function) and updates user profile accodringly.
import mysql.connector as mysql
import time
from datetime import datetime
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
while True: #endless loop as a temporary scheduler
cursor = cnx.cursor()
#getting first request in the queue - 0: id, 1: url, 2: parse, 3: status, 4: user, 5: user_page, 6: req_date, 7: action
cursor.execute("SELECT * FROM `ff_qq` WHERE status = 0 LIMIT 1")
row = cursor.fetchone()
if row:
status = 1 #processed
if row[7] == "verify":
get_user = ("SELECT * FROM `ff_users` WHERE ID = %s LIMIT 1")
cursor.execute(get_user, (row[4],))
user = cursor.fetchone() #0 - ID, 5 - user_url, 8 - user_status, 9 - display_name
#separate function that returns data to insert into mysql
udata = verify(row) #0 - nickname, 1 - fb_av, 2 - fb_bg, 3 - fb_info, 4 - owner
ustat = row[1].split("/authors/")
if udata['owned'] or user[8] == ustat[1]:
update_user = ("UPDATE `ff_users` SET user_status = %s, display_name = %s, user_url = %s WHERE ID = %s LIMIT 1")
cursor.execute(update_user, (ustat[1], udata['nickname'], row[1], user[0]))
status = 2 #success
get = ("SELECT `meta_value` FROM `ff_usermeta` WHERE `user_id`= %s AND `meta_key`='ff_capabilities' LIMIT 1")
cursor.execute(get, (row[4],))
rights = cursor.fetchone()
if rights == 'a:1:{s:10:"subscriber";b:1;}':
promote = ("UPDATE `ff_usermeta` SET `meta_value` = 'a:1:{s:6:\"author\";b:1;}' "
"WHERE `user_id` = %s AND `meta_key`='ff_capabilities' LIMIT 1")
cursor.execute(promote, (row[0],))
#list of meta_key values in same order as returned data
ff = ['nickname', 'fb_av', 'fb_bg', 'fb_info']
for x in range(0,3): #goes through each one of the above list
if udata[ff[x]]: #yes this actually works, who would've thought?..
#current meta_key added directly into the string
get = ("SELECT `meta_value` FROM `ff_usermeta` WHERE `user_id`= %s AND `meta_key`='" + ff[x] + "' LIMIT 1")
cursor.execute(get, (row[4],))
meta = cursor.fetchone()
if(meta): #update if it exists, otherwise insert new row
qq = ("UPDATE `ff_usermeta` SET `meta_value` = %s "
"WHERE `user_id` = %s AND `meta_key`='" + ff[x] + "' LIMIT 1")
else:
qq = ("INSERT INTO `ff_usermeta`(`meta_value`, `meta_key`, `user_id`) "
"VALUES ('%s','" + ff[x] + "','%s'")
cursor.execute(qq, (udata[ff[x]], row[0])) #same execute works for both
else:
status = 3 #verification failed
#update queue to reflect its status
update = ("UPDATE `ff_qq` SET status = %s WHERE id = %s LIMIT 1")
cursor.execute(update, (status, row[0]))
cnx.commit()
cursor.close()
now = datetime.now()
print(now.strftime("%d.%m.%Y %H:%M:%S"))
time.sleep(180) #sleep until it's time to re-check the queue
cnx.close()

I keep getting the error 'unread result found', no matter what I change

The code is given below which attempts to insert a value into the database and create one and insert if not available according to the usernames of the users.
I keep getting an error 'Unread result found' on the line highlighted with a comment below.
Thank you in advance!
def update_userDatabase(username,text_input):
available = False
mydb = mysql.connector.connect(host="localhost",user="root",passwd="1234",database="ChatBot")
mycursor = mydb.cursor()
now = datetime.datetime.utcnow()
mycursor.execute("SHOW TABLES")
result = mycursor.fetchone()
sg.popup()
if text_input == "":
for x in result:
sg.popup(x)
if x == username:
available = True
if available == True:
sql = """INSERT INTO {tab} (Date) VALUES (%s)""".format(tab=username)
val = (now.strftime('%Y-%m-%d %H:%M:%S'))
mycursor.execute(sql, val)
else:
sql = """CREATE TABLE {tab} (Date varchar(50),Questions varchar(20))""".format(tab=username)
mycursor.execute(sql)
sql = """INSERT INTO {tab} (Date) VALUES (%s)""".format(tab=username)
val = (now.strftime('%Y-%m-%d %H:%M:%S'))
*mycursor.execute(sql, val)* #THIS IS THE LINE WITH THE ERROR
elif username=="":
sql = "INSERT INTO %s (Questions) VALUES (%s)"
val = (text_input)
mycursor.execute(sql, val)

how can I print unique code according to the loop if there is the same value?

how can I print unique code according to the loop if there is the same value?
FullChar = 'CEFLMPRTVWXYK0123456789'
total = 1000
count = 10
count = int(count)
entries = []
bcd = ""
flg = ""
rll = ""
try:
conn = psycopg2.connect(host="192.168.13.10",database="postgres", port="5432", user="postgres", password="potatona1")
cursor = conn.cursor()
def inputDatabase(data):
postgres_insert_query = """INSERT INTO unique_code(unique_code, barcode, flag, roll) VALUES (%s,%s,%s,%s)"""
cursor.executemany(postgres_insert_query, data)
conn.commit()
for i in range(5):
for x in range(total): # banyaknya code yang di print
unique_code = ''.join(random.sample(FullChar, count - 1))
unique_code = ''.join(random.sample(unique_code, len(unique_code)))
entry = (unique_code, bcd, flg, rll)
entries.append(entry)
inputDatabase(entries)
print(i)
count = cursor.rowcount
print (count, "Record inserted successfully into mobile table")
except (Exception, psycopg2.DatabaseError) as error:
print(error)
conn.rollback()
exmp :
if this code MTY9X4L2E show up again/ duplicate, the loop will stop
and i get this message
duplicate key value violates unique constraint "unique_code_pkey"

To keep track of unique values, use a set.
unique_codes = set()
...
for i in range(5):
for x in range(total): # banyaknya code yang di print
unique_code = ''.join(random.sample(FullChar, count - 1))
unique_code = ''.join(random.sample(unique_code, len(unique_code)))
if unique_code in unique_codes:
# The unique code has already been used.
# Do something?
else:
# Add the code to the set of used codes.
unique_codes.add(unique_code)
It's not very clear what those loops are doing; unique_code gets overwritten in every iteration of the inner loop.
The example code has another problem: the entries list is never cleared, so the second iteration of the outer loop will cause a duplicate key error because entries contains not only the new data but also the data from the previous iteration. entries should be cleared or reinitialised after each call to inputDatabase.
inputDatabase(entries)
entries.clear()

Categories

Resources