Iterating over resultset with comparison of column name - in python with Mysqldb - python

i want to use python to populate a database. I have to get values out of a table, calculate a score and fill this score into anoter table. I cant figure out how i can compare the column names of a row from a resultset. Because i dont need all of the columns to calculate, but need a few others as id and type_name for none calculation. So basically i want to do this:
cur = connection.cursor()
QUERY ="SELECT * FROM table"
cur.execute(QUERY)
rs = cur.fetchall()
for row in rs:
for col in row:
// if(col = "X" or col = "Y" or col = "Z"):
calc ...
// else:
use id, type_name whatever ...
how can achieve something like this? Else the code would just blow up like a bomb.


Maybe someone is searching for the answer too. With help of the previous comment, i could solve it like that
field_names = cur.description
for row in rs:
for index, col in enumerate(row):
name = field_names[index][0]
if(name == "..."):
...
elif(name == "..."):
...

Related

Unclear behavior in for-loop

I have this code in python:
query = "SELECT product_id FROM product_orders WHERE table_number = "+e
cursor.execute(query)
records = cursor.fetchall()
for record in records:
query2 = "SELECT * FROM productss WHERE id = "+str(record[0])
cursor.execute(query2)
record2 = cursor.fetchall()
sum=0
for record1 in record2:
sum = sum + record1[2]
tree.insert("", tk.END, values=record1)
tree2.insert("", tk.END, values=sum)
The problem is the sum variable does not make summing, but stores only the last value of record1. Any solution for this?

It looks like you have sum = 0 inside of a for loop. Maybe if you take it out of the loop and make it a list of sums that will fix your issue. Also, as another user has said, sum is a built-in name, you can name your variable s instead.

Is there some way to save a mysql columns value in a python var?

I'm trying to save a column value into a python variable; I query my DB for a int in a columns called id_mType and id_meter depending of a value that I get from a XML. To test it I do the next (I'm new using databases):
m = 'R1'
id_cont1 = 'LGZ0019800712'
xdb = cursor.execute("SELECT id_mType FROM mType WHERE m_symbol = %s", m)
xdb1 = cursor.execute("select id_meter from meter where nombre = %s",
id_cont1)
print (xdb)
print (xdb1)
I get every time the value "1" where the id_mType for 'R1' = 3 and id_meter= 7 for id_cont1 value. I need this to insert in another table (where there are both FK: id_meter and id_mType. Dont know if there is an easiest way)

You can store it in a list. Is that okay?
results=cursor.fetchall()
my_list=[]
for result in results:
my_list.append(result[0])
Now my_list should hold the SQL column you get returned with your query.

Use the fetchone() method to fetch a row from a cursor.
row = xdb.fetchone()
if row:
mtype = row[0]
row = xdb1.fetchone()
if row:
meter = row[0]

Fetching some values into variables

I got this code:
cursor.execute('SELECT nom FROM productes WHERE listacompra = 1')
rows = cursor.fetchall()
for row in rows:
print(row[0])
I'd like to have whatever it returns into some variables. How could I do it?
EDIT:
I think I'm not explaining myself properly. What I want is to have two variables with two values of the same column, not of the same row. For example:
There's two rows:
id 1, nom Natillas, listacompra 1
id 2, nom Chocolate, listacompra 1
I'd like to have two (or more) variables in order to have one variable with "Natillas" and other one with "Chocolate".
Thanks

Using list comprehensions:
cursor.execute('SELECT nom FROM productes WHERE listacompra = 1')
rows = cursor.fetchall()
var1 = row[0][0] # or row['nom'] if you are fetching as dict
var2 = row[1][0]
Obviously for this to work you have to be sure query will return at least two rows.
Old answer
The iterator will return a tuple representing the row specified in the query. For example, for the query SELECT id, password FROM users the variable row will contain the id value in the first position and password in the second.
For example:
for row in rows:
id = row[0]
pwd = row[1]
Or, more coincise:
for row in rows:
id, pwd = row
Unless you specify the option cursorclass=pymysql.cursors.DictCursor when defining the connection, in this case it will return a dictionary:
for row in rows:
id = row['id']
pwd = row['password']

rows = cursor.fetchall()
for row in rows:
print(row['<key of table field>'])

How do I perform an UPDATE of existing rows of a db table using a Pandas DataFrame?

I am attempting to query a subset of a MySql database table, feed the results into a Pandas DataFrame, alter some data, and then write the updated rows back to the same table. My table size is ~1MM rows, and the number of rows I will be altering will be relatively small (<50,000) so bringing back the entire table and performing a df.to_sql(tablename,engine, if_exists='replace') isn't a viable option. Is there a straightforward way to UPDATE the rows that have been altered without iterating over every row in the DataFrame?
I am aware of this project, which attempts to simulate an "upsert" workflow, but it seems it only accomplishes the task of inserting new non-duplicate rows rather than updating parts of existing rows:
GitHub Pandas-to_sql-upsert
Here is a skeleton of what I'm attempting to accomplish on a much larger scale:
import pandas as pd
from sqlalchemy import create_engine
import threading
#Get sample data
d = {'A' : [1, 2, 3, 4], 'B' : [4, 3, 2, 1]}
df = pd.DataFrame(d)
engine = create_engine(SQLALCHEMY_DATABASE_URI)
#Create a table with a unique constraint on A.
engine.execute("""DROP TABLE IF EXISTS test_upsert """)
engine.execute("""CREATE TABLE test_upsert (
A INTEGER,
B INTEGER,
PRIMARY KEY (A))
""")
#Insert data using pandas.to_sql
df.to_sql('test_upsert', engine, if_exists='append', index=False)
#Alter row where 'A' == 2
df_in_db.loc[df_in_db['A'] == 2, 'B'] = 6
Now I would like to write df_in_db back to my 'test_upsert' table with the updated data reflected.
This SO question is very similar, and one of the comments recommends using an "sqlalchemy table class" to perform the task.
Update table using sqlalchemy table class
Can anyone expand on how I would implement this for my specific case above if that is the best (only?) way to implement it?

I think the easiest way would be to:
first delete those rows that are going to be "upserted". This can be done in a loop, but it's not very efficient for bigger data sets (5K+ rows), so i'd save this slice of the DF into a temporary MySQL table:
# assuming we have already changed values in the rows and saved those changed rows in a separate DF: `x`
x = df[mask] # `mask` should help us to find changed rows...
# make sure `x` DF has a Primary Key column as index
x = x.set_index('a')
# dump a slice with changed rows to temporary MySQL table
x.to_sql('my_tmp', engine, if_exists='replace', index=True)
conn = engine.connect()
trans = conn.begin()
try:
# delete those rows that we are going to "upsert"
engine.execute('delete from test_upsert where a in (select a from my_tmp)')
trans.commit()
# insert changed rows
x.to_sql('test_upsert', engine, if_exists='append', index=True)
except:
trans.rollback()
raise
PS i didn't test this code so it might have some small bugs, but it should give you an idea...

A MySQL specific solution using Panda's to_sql "method" arg and sqlalchemy's mysql insert on_duplicate_key_update features:
def create_method(meta):
def method(table, conn, keys, data_iter):
sql_table = db.Table(table.name, meta, autoload=True)
insert_stmt = db.dialects.mysql.insert(sql_table).values([dict(zip(keys, data)) for data in data_iter])
upsert_stmt = insert_stmt.on_duplicate_key_update({x.name: x for x in insert_stmt.inserted})
conn.execute(upsert_stmt)
return method
engine = db.create_engine(...)
conn = engine.connect()
with conn.begin():
meta = db.MetaData(conn)
method = create_method(meta)
df.to_sql(table_name, conn, if_exists='append', method=method)

Here is a general function that will update each row (but all values in the row simultaneously)
def update_table_from_df(df, table, where):
'''Will take a dataframe and update each specified row in the SQL table
with the DF values -- DF columns MUST match SQL columns
WHERE statement should be triple-quoted string
Will not update any columns contained in the WHERE statement'''
update_string = f'UPDATE {table} set '
for idx, row in df.iterrows():
upstr = update_string
for col in list(df.columns):
if (col != 'datetime') & (col not in where):
if col != df.columns[-1]:
if type(row[col] == str):
upstr += f'''{col} = '{row[col]}', '''
else:
upstr += f'''{col} = {row[col]}, '''
else:
if type(row[col] == str):
upstr += f'''{col} = '{row[col]}' '''
else:
upstr += f'''{col} = {row[col]} '''
upstr += where
cursor.execute(upstr)
cursor.commit()```

I was struggling with this before and now I've found a way.
Basically create a separate data frame in which you keep data that you only have to update.
df #updating data in dataframe
s_update = "" #String of updations
# Loop through the data frame
for i in range(len(df)):
s_update += "update your_table_name set column_name = '%s' where column_name = '%s';"%(df[col_name1][i], df[col_name2][i])
Now pass s_update to cursor.execute or engine.execute (wherever you execute SQL query)
This will update your data instantly.

Increment row MySql Python

I am trying to increment row in MySQL database like this
rows = cursor.fetchall()
i = 0
for row in rows:
cursor.execute("UPDATE Table SET order = %s WHERE name = 'JAMES'", (i,))
db.commit()
i += 1
But at the end order for all of the items is 19, and the length of rows is 20. How can I have it go form 0 to 19, I though if I commit() after each loop this would be solved?
Thanks

Maybe you meant something like this (WHERE clause change for rows):
rows = cursor.fetchall()
for i, row in enumerate(rows):
cursor.execute("UPDATE Table SET order = %s WHERE name = %s", (i, row.name))
db.commit()
Otherwise, order fields or one record is updated multiple times.

Categories

Resources