Develop Reference

Python - update records with for loop

firstly apologies for the basic question, just starting off with Python.
I have the following code:
import sqlite3
conn = sqlite3.connect("test.sqb")
cursor = conn.cursor()
sql = "SELECT * FROM report WHERE type LIKE 'C%'"
cursor.execute(sql)
data = cursor.fetchall()
for row in data:
print (row[0])
cursor.execute("UPDATE report SET route='ABCDE'")
conn.commit()
conn.close()
Why is it updating all records and not just the filtered records from sql query, even though the print (row[0]) just shows the filtered records.
Many thanks.

What's actually happening is you are running this query for each record returned from the SELECT query.
UPDATE report SET route='ABCDE'
If you only want to update route where type starts with C add the criteria to the UPDATE query and execute it once.
import sqlite3
conn = sqlite3.connect("test.sqb")
cursor = conn.cursor()
sql = "SELECT * FROM report WHERE type LIKE 'C%'"
cursor.execute(sql)
data = cursor.fetchall()
cursor.execute("UPDATE report SET route='ABCDE' WHERE type LIKE 'C%'")
conn.commit()
conn.close()

Inserting Data to SQL Server from a Python Dataframe Quickly

I have been trying to insert data from a dataframe in Python to a table already created in SQL Server. The data frame has 90K rows and wanted the best possible way to quickly insert data in the table. I only have read,write and delete permissions for the server and I cannot create any table on the server.
Below is the code which is inserting the data but it is very slow. Please advise.
import pandas as pd
import xlsxwriter
import pyodbc
df = pd.read_excel(r"Url path\abc.xlsx")
conn = pyodbc.connect('Driver={ODBC Driver 11 for SQL Server};'
'SERVER=Server Name;'
'Database=Database Name;'
'UID=User ID;'
'PWD=Password;'
'Trusted_Connection=no;')
cursor= conn.cursor()
#Deleting existing data in SQL Table:-
cursor.execute("DELETE FROM datbase.schema.TableName")
conn.commit()
#Inserting data in SQL Table:-
for index,row in df.iterrows():
cursor.execute("INSERT INTO Table Name([A],[B],[C],) values (?,?,?)", row['A'],row['B'],row['C'])
conn.commit()
cursor.close()
conn.close()

To insert data much faster, try using sqlalchemy and df.to_sql. This requires you to create an engine using sqlalchemy, and to make things faster use the option fast_executemany=True
connect_string = urllib.parse.quote_plus(f'DRIVER={{ODBC Driver 11 for SQL Server}};Server=<Server Name>,<port>;Database=<Database name>')
engine = sqlalchemy.create_engine(f'mssql+pyodbc:///?odbc_connect={connect_string}', fast_executemany=True)
with engine.connect() as connection:
df.to_sql(<table name>, connection, index=False)

Here is the script and hope this works for you.
import pandas as pd
import pyodbc as pc
connection_string = "Driver=SQL Server;Server=localhost;Database={0};Trusted_Connection=Yes;"
cnxn = pc.connect(connection_string.format("DataBaseNameHere"), autocommit=True)
cur=cnxn.cursor()
df= pd.read_csv("your_filepath_and_filename_here.csv").fillna('')
query = 'insert into TableName({0}) values ({1})'
query = query.format(','.join(df.columns), ','.join('?' * len(df1.columns)))
cur.fast_executemany = True
cur.executemany(query, df.values.tolist())
cnxn.close()

This should do what you want...very generic example...
# Insert from dataframe to table in SQL Server
import time
import pandas as pd
import pyodbc
# create timer
start_time = time.time()
from sqlalchemy import create_engine
df = pd.read_csv("C:\\your_path\\CSV1.csv")
conn_str = (
r'DRIVER={SQL Server Native Client 11.0};'
r'SERVER=Excel-PC\SQLEXPRESS;'
r'DATABASE=NORTHWND;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str)
cursor = cnxn.cursor()
for index,row in df.iterrows():
cursor.execute('INSERT INTO dbo.Table_1([Name],[Address],[Age],[Work]) values (?,?,?,?)',
row['Name'],
row['Address'],
row['Age'],
row['Work'])
cnxn.commit()
cursor.close()
cnxn.close()
# see total time to do insert
print("%s seconds ---" % (time.time() - start_time))
Try that and post back if you have additional questions/issues/concerns.

Replace df.iterrows() with df.apply() for one thing. Remove the loop for something much more efficient.

Try to populate a temp table with 1 or none indexes then insert it into your good table all at once.
Might speed things up due to not having to update the indexes after each insert??

Not all parameters were used in the SQL statement when using python and mysql

hi I am doing the python mysql at this project, I initial the database and try to create the table record, but it seems cannot load data to the table, can anyone here can help me out with this
import mysql.connector
mydb = mysql.connector.connect( host="localhost",user="root",password="asd619248636",database="mydatabase")
mycursor = mydb.cursor()
mycursor.excute=("CREATE TABLE record (temperature FLOAT(20) , humidity FLOAT(20))")
sql = "INSERT INTO record (temperature,humidity) VALUES (%d, %d)"
val = (2.3,4.5)
mycursor.execute(sql,val)
mydb.commit()
print(mycursor.rowcount, "record inserted.")
and the error shows "Not all parameters were used in the SQL statement")
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement

Changing the following should fix your problem:
sql = "INSERT INTO record (temperature,humidity) VALUES (%s, %s)"
val = ("2.3","4.5") # You can also use (2.3, 4.5)
mycursor.execute(sql,val)
The database API takes strings as arguments, and later converts them to the appropriate datatype. Your code is throwing an error because it isn't expecting %d or %f (int or float) datatypes.
For more info on this you can look here

simply change insert method to
sql = "INSERT INTO record (temperature,humidity) VALUES (%s, %s)"
then it works fine

This works for me.
# Insert from dataframe to table in SQL Server
import time
import pandas as pd
import pyodbc
# create timer
start_time = time.time()
from sqlalchemy import create_engine
df = pd.read_csv("C:\\your_path_here\\CSV1.csv")
conn_str = (
r'DRIVER={SQL Server Native Client 11.0};'
r'SERVER=Excel-Your_Server_Name;'
r'DATABASE=NORTHWND;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str)
cursor = cnxn.cursor()
for index,row in df.iterrows():
cursor.execute('INSERT INTO dbo.Table_1([Name],[Address],[Age],[Work]) values (?,?,?,?)',
row['Name'],
row['Address'],
row['Age'],
row['Work'])
cnxn.commit()
cursor.close()
cnxn.close()

python: update mysql table

I am trying to update a mysql table with variable names. Below is the code that is not working for me:
import mysql.connector
conn= mysql.connector.connect(
host=host,
user=user,
passwd=password,
database=database
)
cur = conn.cursor()
cur.execute("update player_list set country = '%s', region = '%s',name = '%s' where id = %s "
% (country, region,name, id))
Running the "cur execute" line returns the following error:
mysql.connector.errors.InternalError: Unread result found
The ID column is an integer if it has any importance.

I don't see any code here how you've created your cursor, but looks like you need to specify buffered mode for your sql class to read.
Please, refer to official documentation and change your code to use buffer=True while creating your cursor and use it afterwards.
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorbuffered.html

Try
with conn.cursor() as cur:
sql = "update player_list set country = '%s', region = '%s',name = '%s' where id = %s" % (country, region,name, id)
cur.execute(sql)
conn.commit()
and add buffered = True into your conn like
connection = mysql.connector.connect([...], buffered = True)

How to use select now() result in another query in Python?

I want to use the select now() query result which is executed in Python in another query but I am unable to do so.
My code :
import MySQLdb
db = MySQLdb.connect(host,username,password,databasename)
cursor = db.cursor()
cursor.execute("SELECT NOW()")
dt = cursor.fetchone()
dt = str(dt) #1
cursor2 = db.cursor()
sql2 = "SELECT pid from PRODUCTS where date between DATE_SUB(" + dt + ", INTERVAL 2 HOUR) and " + dt #... query2
cursor2.execute(sql2)
How can I use the date got in #1 in the #query2. It gives me error.
I even used the DATE_FORMAT function to convert it to the same format in which the NOW() function in SQL gives the output. and then tried using it in the SQL query. But still it gives me syntax error.

You can try using %s on that respective dates
sql2 = "SELECT pid from PRODUCTS where date between DATE_SUB(%s, INTERVAL 2 HOUR) and %s"
cursor2.execute(sql2,(dt,dt))

MySQLdb will automagically transform MySQL datetimes into Python native datetime.datetime objects, and Python native datetime.datetime objects into MySQL-correct datetimes, so you don't have any transformation / formatting or whatever to do by yourself. All it takes is to correctly use the db api:
import MySQLdb
db = MySQLdb.connect(host,username,password,databasename)
cursor = db.cursor()
cursor.execute("SELECT NOW()")
dt = cursor.fetchone()[0] # fetchone returns a tuple
print dt # should print a `datetime.datetime` object
# no need for a second cursor here now you have fetched results
# from your previous query
#cursor2 = db.cursor()
# **ALWAYS** use your db connector's placeholders
sql2 = "SELECT pid from PRODUCTS where date between DATE_SUB(%s, INTERVAL 2 HOUR) and %s"
# this way your db connector will take care of proper transformation / formatting / sanitization / escaping etc
cursor.execute(sql2, (dt, dt))

This link could be useful, try it out!:
https://dev.mysql.com/doc/connector-python/en/connector-python-example-cursor-select.html
Here's what it has:
"The task is to select all employees hired in the year 1999 and print their names and hire dates to the console."
import datetime
import mysql.connector
cnx = mysql.connector.connect(user='scott', database='employees')
cursor = cnx.cursor()
query = ("SELECT first_name, last_name, hire_date FROM employees "
"WHERE hire_date BETWEEN %s AND %s")
hire_start = datetime.date(1999, 1, 1)
hire_end = datetime.date(1999, 12, 31)
cursor.execute(query, (hire_start, hire_end))
for (first_name, last_name, hire_date) in cursor:
print("{}, {} was hired on {:%d %b %Y}".format(
last_name, first_name, hire_date))
cursor.close()
cnx.close()