I have been banging my head trying to get my Sqlite table to update given the code below.. looping through Pandas dataframe cells in specific columns to update specific columns in the sqlite table given a date match.
Everything works up until updating the table but the table won't update. It is always printing Fail. Any help would be greatly appreciated!
cur.execute('ALTER TABLE Car_parts'+Part_str+' ADD COLUMN Close_'+Column_ID+' number')
conn.commit()
cur.execute('ALTER TABLE Car_parts'+Part_str+' ADD COLUMN Volume_'+Column_ID+' number')
conn.commit()
y=0
while y<len(Car_Data.index):
print(y)
Update_date=Car_Data.iloc[y,0]
#Update_date=datetime.strptime(UpdateDate,date_format)
Update_close=str(CAR_Data.iloc[y,1])
Update_volume=str(Car_Data.iloc[y,2])
print(type(Update_date),type(Update_close), type(Update_volume))
try:
cur.execute('UPDATE Car_parts'+Part_str+' SET Close_'+Column_ID+' = ?, Volume_'+Column_ID+' = ? WHERE Date= ?',(Update_Close, Update_Volume, Update_date,))
conn.commit()
print("Success")
except:
print("fail")
pass
y+=1
Related
I am using sqlite3 in python 3 I want to get only the updated data from the database. what I mean by that can be explained as follows: the database already has 2 rows of data and I add 2 more rows of data. How can I read only the updated rows instead of total rows
Note: indexing may not help here because the no of rows updating will change.
def read_all():
cur = con.cursor()
cur.execute("SELECT * FROM CVT")
rows = cur.fetchall()
# print(rows[-1])
assert cur.rowcount == len(rows)
lastrowids = range(cur.lastrowid - cur.rowcount + 1, cur.lastrowid + 1)
print(lastrowids)
If you insert rows "one by one" like that
cursor.execute('INSERT INTO foo (xxxx) VALUES (xxxx)')
You then can retrieve the last inserted rows id :
last_inserted_id = cursor.lastrowid
BUT it will work ONLY if you insert a single row with execute. It will return None if you try to use it after a executemany.
If you are trying to get multiple ids of rows that were inserted at the same time see that answer that may help you.
I am new to python and I don't really understand the sql thing that well. Currently on the 6th week of team treehouse so please bare with me here if these are noob questions.
Goal
Import CSV with stock_tickers and 5 other columns of data
Convert CSV into pandas dataframe
Import dataframe into database. If there is already the unique stock_ticker for it to not add a new row, but next to check if the data in the other 5 columns is different. If it is than update it.
Right now I can do steps #1 and #2 and half of #3. With the help on here was able to get the looping thing to work. If there is a new stock_ticker row in the csv it will add it to database. If the data changes for an existing stock_ticker it won't do any updates.
for i in range(len(df)):
try:
df[df.index == i].to_sql(name='stocks', con=conn, if_exists='append', index=False)
conn.commit()
except sqlite3.IntegrityError:
pass
Current Code looks like this
import pandas as pd
from pandas import DataFrame
from pandas import ExcelWriter
import csv
import sqlite3
### IMPORT CSV ###
stock_csv_file = pd.read_csv (r'C:\Users\home\Desktop\code_projects\FIRE_Dashboard\db\alpha_vantage_active_stocks.csv')
### CHANGING INDEX NAMES FROM CSV TO TABLE NAMES ###
df = pd.DataFrame(stock_csv_file)
df = df.rename(columns = {"symbol":"stock_ticker", "name":"stock_name", "exchange":"stock_exchange", "ipoDate":"stock_ipoDate", "delistingDate":"stock_delistingDate", "status":"stock_status"})
### UPDATING DATABSE WITH SQLITE3###
conn = sqlite3.connect('stockmarket.db')
c = conn.cursor()
insert_statement = """
INSERT INTO stocks (stock_ticker,
stock_name,
stock_exchange,
stock_ipoDate,
stock_delistingDate,
stock_status
)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (stock_ticker) DO UPDATE
SET (stock_status)"""
for i in range(len(df)):
values = tuple(df.iloc[i])
c.execute(insert_statement, values)
The error I am getting
Traceback (most recent call last):
File "update_stock_tickers.py", line 71, in <module>
c.execute(insert_statement, values)
sqlite3.OperationalError: incomplete input
Found these posts that talk about it, but still getting lost >.<
How to use variables in SQL statement in Python?
python Datetime and SQLite
Loop through individual rows and update those rows SQLite3 Python
Any help is much appreciated.
Code after solution
import pandas as pd
from pandas import DataFrame
from pandas import ExcelWriter
import csv
import sqlite3
### IMPORT CSV ###
stock_csv_file = pd.read_csv (r'C:\Users\home\Desktop\code_projects\FIRE_Dashboard\db\alpha_vantage_active_stocks.csv')
### CHANGING INDEX NAMES FROM CSV TO TABLE NAMES ###
df = pd.DataFrame(stock_csv_file)
df = df.rename(columns = {"symbol":"stock_ticker", "name":"stock_name", "exchange":"stock_exchange", "ipoDate":"stock_ipoDate", "delistingDate":"stock_delistingDate", "status":"stock_status"})
### UPDATING DATABSE WITH SQLITE3###
conn = sqlite3.connect('stockmarket.db')
c = conn.cursor()
insert_statement = """
INSERT INTO stocks (stock_ticker,
stock_name,
stock_exchange,
stock_ipoDate,
stock_delistingDate,
stock_status
)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (stock_ticker) DO UPDATE
SET stock_status = EXCLUDED.stock_status"""
for i in range(len(df)):
values = tuple(df.iloc[i])
c.execute(insert_statement, values)
conn.commit()
This is the ON CONFLICT clause of your query:
ON CONFLICT (stock_ticker) DO UPDATE
SET (stock_status)
This is not valid SQLite syntax. If you want to update stock_status when another row already exists with the same stock_ticker, you can use pseudo-table EXCLUDED like so:
INSERT INTO stocks (stock_ticker,
stock_name,
stock_exchange,
stock_ipoDate,
stock_delistingDate,
stock_status
)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (stock_ticker) DO UPDATE
SET stock_status = EXCLUDED.status
I am trying to insert dates into a MySql database using pymysql and parameters. Some of the rows have a date field but for some others that particular date field is missing. The empty rows give an error of the type "pymysql.err.InternalError: (1292, "Incorrect date value:". Below is a piece of code that reproduces the error:
import pymysql
db=pymysql.connect("localhost","testuser","test1234","TESTDB")
cursor = db.cursor()
cursor.execute("SELECT VERSION()")
data = cursor.fetchone()
print ("Database version : %s " % data)
query = "DROP TABLE IF EXISTS birthdays"
cursor.execute(query)
query="CREATE TABLE birthdays(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\
name VARCHAR(20),\
birthday DATE NULL DEFAULT NULL)"
cursor.execute(query)
db.commit()
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':''}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
The problem is with Barry and its empty date. I have tried setting Barry's date to None but it is not working. If I set it up to "NULL" and remove the quotes from the date parameter (('%s',%s) instead of ('%s','%s')) it works for Barry but note for the others.
thank you very much in advance,
Gabriel Vidal
You can use below code i have just change in your array and set 'Barry':None because When using mysqldb and cursor.execute(), pass the value None:
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':None}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
For further detail here
In case someone drops by:
The following code snippet solves the problem you had. Note that you should keep your execute statements outside of the for loop to minimize the number of connections you do with the database.
birthdays = {'David':'2014-05-22','Larry':'2014-05-22', 'Barry':'NULL'}
values = ""
for k,v in birthdays.items():
values += "('%s', '%s')," % (k,v)
values = values[:-1].replace("'NULL'", "NULL") # Drop the final comma and the quotes of NULL values
query = f"INSERT INTO birthdays (name,birthday) VALUES {values}" # >= 3.6
cursor.execute(query) # Interaction with the database occurs only once
db.commit()
This generates the following statement:
INSERT INTO birthdays (name,birthday) VALUES ('David', '2014-05-22'),('Larry', '2014-05-22'),('Barry', NULL)
which is a valid sql statement
I am trying to take a dataframe and convert it into sql. I am creating the table first to set the unique indexing to allow for a rolling update with out having duplicates if there happens to be two A. Rods over time. Though I can't seem to shake this table column error and i don't know why.
import pandas as pd
import sqlite3 as sq
conn = sq.connect('test.db')
c = conn.cursor()
def set_table():
c.execute("""CREATE TABLE IF NOT EXISTS players(
"#" INTEGER,
" " REAL,
"Named" TEXT,
"B/T" TEXT,
"Ht" TEXT,
"Wt" TEXT,
"DOB" TEXT);""")
conn.commit()
def set_index_table():
c.execute(""" CREATE UNIQUE INDEX index_unique
ON players (Named, DOB)""")
conn.commit()
set_table()
set_index_table()
roster_active = pd.read_html('http://m.yankees.mlb.com/roster',index_col=0)
df = roster_active[0]
df = df.rename(columns={'Name': 'Named'})
df.to_sql('players', conn, if_exists='append')
conn.commit()
conn.close()
sqlite3.OperationalError: table players has no column named
Thank you for your time.
So I am not completely sure why this doesn't work but I found how I could get it to work. I believe it had something to do with the dataframe index. So I defined what columns I wanted to select for the dataframe and that worked.
df = df[['Named','B/T', 'Ht','Wt','DOB']]
I am attempting to create a function that I can call with a column name and have it return the contents of the column from a table (tablename is in the code).
I am past the point of errors but cannot get it to return the values in the column, instead it returns a list filled with the column name that is as long as the fields in the column.
def retrieve_data_game(column):
with sqlite3.connect("gamereview.db") as db:
cursor = db.cursor()
cursor.execute ("select ? from game",[column])
output= cursor.fetchall()
db.commit()
return output
calling this with:
if __name__ == "__main__":
a="gameID"
gameID = retrieve_data_game(a)
print(gameID)
this print statement will return:
[('gameID',), ('gameID',), ('gameID',), ('gameID',)]
any help would be greatly appreciated
kind regards