Related
I'm trying to upsert in SQL Server from python.
Basically I have scraped a website, converted it to DF and I'm already inserting it in my DB.
What I need: When there is data different from the scraped like the item price for example, then update it, and if the id does not exist, then insert.
Follows my code:
for index, row in df.iterrows():
cursor.execute("""INSERT INTO db_demo1.[dbo].[scrape]
(market, product_id, section_item, title_item, title_item_new, price_item,
qty, unit, sku, product_image, url, delivery_available,
delivery_long_distance, barcode, scrape_date) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
row.market, row.product_id, row.section_item, row.title_item, row.title_item_new,
row.price_item, row.qty, row.unit, row.sku, row.product_image, row.url,
row.delivery_available, row.delivery_long_distance, row.barcode, row.scrape_date)
cnxn.commit()
cursor.close()
The data that I have is something like this:
In this case, for example, it will check the product_id and see if the price_item or another column have changed, if it did, then it replaces the current value with the new one, and also update the scrape_date with the new date.
#Charlieface
I tried some solutions and none of them worked.
First I tried the articles that you posted, I have to change some parameters because it was raising some errors, but the final output:
'''
BEGIN TRANSACTION;
DECLARE #val as float
DECLARE #pid as int
DECLARE #pk as int
SET IDENTITY_INSERT scrape ON
select * from scrape;
UPDATE dbo.scrape WITH (UPDLOCK, SERIALIZABLE) SET [price_item] = #val, [product_id] = #pid WHERE id = #pk;
GO
IF ##ROWCOUNT = 0
BEGIN
INSERT INTO dbo.scrape([id], [product_id], [price_item]) VALUES(1, 1000, 17.5);
SELECT * FROM scrape
END
SET IDENTITY_INSERT scrape OFF;
COMMIT TRANSACTION;
'''
ERRORS>
Msg 1088, Level 16, State 11, Line 11
Cannot find the object "scrape" because it does not exist or you do not have permissions.
Msg 208, Level 16, State 1, Line 19
Invalid object name 'dbo.scrape'.
I'm the ADM btw, dont get the permission stuff.
second one:
'''
INSERT INTO db_demo1.dbo.scrape(id, product_id, price_item) VALUES(1, 'X', 'X'); -- to be updated
SELECT * FROM db_demo1.dbo.scrape;
MERGE scrape trg
USING (VALUES ('1','2','3'),
('C','D','E'),
('F','G','H'),
('I','J','K')) src(id, product_id, price_item)
ON trg.id = src.id
WHEN MATCHED THEN
UPDATE SET product_id = src.product_id, price_item = src.price_item
WHEN NOT MATCHED THEN
INSERT(id, product_id, price_item)
VALUES(src.id, src.product_id, src.price_item);
SELECT * FROM scrape;
'''
Same error: Invalid object name 'db_demo1.dbo.scrape'.
Someone please can save me?
This is what I have so far:
def save_invoice(self):
con = QSqlDatabase.addDatabase("QODBC")
con.setDatabaseName("C:/Users/Egon/Documents/Invoice/Invoice.accdb")
# Open the connection
con.open()
# Creating a query for later execution using .prepare()
insertDataQuery = QSqlQuery()
insertDataQuery.prepare(
"""
INSERT INTO Test01 (
Quantity,
ProductId,
Description,
Price,
Tax,
NetTotal,
GrossTotal
)
VALUES (?, ?, ?, ?, ?, ?, ?)
"""
)
data = getData(self.tableWidgetInvoiceItem)
for Quantity, ProductId, Description, Price, Tax, NetTotal, GrossTotal in data:
insertDataQuery.addBindValue(Quantity)
insertDataQuery.addBindValue(ProductId)
insertDataQuery.addBindValue(Description)
insertDataQuery.addBindValue(Price)
insertDataQuery.addBindValue(Tax)
insertDataQuery.addBindValue(NetTotal)
insertDataQuery.addBindValue(GrossTotal)
insertDataQuery.exec_()
print(insertDataQuery.lastError().text())
con.commit()
Fetch the data from the QTableWidget and return it as data.
def getData(table: QTableWidget) -> List[Tuple[str]]:
data = []
for row in range(table.rowCount()):
rowData = []
for col in range(table.columnCount()):
rowData.append(table.item(row, col).data(Qt.EditRole))
data.append(tuple(rowData))
return data
No error message is displayed but also no records are inserted into database. How can I solve this?
Try using QSqlDatabase.commit() instead of con.commit().
I am trying to update existing sqlite db rows with rows from my csv file.
Not delet and insert, but update existing row (keeping id).
dqlite db has 4 columns and only 4th column is different (to update).
if its not possible to update only one column i can accept updating whole row but keeping its place in db.
db before update:
cfthostname,cftshortname,cftenv,cert_time
1904h.net,1904h,tst,DD/MM/RRRR
19053.net,19053,tst,26/03/2021
2210010315.net,2210010315,prd,DD/MM/RRRR
1809m.net,1809m,tst,26/03/2021
13jw.net,13jw,acc,DD/MM/RRRR
csv to update:
cfthostname,cftshortname,cftenv,cert_time
1904h.net,1904h,tst,13/05/2023
19053.net,19053,tst,23/07/2023
13jw.net,13jw,acc,14/06/2029
update code:
import sqlite3
import csv
conn = sqlite3.connect("C:\db.sqlite3")
cursor = conn.cursor()
[...]
with open('C:\\csv\\update.csv','rt') as fin:
dr = csv.DictReader(fin)
to_db = [(i['hostname'], i['shortname'], i['env'], i['cert_time']) for i in dr]
cursor.executemany("UPDATE itpassed_host SET hostname = ?, shortname = ?, env = ?, cert_time = ?", to_db)
conn.commit()
conn.close()
tried also with () on to_db but it gives same output on db
cursor.executemany("UPDATE itpassed_host SET hostname = ?, shortname = ?, env = ?, cert_time = ?", (to_db))
db after update:
cfthostname,cftshortname,cftenv,cert_time
13jw.net,13jw,acc,14/06/2029
13jw.net,13jw,acc,14/06/2029
13jw.net,13jw,acc,14/06/2029
13jw.net,13jw,acc,14/06/2029
13jw.net,13jw,acc,14/06/2029
how to update only rows from csv to update correctly in db?
Use the WHERE condition.
import sqlite3
import csv
conn = sqlite3.connect("db.sqlite3")
cursor = conn.cursor()
with open('update.csv','rt') as fin:
dr = csv.DictReader(fin)
to_db = [(i['cert_time'], i['cfthostname'], i['cftshortname'], i['cftenv'],) for i in dr]
cursor.executemany("UPDATE itpassed_host SET cert_time = ? WHERE cfthostname = ? AND cftshortname = ? AND cftenv = ?", to_db)
conn.commit()
conn.close()
Notice the order changed in the line with to_db=.
i have a script like below :
import psutil
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
result = c.execute("SELECT * FROM clientinfo WHERE IP = ?", ("192.168.10.111",))
if (len(result.fetchall()) > 0):
for x in DISK :
c.execute("UPDATE disk SET Device = ?, 'Mount Point' = ?, 'fstyle' = ?, 'opts' = ?, 'total' = ?, 'used' = ?, 'free' = ?, 'percent' = ? WHERE Client_IP = ?", (DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7], "192.168.10.111"))
else :
for x in DISK :
c.execute("INSERT INTO disk('Client_IP', 'Device', 'Mount Point', 'fstyle', 'opts', 'total', 'used', 'free', 'percent') VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", ("192.168.10.111", DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7]))
conn.commit()
conn.close()
Script will check in db if there is any record about IP "192.168.10.111". If db already have record about IP "192.168.10.111", Script will update data from dict DISK to table disk of db.
If db don't have record about IP "192.168.10.111", Script will create record to insert **DISK"" to database.
INSERT command work well but the UPDATE command don't work like i want. After INSERT command run, in table disk i will have two record about disk C and disk D with same value of column Client_IP (192.168.10.111).
After UPDATE, two record of IP "192.168.10.111" get same value on every column ehich is very wrong. One record must be contain information about disk C and another record cotain disk D information.
How can i make the UPDATE work right ? length of dict DISK depend on how many mounted devices the computer has. So i need to use for loop to UPDATE but not static UPDATE.
Please tell me how to fix this,
Many thanks,
Francis
Your SELECT query column value is incorrect based on your UPDATE and INSERT queries later on. Instead of IP, should't it be CLIENT_IP? Also, you need to change your string query formatting. Remove the single quotes form your column names. Lastely, you can shorten your code by using a simple list comprehension and cursor.executemany:
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
if not list(c.execute('SELECT * FROM lientinfo WHERE CLIENT_IP = ?', ("192.168.10.111",))):
c.executemany("INSERT INTO disk (Client_IP, Device, Mount Point, fstyle, opts, total, used, free, percent) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", [["192.168.10.111", *i] for i in DISK.values()])
else:
c.executemany('UPDATE disk SET Device = ?, Mount Point = ?, fstyle = ?, opts = ?, total = ?, used = ?, free = ?, percent = ? WHERE Client_IP = ?', [[*i, "192.168.10.111"] for i in DISK.values()])
import csv
import sqlite3
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
# Close the database
holidayDatabase.close()
The following line
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
is causing this error
Incorrect number of bindings supplied. The current statement uses 5,
and there are 4 supplied.
currentRow is already a sequence. It's a list of all of the fields in the row.
If you were to print out currentRow, you'd get output like this (assuming this is your data set https://data.gov.au/dataset/australian-holidays-machine-readable-dataset):
['Date', 'Holiday Name', 'Information', 'More Information', 'Applicable To']
['20150101', "New Year's Day", "New Year's Day is the first day of the calendaryear and is celebrated each January 1st", '', 'NAT']
['20150126', 'Australia Day', 'Always celebrated on 26 January', 'http://www.australiaday.org.au/', 'NAT']
['20150302', 'Labour Day', 'Always on a Monday, creating a long weekend. It celebrates the eight-hour working day, a victory for workers in the mid-late 19th century.',http://www.commerce.wa.gov.au/labour-relations/public-holidays-western-australia', 'WA']
...
When you do
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
You're actually getting a list of all of the characters in the first element in the list.
Because you didn't skip the header row, you're actually getting a list of the characters in the word "Date". Which equals 4 characters and is causing the error:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current sta
tement uses 5, and there are 4 supplied.
If you were to skip the header line using next(listOfPublicHolidays, None) as in:
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
next(listOfPublicHolidays, None)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
you would get the following error message because currentEntry would be a list of the characters in "20150101", having a length of 8:
Traceback (most recent call last):
File "holidaysorig.py", line 25, in <module>
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', tuple(currentEntry)
)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 5, and there are 8 supplied.
That's why it works (for the most part) when you remove the for currentEntry in currentRow : block and rewrite it as:
import csv
import sqlite3
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
# Close the database
holidayDatabase.close()
NOTE: On my machine, I got the following errors:
(holidays) C:\Users\eyounjo\projects\holidays>python holidaysorig.py
Traceback (most recent call last):
File "holidaysorig.py", line 22, in <module>
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
So I rewrote your script as the following to deal with the above:
import csv, codecs
import sqlite3
# Encoding fix
def latin_1_encoder(unicode_csv_data):
for line in unicode_csv_data:
yield line.encode('latin-1')
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
# Encoding fix
with codecs.open(fileName, accessMode, encoding='latin-1') as publicHolidays :
listOfPublicHolidays = csv.reader(latin_1_encoder(publicHolidays))
# Skip the header row
next(listOfPublicHolidays, None)
entries = []
for currentRow in listOfPublicHolidays:
# Work-around for "You must not use 8-bit bytestrings" error
entries.append(tuple([unicode(field, 'latin-1') for field in currentRow]))
c.executemany('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', entries)
# Close the database
holidayDatabase.close()
I have corrected the error by removing the nested for loop
Replaced the following
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
With the following
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
However the cause of the error is still unclear to me.