I'm trying to upsert in SQL Server from python.
Basically I have scraped a website, converted it to DF and I'm already inserting it in my DB.
What I need: When there is data different from the scraped like the item price for example, then update it, and if the id does not exist, then insert.
Follows my code:
for index, row in df.iterrows():
cursor.execute("""INSERT INTO db_demo1.[dbo].[scrape]
(market, product_id, section_item, title_item, title_item_new, price_item,
qty, unit, sku, product_image, url, delivery_available,
delivery_long_distance, barcode, scrape_date) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
row.market, row.product_id, row.section_item, row.title_item, row.title_item_new,
row.price_item, row.qty, row.unit, row.sku, row.product_image, row.url,
row.delivery_available, row.delivery_long_distance, row.barcode, row.scrape_date)
cnxn.commit()
cursor.close()
The data that I have is something like this:
In this case, for example, it will check the product_id and see if the price_item or another column have changed, if it did, then it replaces the current value with the new one, and also update the scrape_date with the new date.
#Charlieface
I tried some solutions and none of them worked.
First I tried the articles that you posted, I have to change some parameters because it was raising some errors, but the final output:
'''
BEGIN TRANSACTION;
DECLARE #val as float
DECLARE #pid as int
DECLARE #pk as int
SET IDENTITY_INSERT scrape ON
select * from scrape;
UPDATE dbo.scrape WITH (UPDLOCK, SERIALIZABLE) SET [price_item] = #val, [product_id] = #pid WHERE id = #pk;
GO
IF ##ROWCOUNT = 0
BEGIN
INSERT INTO dbo.scrape([id], [product_id], [price_item]) VALUES(1, 1000, 17.5);
SELECT * FROM scrape
END
SET IDENTITY_INSERT scrape OFF;
COMMIT TRANSACTION;
'''
ERRORS>
Msg 1088, Level 16, State 11, Line 11
Cannot find the object "scrape" because it does not exist or you do not have permissions.
Msg 208, Level 16, State 1, Line 19
Invalid object name 'dbo.scrape'.
I'm the ADM btw, dont get the permission stuff.
second one:
'''
INSERT INTO db_demo1.dbo.scrape(id, product_id, price_item) VALUES(1, 'X', 'X'); -- to be updated
SELECT * FROM db_demo1.dbo.scrape;
MERGE scrape trg
USING (VALUES ('1','2','3'),
('C','D','E'),
('F','G','H'),
('I','J','K')) src(id, product_id, price_item)
ON trg.id = src.id
WHEN MATCHED THEN
UPDATE SET product_id = src.product_id, price_item = src.price_item
WHEN NOT MATCHED THEN
INSERT(id, product_id, price_item)
VALUES(src.id, src.product_id, src.price_item);
SELECT * FROM scrape;
'''
Same error: Invalid object name 'db_demo1.dbo.scrape'.
Someone please can save me?
Related
I have a table "Users" with the column "g_score". The other column I am storing are "username". I am trying to send an update to g_score via the username I get. I send the request and the value does not update. g_score is stored as an INT. I am looking to increment the value by + 1 each time.
The g_score value is default = 0
The value is not being updated by the following code
I'm going to leave some snippets here-
Creating the table -
cursor.execute("CREATE TABLE IF NOT EXISTS Users(username TEXT,hash TEXT,salt TEXT,g_score INT)")
If the user does not exist- we do
cursor.execute("INSERT INTO Users VALUES(?, ?, ?, ?)", (username, hashed_password, salt, 0))
This following code does not update the g_score-
db = lite.connect('log.db', check_same_thread=False)
cursor = db.cursor()
sql = ("UPDATE Users SET g_score = g_score + 1 WHERE username = ?")
cursor.execute(sql, [g_winner.get_name()])
This is what I have so far:
def save_invoice(self):
con = QSqlDatabase.addDatabase("QODBC")
con.setDatabaseName("C:/Users/Egon/Documents/Invoice/Invoice.accdb")
# Open the connection
con.open()
# Creating a query for later execution using .prepare()
insertDataQuery = QSqlQuery()
insertDataQuery.prepare(
"""
INSERT INTO Test01 (
Quantity,
ProductId,
Description,
Price,
Tax,
NetTotal,
GrossTotal
)
VALUES (?, ?, ?, ?, ?, ?, ?)
"""
)
data = getData(self.tableWidgetInvoiceItem)
for Quantity, ProductId, Description, Price, Tax, NetTotal, GrossTotal in data:
insertDataQuery.addBindValue(Quantity)
insertDataQuery.addBindValue(ProductId)
insertDataQuery.addBindValue(Description)
insertDataQuery.addBindValue(Price)
insertDataQuery.addBindValue(Tax)
insertDataQuery.addBindValue(NetTotal)
insertDataQuery.addBindValue(GrossTotal)
insertDataQuery.exec_()
print(insertDataQuery.lastError().text())
con.commit()
Fetch the data from the QTableWidget and return it as data.
def getData(table: QTableWidget) -> List[Tuple[str]]:
data = []
for row in range(table.rowCount()):
rowData = []
for col in range(table.columnCount()):
rowData.append(table.item(row, col).data(Qt.EditRole))
data.append(tuple(rowData))
return data
No error message is displayed but also no records are inserted into database. How can I solve this?
Try using QSqlDatabase.commit() instead of con.commit().
I want to insert variable values into db2 table using Python code
id = input("table id: ")
tabname = input("Enter Table name: ")
descr = input("Enter description : ")
inser_sql = "INSERT INTO schema.table VALUES (?, ?, ?)",(id, tabname, descr)
stmt = ibm_db.prepare(conn, inser_sql)
ibm_db.execute(stmt)
this code gives me error:
stmt = ibm_db.prepare(conn, inser_sql)
Exception: statement must be a string or unicode
Assuming that your table is defined like this:
"create table myschema.mytable(id int, tabname varchar(10), description varchar(10))"
I understand your intention is to insert a specific row into it with a prepared statement and parameter markers.
Skipping the input part:
In [14]: id = 1
In [15]: tabname = 'TAB'
In [16]: descr = 'my desc'
you just need to prepare the statement first, bind the parameters later and then execute:
insert_sql = "INSERT INTO myschema.mytable VALUES (?, ?, ?)"
prep_stmt = ibm_db.prepare(conn, insert_sql)
ibm_db.bind_param(prep_stmt, 1, id)
ibm_db.bind_param(prep_stmt, 2, tabname)
ibm_db.bind_param(prep_stmt, 3, descr)
ibm_db.execute(prep_stmt)
The exact answer will depend on the specific DB2 library you're using, but the variable holding your query should just be a plain string.
You will probably need to pass parameters to it when you execute the statement:
inser_sql = "INSERT INTO schema.table VALUES (?, ?, ?)"
stmt = ibm_db.prepare(conn, inser_sql)
ibm_db.execute(stmt, (id, tabname, descr))
# ^^^^^^^^^^^^^^^^^^^^
Dears,
how can I check if pos_cli from database is equal to variable pos_id? for now with code below I get the following error
cur.execute("CREATE TABLE IF NOT EXISTS Magnit_Coor (pos_cli INTEGER PRIMARY KEY, lat INTEGER, long INTEGER);")
cur.execute('SELECT * FROM Magnit_pos')
data = cur.fetchall()
while True:
for coo in data:
full_add = coo[6:11]
pos_id = coo[0]
print (pos_id)
yand_add = ", ".join(full_add)
g = cur.execute('SELECT EXISTS (SELECT * FROM Magnit_Coor WHERE pos_cli = (?))',pos_id)
g = cur.fetchone()[0]
error below
10001
Traceback (most recent call last):
File "geoco.py", line 17, in <module>
g = cur.execute('SELECT EXISTS (SELECT * FROM Magnit_pos WHERE pos_cli = (?))',pos_id)
ValueError: parameters are of unsupported type
The initial code to create Magnit_pos table and pos_cli especially below
cur.execute("DROP TABLE IF EXISTS Magnit_Pos;")
cur.execute(
"CREATE TABLE Magnit_Pos (pos_cli INTEGER PRIMARY KEY, magnit_name TEXT, codesfa TEXT, codewsot TEXT, pos_sap TEXT, source_dc TEXT, zip TEXT, region TEXT, area TEXT, city TEXT, street TEXT, house TEXT, build TEXT);")
with open('magnit.csv') as csvfile:
magnit = csv.reader(csvfile, delimiter=';')
print(magnit)
for row in magnit:
print(row[0])
# to_db = [unicode(row[0], "utf8"), unicode(row[1], "utf8")]
cur.execute("INSERT INTO Magnit_Pos (pos_cli, magnit_name, codesfa, codewsot, pos_sap, source_dc, zip, region, area, city, street, house, build) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);", row)
From python's sqlite3 documentation (emphasis mine):
Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.
So you should be using:
g = cur.execute('SELECT EXISTS (SELECT * FROM Magnit_Coor WHERE pos_cli = (?))',(pos_id,))
import csv
import sqlite3
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
# Close the database
holidayDatabase.close()
The following line
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
is causing this error
Incorrect number of bindings supplied. The current statement uses 5,
and there are 4 supplied.
currentRow is already a sequence. It's a list of all of the fields in the row.
If you were to print out currentRow, you'd get output like this (assuming this is your data set https://data.gov.au/dataset/australian-holidays-machine-readable-dataset):
['Date', 'Holiday Name', 'Information', 'More Information', 'Applicable To']
['20150101', "New Year's Day", "New Year's Day is the first day of the calendaryear and is celebrated each January 1st", '', 'NAT']
['20150126', 'Australia Day', 'Always celebrated on 26 January', 'http://www.australiaday.org.au/', 'NAT']
['20150302', 'Labour Day', 'Always on a Monday, creating a long weekend. It celebrates the eight-hour working day, a victory for workers in the mid-late 19th century.',http://www.commerce.wa.gov.au/labour-relations/public-holidays-western-australia', 'WA']
...
When you do
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
You're actually getting a list of all of the characters in the first element in the list.
Because you didn't skip the header row, you're actually getting a list of the characters in the word "Date". Which equals 4 characters and is causing the error:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current sta
tement uses 5, and there are 4 supplied.
If you were to skip the header line using next(listOfPublicHolidays, None) as in:
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
next(listOfPublicHolidays, None)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
you would get the following error message because currentEntry would be a list of the characters in "20150101", having a length of 8:
Traceback (most recent call last):
File "holidaysorig.py", line 25, in <module>
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', tuple(currentEntry)
)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 5, and there are 8 supplied.
That's why it works (for the most part) when you remove the for currentEntry in currentRow : block and rewrite it as:
import csv
import sqlite3
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
# Close the database
holidayDatabase.close()
NOTE: On my machine, I got the following errors:
(holidays) C:\Users\eyounjo\projects\holidays>python holidaysorig.py
Traceback (most recent call last):
File "holidaysorig.py", line 22, in <module>
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
So I rewrote your script as the following to deal with the above:
import csv, codecs
import sqlite3
# Encoding fix
def latin_1_encoder(unicode_csv_data):
for line in unicode_csv_data:
yield line.encode('latin-1')
fileName = 'australianpublicholidays.csv'
accessMode = 'r'
# Create a database in RAM
holidayDatabase = sqlite3.connect(':memory:')
# Create a cursor
c = holidayDatabase.cursor()
# Create a table
c.execute('''CREATE TABLE holidays
(date text, holidayName text, information text, moreInformation text, applicableTo text)''')
# Read the file contents in to the table
# Encoding fix
with codecs.open(fileName, accessMode, encoding='latin-1') as publicHolidays :
listOfPublicHolidays = csv.reader(latin_1_encoder(publicHolidays))
# Skip the header row
next(listOfPublicHolidays, None)
entries = []
for currentRow in listOfPublicHolidays:
# Work-around for "You must not use 8-bit bytestrings" error
entries.append(tuple([unicode(field, 'latin-1') for field in currentRow]))
c.executemany('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', entries)
# Close the database
holidayDatabase.close()
I have corrected the error by removing the nested for loop
Replaced the following
# Read the file contents in to the table
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
for currentEntry in currentRow :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentEntry)
With the following
with open(fileName, accessMode) as publicHolidays :
listOfPublicHolidays = csv.reader(publicHolidays)
for currentRow in listOfPublicHolidays :
c.execute('INSERT INTO holidays VALUES (?, ?, ?, ?, ?)', currentRow)
However the cause of the error is still unclear to me.