Develop Reference

Upsert SQL server and python pandas

I'm trying to upsert in SQL Server from python.
Basically I have scraped a website, converted it to DF and I'm already inserting it in my DB.
What I need: When there is data different from the scraped like the item price for example, then update it, and if the id does not exist, then insert.
Follows my code:
for index, row in df.iterrows():
cursor.execute("""INSERT INTO db_demo1.[dbo].[scrape]
(market, product_id, section_item, title_item, title_item_new, price_item,
qty, unit, sku, product_image, url, delivery_available,
delivery_long_distance, barcode, scrape_date) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
row.market, row.product_id, row.section_item, row.title_item, row.title_item_new,
row.price_item, row.qty, row.unit, row.sku, row.product_image, row.url,
row.delivery_available, row.delivery_long_distance, row.barcode, row.scrape_date)
cnxn.commit()
cursor.close()
The data that I have is something like this:
In this case, for example, it will check the product_id and see if the price_item or another column have changed, if it did, then it replaces the current value with the new one, and also update the scrape_date with the new date.
#Charlieface
I tried some solutions and none of them worked.
First I tried the articles that you posted, I have to change some parameters because it was raising some errors, but the final output:
'''
BEGIN TRANSACTION;
DECLARE #val as float
DECLARE #pid as int
DECLARE #pk as int
SET IDENTITY_INSERT scrape ON
select * from scrape;
UPDATE dbo.scrape WITH (UPDLOCK, SERIALIZABLE) SET [price_item] = #val, [product_id] = #pid WHERE id = #pk;
GO
IF ##ROWCOUNT = 0
BEGIN
INSERT INTO dbo.scrape([id], [product_id], [price_item]) VALUES(1, 1000, 17.5);
SELECT * FROM scrape
END
SET IDENTITY_INSERT scrape OFF;
COMMIT TRANSACTION;
'''
ERRORS>
Msg 1088, Level 16, State 11, Line 11
Cannot find the object "scrape" because it does not exist or you do not have permissions.
Msg 208, Level 16, State 1, Line 19
Invalid object name 'dbo.scrape'.
I'm the ADM btw, dont get the permission stuff.
second one:
'''
INSERT INTO db_demo1.dbo.scrape(id, product_id, price_item) VALUES(1, 'X', 'X'); -- to be updated
SELECT * FROM db_demo1.dbo.scrape;
MERGE scrape trg
USING (VALUES ('1','2','3'),
('C','D','E'),
('F','G','H'),
('I','J','K')) src(id, product_id, price_item)
ON trg.id = src.id
WHEN MATCHED THEN
UPDATE SET product_id = src.product_id, price_item = src.price_item
WHEN NOT MATCHED THEN
INSERT(id, product_id, price_item)
VALUES(src.id, src.product_id, src.price_item);
SELECT * FROM scrape;
'''
Same error: Invalid object name 'db_demo1.dbo.scrape'.
Someone please can save me?

Python Insert Data from Qtable Widget into Ms Access with QSqlDatabase

This is what I have so far:
def save_invoice(self):
con = QSqlDatabase.addDatabase("QODBC")
con.setDatabaseName("C:/Users/Egon/Documents/Invoice/Invoice.accdb")
# Open the connection
con.open()
# Creating a query for later execution using .prepare()
insertDataQuery = QSqlQuery()
insertDataQuery.prepare(
"""
INSERT INTO Test01 (
Quantity,
ProductId,
Description,
Price,
Tax,
NetTotal,
GrossTotal
)
VALUES (?, ?, ?, ?, ?, ?, ?)
"""
)
data = getData(self.tableWidgetInvoiceItem)
for Quantity, ProductId, Description, Price, Tax, NetTotal, GrossTotal in data:
insertDataQuery.addBindValue(Quantity)
insertDataQuery.addBindValue(ProductId)
insertDataQuery.addBindValue(Description)
insertDataQuery.addBindValue(Price)
insertDataQuery.addBindValue(Tax)
insertDataQuery.addBindValue(NetTotal)
insertDataQuery.addBindValue(GrossTotal)
insertDataQuery.exec_()
print(insertDataQuery.lastError().text())
con.commit()
Fetch the data from the QTableWidget and return it as data.
def getData(table: QTableWidget) -> List[Tuple[str]]:
data = []
for row in range(table.rowCount()):
rowData = []
for col in range(table.columnCount()):
rowData.append(table.item(row, col).data(Qt.EditRole))
data.append(tuple(rowData))
return data
No error message is displayed but also no records are inserted into database. How can I solve this?

Try using QSqlDatabase.commit() instead of con.commit().

Dynamically inserting values into a sqlite table doesn't work (Code attached)

I'm trying to dynamically insert values into the sqlite table but it is not working. Can you please look at below code and point out if I'm making any mistakes.
import sqlite3
def Create_Connection(db_file):
Conn = None
try:
Conn = sqlite3.Connection(db_file)
print("Connection Established")
except Exception as E:
print(E) #LOG THIS
return Conn
def Enter_Values(Group, Timestamp, URL, Response, StatusCode, Message, IsCriticalError):
if (Group == 'Google'):
Query = "INSERT INTO Google(Timestamp, URL, Response, StatusCode, Message, IsCriticalError) VALUES(?, ?, ?, ?, ?, ?)",(
Timestamp, URL, Response, StatusCode, Message, IsCriticalError)
print(Query)
Conn1 = Create_Connection('Urldb.db')
if Conn1 is not None:
Cursor = Conn1.cursor()
Cursor.execute(Query)
Conn1.commit()
elif (Group == 'Wikipedia'):
Query = "INSERT INTO Wikipedia (Timestamp,URL,Response,StatusCode,Message,IsCriticalError) VALUES (?, ?, ?, ?, ?, ?)",(
Timestamp, URL, Response, StatusCode, Message, IsCriticalError)
Conn1 = Create_Connection('Urldb.db')
if Conn1 is not None:
print('Creating Cursor')
Cursor = Conn1.cursor()
Cursor.execute(Query)
Conn1.commit()
The 'Query' variable with the dynamic values amounts to this when I run the code:
('INSERT INTO Google(Timestamp, URL, Response, StatusCode, Message, IsCriticalError) VALUES(?, ?, ?, ?, ?, ?)', ('04:38:41', 'https://www.google.com/', 'OK', 200, 'OK', 0))
The datatypes I used while creating the table are as follows:
CREATE TABLE IF NOT EXISTS {} (
Timestamp text,
URL text,
Response text,
StatusCode integer,
Message Text,
IsCriticalError integer
)""".format(Group)
I'm using this function in another module. I don't get any errors when I run this, just that data is not added to the database.
Thanks in advance.

When using a parameterized query you need 2 arguments to execute, try
query = """INSERT INTO Wikipedia(Timestamp, URL, Response, StatusCode, Message, IsCriticalError) VALUES (?, ?, ?, ?, ?, ?)"""
params = (Timestamp, URL, Response, StatusCode, Message, IsCriticalError)
Cursor.execute(query, params)

Loop breaks when executing function

I have the code below:
def scrapeFacebookPageFeedStatus(access_token):
query = "SELECT page_id FROM falken"
result_list = c.execute(query)
for single_row in result_list:
str_single_row = str(single_row)
str_norm_single_row = str_normalize(str_single_row)
print(str_norm_single_row)
When I execute the code above, It displays every single_row values from result_list.
But when I pass single_row to a function like below:
def scrapeFacebookPageFeedStatus(access_token):
query = "SELECT page_id FROM falken"
result_list = c.execute(query)
for single_row in result_list:
str_single_row = str(single_row)
str_norm_single_row = str_normalize(str_single_row)
print(str_norm_single_row)
statuses = getFacebookPageFeedData(str_norm_single_row, access_token, 100)
for status in statuses['data']:
query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
c.execute(query,(processFacebookPageFeedStatus(status, access_token)))
conn.commit()
It only passes the first value of single_row to the function and the loop stops.
getFacebookPageFeedData function
def getFacebookPageFeedData(page_id, access_token, num_statuses):
base = "https://graph.facebook.com/v2.6"
node = "/%s/posts" % page_id
fields = "/?fields=message,link,created_time,type,name,id," + \
"comments.limit(0).summary(true),shares,reactions" + \
".limit(0).summary(true)"
parameters = "&limit=%s&access_token=%s" % (num_statuses, access_token)
url = base + node + fields + parameters
# retrieve data
data = json.loads(request_until_succeed(url))
return data
It retrieves data from Facebook Graph API of posts from pages.
processFacebookPageFeedStatus function
def processFacebookPageFeedStatus(status, access_token):
status_id = status['id']
status_message = '' if 'message' not in status.keys() else \
unicode_normalize(status['message'])
link_name = '' if 'name' not in status.keys() else \
unicode_normalize(status['name'])
status_type = status['type']
status_link = '' if 'link' not in status.keys() else \
unicode_normalize(status['link'])
status_published = datetime.datetime.strptime(
status['created_time'],'%Y-%m-%dT%H:%M:%S+0000')
status_published = status_published + \
datetime.timedelta(hours=-5) # EST
status_published = status_published.strftime(
'%Y-%m-%d %H:%M:%S')
num_reactions = 0 if 'reactions' not in status else \
status['reactions']['summary']['total_count']
num_comments = 0 if 'comments' not in status else \
status['comments']['summary']['total_count']
num_shares = 0 if 'shares' not in status else status['shares']['count']
reactions = getReactionsForStatus(status_id, access_token) if \
status_published > '2016-02-24 00:00:00' else {}
num_likes = 0 if 'like' not in reactions else \
reactions['like']['summary']['total_count']
num_likes = num_reactions if status_published < '2016-02-24 00:00:00' \
else num_likes
It stores needed data from status dictionary and stores it to variables for inserting into database.

sqlite's cursor.execute() returns the cursor itself. So after this line:
result_list = c.execute(query)
result_list is actually an alias for c.
Now you start iterating over c:
for single_row in result_list:
# code here
and then call c.execute() again:
query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
c.execute(query,(processFacebookPageFeedStatus(status, access_token)))
which discards c's previous result set with the result of this new query. Since this query doesn't select anything, c becomes an empty iterator, and your loop stops there.
The cure is plain and simple: use another cursor for the insert queries so you don't overwrite c's result set:
# create a second cursor for insert statements
writer = conn.cursor()
# no need to recreate this same string anew for each iteration,
# we can as well define it here once for all
insert_query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
# no need for result_list - just iterate over `c`
c.execute(query)
for single_row in c:
# code here
writer.execute(insert_query,(processFacebookPageFeedStatus(status, access_token)))
As a side note, if performances are an issue, you may also want to commit only once after the whole loop instead of after each insert statement.

python dynamic input, update table

I wrote a program in order to dynamically update a database table but I am getting an error. I stuffed the program with whatever I know little about. Here's my code:
import MySQLdb
class data:
def __init__(self):
self.file123 = raw_input("Enter film: ")
self.title_ = raw_input("Enter film: ")
self.year = raw_input("Enter year: ")
self.director = raw_input("Enter director: ")
a=data()
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="root", # your username
passwd="mysql", # your password
db="sakila") # name of the data base
cursor = db.cursor()
cursor.execute("INSERT INTO films (file123, title_, year, director) VALUES (?, ?, ?, ?)", (a.file123, a.title_, a.year, a.director))
db.commit()
db.close()
This is the error:
File "C:\Python27\maybe1.py", line 20, in <module>
cursor.execute("INSERT INTO films (file123, title_, year, director) VALUES (?, ?, ?, ?)", (a.file123, a.title_, a.year, a.director))
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 184, in execute
query = query % db.literal(args)
TypeError: not all arguments converted during string formatting
How can I fix this issue ?

You should change ? to %s.
Here is question about why mysqldb use %s instead of ?.

I would do it this way:
query = "INSERT INTO films (file123, title_, year, director) VALUES (%s, %s, %s, %s)" % (a.file123, a.title_, a.year, a.director)
cursor.execute(query)
Replace %s with correct data type, else it will try everything as string which might break at table level.