Develop Reference

Loop works fine but when sqlite execute called it stops after first line

It's the first stage of a Python for Everyone data visualisation project.
I can't work out why the loop stops working during the SQL commands in lines 90-91.
I've tested it segment by segment and the loop works fine if you comment out the last MySQL commands, but it stops working after adding one successful row when you leave them in.
import urllib.request, urllib.parse, urllib.error
import sqlite3
import json
import ssl
api_key = "800a5c3b"
serviceurl = "http://www.omdbapi.com/?"
conn = sqlite3.connect('omdb.sqlite')
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS Omdbdump;')
cur.execute('''CREATE TABLE Omdbdump (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT, year TEXT, rated TEXT, released TEXT, runtime TEXT, genre TEXT, director TEXT, writer TEXT, actors TEXT, plotlong TEXT, language TEXT, country TEXT, awards TEXT, poster URL, imdbrating REAL, rtrating REAL, mcrating REAL, imdbid TEXT, type TEXT, dvd TEXT, boxoffice TEXT, production TEXT, website URL)
''')
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
fh = cur.execute('''SELECT id, title, year FROM top50''')
rty = list()
for row in fh:
#row = cur.fetchone()
qtitle = str(row[1])
qyear = str(row[2])
#print(rty)
print(qtitle)
print(qyear)
#parms sets up the query url: url concatenated with address and api key
#query format https://www.omdbapi.com/?t=blade+runner&y=2018&plot=full&apikey=800a5c3b
parms = dict()
parms["t"] = qtitle
parms["y"] = qyear
parms["plot"] = "full"
parms["apikey"] = api_key
url = serviceurl + urllib.parse.urlencode(parms)
print('Retrieving', url)
uh = urllib.request.urlopen(url, context=ctx)
data = uh.read().decode()
print('Retrieved', len(data), 'characters', data[:20].replace('\n', ' '))
js = json.loads(data)
if js['Response'] == 'False':
print('==== Failure To Retrieve ====')
print(data)
continue
title = js['Title']
year = js['Year']
rated = js['Rated']
released = js['Released']
runtime = js['Runtime']
genre = js['Genre']
director = js['Director']
writer = js['Writer']
actors = js['Actors']
plotlong = js['Plot']
language = js['Language']
country = js['Country']
awards = js['Awards']
poster = js['Poster']
imdbrating = js['imdbRating']
mcrating = js['Metascore']
imdbid = js['imdbID']
type = js['Type']
dvd = js['DVD']
boxoffice = js['BoxOffice']
production = js['Production']
website = js['Website']
try:
rtrating = js['Ratings'][1]['Value']
except:
rtrating = 'N/A'
print(title)
print(imdbid)
print(runtime)
#the loop works until here -- with the following lines, it goes through once then stops...
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) )
conn.commit()
print("Done")

Program changes the "value" of the cursor here
cur.execute('''INSERT INTO Omdbdump.......)
while iterating over the cursor here for row in fh:.
Possible solutions:
create another cursor for the insert
use the connection's execute method for the insert.

The accepted answer is a great explanation from [DinoCoderSaurus][1] that helped me answer my own [question][2] but the connection.execute() suggestion does not work in apsw. I had to concatenate some sql and take some of the calls into a new loop after using a list of tuples to link the two loops.
The answer to the OP's case would be to change:
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) )
to:
values=[] #outside the original loop
#then inside the loop
values.append((title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website))
#then after the values harvesting loop has closed a new loop:
for value in values:
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', value)
[1]: https://stackoverflow.com/users/5577076/dinocodersaurus
[2]: https://stackoverflow.com/questions/69706486/why-does-apsw-cursor-executesql-vars-break-outer-for-loop/69712751#69712751

Sqlite3 error 2 arguments given (7given) when finding data

def addPersonRec(Firstname, Surname, DoB, Age, Notes, DateCreated):
con=sqlite3.connect("silver.db")
cur = con.cursor()
cur.execute("INSERT INTO silver VAULES (NULL, ?, ?, ?, ?, ?, ?)", Firstname, Surname, DoB, Age, Notes, DateCreated)
con.commit()
con.close()
so my issue is i have improted sqlite 3 and when i run it i get this error line 14, in addPersonRec
cur.execute("INSERT INTO silver VAULES (NULL, ?, ?, ?, ?, ?, ?)",Firstname, Surname, DoB, Age, Notes, DateCreated)
TypeError: function takes at most 2 arguments (7 given) please any ideas help

.execute's second argument should be a tuple.
Also note that you have a typo. VAULES should be VALUES.
cur.execute("INSERT INTO silver VALUES (NULL, ?, ?, ?, ?, ?, ?)",
(Firstname, Surname, DoB, Age, Notes, DateCreated))

Updating SQL with Python - Incorrect number of bindings

I keep getting this error
Traceback (most recent call last):
File "test.py", line 7, in <module>
c.execute("INSERT INTO file_routes (file_route) VALUES (?)", (file_route))
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 4 supplied.
I create the table with this
createTable_filenames = """
CREATE TABLE IF NOT EXISTS
file_routes(
routes_id INTEGER PRIMARY KEY,
file_route TEXT
)
"""
This is the code I am trying to run and it doesn't work
import sqlite3
conn = sqlite3.connect('fantasyresults.db')
c = conn.cursor()
file_route = "test"
c.execute("INSERT INTO file_routes (file_route) VALUES (?)", (file_route))
conn.commit()
I don't understand why this code works, I set up the structure the same way.
c.execute than ("INSERT INTO <tablename> (<column id>) VALUES (?)", (<variable>))
c.execute("INSERT INTO draftkings (account_id, sport, game_type, entry_key, entry_name, opponent, contest_key, contest_date, position, points, winnings, winnings_tickets, contest_entries, entry_fee, prize_pool, places_paid)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", (account_id, sport, game_type, entry_key, entry_name, opponent, contest_key, contest_date, position, points, winnings, winnings_tickets, contest_entries, entry_fee, prize_pool, places_paid))
Any help is much appreciated, I don't know what I am not seeing here.

The problem is that it is considering the parameter as string instead of tuple as that's the expected behavior in python for a tuple with single element.
Try updating the parameter tuple as
(file_route, )
have a comma after the parameter.

Python - SQLite : Update two record with same identify value

i have a script like below :
import psutil
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
result = c.execute("SELECT * FROM clientinfo WHERE IP = ?", ("192.168.10.111",))
if (len(result.fetchall()) > 0):
for x in DISK :
c.execute("UPDATE disk SET Device = ?, 'Mount Point' = ?, 'fstyle' = ?, 'opts' = ?, 'total' = ?, 'used' = ?, 'free' = ?, 'percent' = ? WHERE Client_IP = ?", (DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7], "192.168.10.111"))
else :
for x in DISK :
c.execute("INSERT INTO disk('Client_IP', 'Device', 'Mount Point', 'fstyle', 'opts', 'total', 'used', 'free', 'percent') VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", ("192.168.10.111", DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7]))
conn.commit()
conn.close()
Script will check in db if there is any record about IP "192.168.10.111". If db already have record about IP "192.168.10.111", Script will update data from dict DISK to table disk of db.
If db don't have record about IP "192.168.10.111", Script will create record to insert **DISK"" to database.
INSERT command work well but the UPDATE command don't work like i want. After INSERT command run, in table disk i will have two record about disk C and disk D with same value of column Client_IP (192.168.10.111).
After UPDATE, two record of IP "192.168.10.111" get same value on every column ehich is very wrong. One record must be contain information about disk C and another record cotain disk D information.
How can i make the UPDATE work right ? length of dict DISK depend on how many mounted devices the computer has. So i need to use for loop to UPDATE but not static UPDATE.
Please tell me how to fix this,
Many thanks,
Francis

Your SELECT query column value is incorrect based on your UPDATE and INSERT queries later on. Instead of IP, should't it be CLIENT_IP? Also, you need to change your string query formatting. Remove the single quotes form your column names. Lastely, you can shorten your code by using a simple list comprehension and cursor.executemany:
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
if not list(c.execute('SELECT * FROM lientinfo WHERE CLIENT_IP = ?', ("192.168.10.111",))):
c.executemany("INSERT INTO disk (Client_IP, Device, Mount Point, fstyle, opts, total, used, free, percent) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", [["192.168.10.111", *i] for i in DISK.values()])
else:
c.executemany('UPDATE disk SET Device = ?, Mount Point = ?, fstyle = ?, opts = ?, total = ?, used = ?, free = ?, percent = ? WHERE Client_IP = ?', [[*i, "192.168.10.111"] for i in DISK.values()])

Loop breaks when executing function

I have the code below:
def scrapeFacebookPageFeedStatus(access_token):
query = "SELECT page_id FROM falken"
result_list = c.execute(query)
for single_row in result_list:
str_single_row = str(single_row)
str_norm_single_row = str_normalize(str_single_row)
print(str_norm_single_row)
When I execute the code above, It displays every single_row values from result_list.
But when I pass single_row to a function like below:
def scrapeFacebookPageFeedStatus(access_token):
query = "SELECT page_id FROM falken"
result_list = c.execute(query)
for single_row in result_list:
str_single_row = str(single_row)
str_norm_single_row = str_normalize(str_single_row)
print(str_norm_single_row)
statuses = getFacebookPageFeedData(str_norm_single_row, access_token, 100)
for status in statuses['data']:
query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
c.execute(query,(processFacebookPageFeedStatus(status, access_token)))
conn.commit()
It only passes the first value of single_row to the function and the loop stops.
getFacebookPageFeedData function
def getFacebookPageFeedData(page_id, access_token, num_statuses):
base = "https://graph.facebook.com/v2.6"
node = "/%s/posts" % page_id
fields = "/?fields=message,link,created_time,type,name,id," + \
"comments.limit(0).summary(true),shares,reactions" + \
".limit(0).summary(true)"
parameters = "&limit=%s&access_token=%s" % (num_statuses, access_token)
url = base + node + fields + parameters
# retrieve data
data = json.loads(request_until_succeed(url))
return data
It retrieves data from Facebook Graph API of posts from pages.
processFacebookPageFeedStatus function
def processFacebookPageFeedStatus(status, access_token):
status_id = status['id']
status_message = '' if 'message' not in status.keys() else \
unicode_normalize(status['message'])
link_name = '' if 'name' not in status.keys() else \
unicode_normalize(status['name'])
status_type = status['type']
status_link = '' if 'link' not in status.keys() else \
unicode_normalize(status['link'])
status_published = datetime.datetime.strptime(
status['created_time'],'%Y-%m-%dT%H:%M:%S+0000')
status_published = status_published + \
datetime.timedelta(hours=-5) # EST
status_published = status_published.strftime(
'%Y-%m-%d %H:%M:%S')
num_reactions = 0 if 'reactions' not in status else \
status['reactions']['summary']['total_count']
num_comments = 0 if 'comments' not in status else \
status['comments']['summary']['total_count']
num_shares = 0 if 'shares' not in status else status['shares']['count']
reactions = getReactionsForStatus(status_id, access_token) if \
status_published > '2016-02-24 00:00:00' else {}
num_likes = 0 if 'like' not in reactions else \
reactions['like']['summary']['total_count']
num_likes = num_reactions if status_published < '2016-02-24 00:00:00' \
else num_likes
It stores needed data from status dictionary and stores it to variables for inserting into database.

sqlite's cursor.execute() returns the cursor itself. So after this line:
result_list = c.execute(query)
result_list is actually an alias for c.
Now you start iterating over c:
for single_row in result_list:
# code here
and then call c.execute() again:
query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
c.execute(query,(processFacebookPageFeedStatus(status, access_token)))
which discards c's previous result set with the result of this new query. Since this query doesn't select anything, c becomes an empty iterator, and your loop stops there.
The cure is plain and simple: use another cursor for the insert queries so you don't overwrite c's result set:
# create a second cursor for insert statements
writer = conn.cursor()
# no need to recreate this same string anew for each iteration,
# we can as well define it here once for all
insert_query = "INSERT INTO falken_posts VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
# no need for result_list - just iterate over `c`
c.execute(query)
for single_row in c:
# code here
writer.execute(insert_query,(processFacebookPageFeedStatus(status, access_token)))
As a side note, if performances are an issue, you may also want to commit only once after the whole loop instead of after each insert statement.