Related
It's the first stage of a Python for Everyone data visualisation project.
I can't work out why the loop stops working during the SQL commands in lines 90-91.
I've tested it segment by segment and the loop works fine if you comment out the last MySQL commands, but it stops working after adding one successful row when you leave them in.
import urllib.request, urllib.parse, urllib.error
import sqlite3
import json
import ssl
api_key = "800a5c3b"
serviceurl = "http://www.omdbapi.com/?"
conn = sqlite3.connect('omdb.sqlite')
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS Omdbdump;')
cur.execute('''CREATE TABLE Omdbdump (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT, year TEXT, rated TEXT, released TEXT, runtime TEXT, genre TEXT, director TEXT, writer TEXT, actors TEXT, plotlong TEXT, language TEXT, country TEXT, awards TEXT, poster URL, imdbrating REAL, rtrating REAL, mcrating REAL, imdbid TEXT, type TEXT, dvd TEXT, boxoffice TEXT, production TEXT, website URL)
''')
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
fh = cur.execute('''SELECT id, title, year FROM top50''')
rty = list()
for row in fh:
#row = cur.fetchone()
qtitle = str(row[1])
qyear = str(row[2])
#print(rty)
print(qtitle)
print(qyear)
#parms sets up the query url: url concatenated with address and api key
#query format https://www.omdbapi.com/?t=blade+runner&y=2018&plot=full&apikey=800a5c3b
parms = dict()
parms["t"] = qtitle
parms["y"] = qyear
parms["plot"] = "full"
parms["apikey"] = api_key
url = serviceurl + urllib.parse.urlencode(parms)
print('Retrieving', url)
uh = urllib.request.urlopen(url, context=ctx)
data = uh.read().decode()
print('Retrieved', len(data), 'characters', data[:20].replace('\n', ' '))
js = json.loads(data)
if js['Response'] == 'False':
print('==== Failure To Retrieve ====')
print(data)
continue
title = js['Title']
year = js['Year']
rated = js['Rated']
released = js['Released']
runtime = js['Runtime']
genre = js['Genre']
director = js['Director']
writer = js['Writer']
actors = js['Actors']
plotlong = js['Plot']
language = js['Language']
country = js['Country']
awards = js['Awards']
poster = js['Poster']
imdbrating = js['imdbRating']
mcrating = js['Metascore']
imdbid = js['imdbID']
type = js['Type']
dvd = js['DVD']
boxoffice = js['BoxOffice']
production = js['Production']
website = js['Website']
try:
rtrating = js['Ratings'][1]['Value']
except:
rtrating = 'N/A'
print(title)
print(imdbid)
print(runtime)
#the loop works until here -- with the following lines, it goes through once then stops...
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) )
conn.commit()
print("Done")
Program changes the "value" of the cursor here
cur.execute('''INSERT INTO Omdbdump.......)
while iterating over the cursor here for row in fh:.
Possible solutions:
create another cursor for the insert
use the connection's execute method for the insert.
The accepted answer is a great explanation from [DinoCoderSaurus][1] that helped me answer my own [question][2] but the connection.execute() suggestion does not work in apsw. I had to concatenate some sql and take some of the calls into a new loop after using a list of tuples to link the two loops.
The answer to the OP's case would be to change:
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) )
to:
values=[] #outside the original loop
#then inside the loop
values.append((title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website))
#then after the values harvesting loop has closed a new loop:
for value in values:
cur.execute('''INSERT INTO Omdbdump (title, year, rated, released, runtime, genre, director, writer, actors, plotlong, language, country, awards, poster, imdbrating, rtrating, mcrating, imdbid, type, dvd, boxoffice, production, website) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )''', value)
[1]: https://stackoverflow.com/users/5577076/dinocodersaurus
[2]: https://stackoverflow.com/questions/69706486/why-does-apsw-cursor-executesql-vars-break-outer-for-loop/69712751#69712751
I keep getting this error
Traceback (most recent call last):
File "test.py", line 7, in <module>
c.execute("INSERT INTO file_routes (file_route) VALUES (?)", (file_route))
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 4 supplied.
I create the table with this
createTable_filenames = """
CREATE TABLE IF NOT EXISTS
file_routes(
routes_id INTEGER PRIMARY KEY,
file_route TEXT
)
"""
This is the code I am trying to run and it doesn't work
import sqlite3
conn = sqlite3.connect('fantasyresults.db')
c = conn.cursor()
file_route = "test"
c.execute("INSERT INTO file_routes (file_route) VALUES (?)", (file_route))
conn.commit()
I don't understand why this code works, I set up the structure the same way.
c.execute than ("INSERT INTO <tablename> (<column id>) VALUES (?)", (<variable>))
c.execute("INSERT INTO draftkings (account_id, sport, game_type, entry_key, entry_name, opponent, contest_key, contest_date, position, points, winnings, winnings_tickets, contest_entries, entry_fee, prize_pool, places_paid)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", (account_id, sport, game_type, entry_key, entry_name, opponent, contest_key, contest_date, position, points, winnings, winnings_tickets, contest_entries, entry_fee, prize_pool, places_paid))
Any help is much appreciated, I don't know what I am not seeing here.
The problem is that it is considering the parameter as string instead of tuple as that's the expected behavior in python for a tuple with single element.
Try updating the parameter tuple as
(file_route, )
have a comma after the parameter.
I'm trying to insert the following JSON data into a table on our SQL Server with python code.
('{"ITEMCODE":"A00975605","DISPLAYREVISIONNO":1,"ITEMNAME":"ABCDEFGHIJKLMN","ITEMNAME_ID":"~0000PCLAZ","ITEMTYPE":2,"ITEMTYPEEX":0,"PRODUCTTYPE":12,"LOTTYPE":"S1","EDITIONNOTYPE":0,"ISINVENTORY":2,"HOSTMANAGETYPE":-1,"SHIPMENTJUDGETYPE":null,"UNITVOLUME":1000,"TAREWEIGHT":0,"DEFAULTRECIPECODE":null,"ISQUALITYCHECK":1,"ISALLOCATEMORELOT":0,"ISINSERTLOTCHECK":-1,"RECEIPTVOLUMELIMIT":"10","RECEIPTZONECODE":"FS01","WAREHOUSEZONECODE":null,"SHIPMENTZONECODE":null,"LABELCODE":null,"THEME":null,"ARTICLENO":null,"ARTICLENAME":null,"ARTICLENAME_ID":null,"PACKAGEINDICATOR":null,"PRODUCTIONFLOWNO":null,"RECEIPTFLOWNO":null,"TESTFLOWNO":null,"RECORDREVIEWFLOWNO":null,"SHIPMENTGMPFLOWNO":null,"SHIPMENTGQPFLOWNO":null,"NOTE":null,"NOTE_ID":null,"STRENGTHUNITFLAG":0,"MEDICINEDIVISION":null,"EVALUATIONCLASS":"6200","SAMPLINGVALUEFORLIST":null,"STOCKVALUEFORLIST":null,"FILLINGVOLUME":null,"FILLINGVOLUMEMAXIMUMLIMIT":null,"FILLINGVOLUMEMINIMUMLIMIT":null,"FILTRATIONTYPE":null,"STERILIZATIONTYPE":null,"BOTTLEWASHING":null,"STRENGTHUNITFLAGEX":0,"DEFAULTMAKERCODE":"H75776","PRODUCTCODEFORBCD ":null,"STOCKCONDITIONCODE":5,"METALUSE":null,"AUTOALLOCFLAG":1,"INNERPACKAGEVOLUME ":1000,"AREACODE":"A1","STANDARDSTRENGTHUNITRATIO":1}',)
('{"ITEMCODE":"A00975612","DISPLAYREVISIONNO":1,"ITEMNAME":"BCDEFGHIJKLMNO","ITEMNAME_ID":"~0000PCLHS","ITEMTYPE":2,"ITEMTYPEEX":0,"PRODUCTTYPE":12,"LOTTYPE":"S1","EDITIONNOTYPE":0,"ISINVENTORY":2,"HOSTMANAGETYPE":-1,"SHIPMENTJUDGETYPE":null,"UNITVOLUME":1000,"TAREWEIGHT":0,"DEFAULTRECIPECODE":null,"ISQUALITYCHECK":1,"ISALLOCATEMORELOT":0,"ISINSERTLOTCHECK":-1,"RECEIPTVOLUMELIMIT":"10","RECEIPTZONECODE":"FS01","WAREHOUSEZONECODE":null,"SHIPMENTZONECODE":null,"LABELCODE":null,"THEME":null,"ARTICLENO":null,"ARTICLENAME":null,"ARTICLENAME_ID":null,"PACKAGEINDICATOR":null,"PRODUCTIONFLOWNO":null,"RECEIPTFLOWNO":null,"TESTFLOWNO":null,"RECORDREVIEWFLOWNO":null,"SHIPMENTGMPFLOWNO":null,"SHIPMENTGQPFLOWNO":null,"NOTE":null,"NOTE_ID":null,"STRENGTHUNITFLAG":0,"MEDICINEDIVISION":null,"EVALUATIONCLASS":"6200","SAMPLINGVALUEFORLIST":null,"STOCKVALUEFORLIST":null,"FILLINGVOLUME":null,"FILLINGVOLUMEMAXIMUMLIMIT":null,"FILLINGVOLUMEMINIMUMLIMIT":null,"FILTRATIONTYPE":null,"STERILIZATIONTYPE":null,"BOTTLEWASHING":null,"STRENGTHUNITFLAGEX":0,"DEFAULTMAKERCODE":"H75776","PRODUCTCODEFORBCD ":null,"STOCKCONDITIONCODE":5,"METALUSE":null,"AUTOALLOCFLAG":1,"INNERPACKAGEVOLUME ":1000,"AREACODE":"A1","STANDARDSTRENGTHUNITRATIO":1}',)
('{"ITEMCODE":"A00977746","DISPLAYREVISIONNO":1,"ITEMNAME":"CDEFGHIJKLMNOP","ITEMNAME_ID":"~0000PCJZN","ITEMTYPE":2,"ITEMTYPEEX":0,"PRODUCTTYPE":12,"LOTTYPE":"S1","EDITIONNOTYPE":0,"ISINVENTORY":2,"HOSTMANAGETYPE":-1,"SHIPMENTJUDGETYPE":null,"UNITVOLUME":1000,"TAREWEIGHT":0,"DEFAULTRECIPECODE":null,"ISQUALITYCHECK":1,"ISALLOCATEMORELOT":0,"ISINSERTLOTCHECK":-1,"RECEIPTVOLUMELIMIT":"10","RECEIPTZONECODE":"FS01","WAREHOUSEZONECODE":null,"SHIPMENTZONECODE":null,"LABELCODE":null,"THEME":null,"ARTICLENO":null,"ARTICLENAME":null,"ARTICLENAME_ID":null,"PACKAGEINDICATOR":null,"PRODUCTIONFLOWNO":null,"RECEIPTFLOWNO":null,"TESTFLOWNO":null,"RECORDREVIEWFLOWNO":null,"SHIPMENTGMPFLOWNO":null,"SHIPMENTGQPFLOWNO":null,"NOTE":null,"NOTE_ID":null,"STRENGTHUNITFLAG":0,"MEDICINEDIVISION":null,"EVALUATIONCLASS":"6200","SAMPLINGVALUEFORLIST":null,"STOCKVALUEFORLIST":null,"FILLINGVOLUME":null,"FILLINGVOLUMEMAXIMUMLIMIT":null,"FILLINGVOLUMEMINIMUMLIMIT":null,"FILTRATIONTYPE":null,"STERILIZATIONTYPE":null,"BOTTLEWASHING":null,"STRENGTHUNITFLAGEX":0,"DEFAULTMAKERCODE":"H75776","PRODUCTCODEFORBCD ":null,"STOCKCONDITIONCODE":5,"METALUSE":null,"AUTOALLOCFLAG":1,"INNERPACKAGEVOLUME ":1000,"AREACODE":"A1","STANDARDSTRENGTHUNITRATIO":1}',)
('{"ITEMCODE":"A00977753","DISPLAYREVISIONNO":1,"ITEMNAME":"DEFGHIJKLMNOPQ","ITEMNAME_ID":"~0000PCJZP","ITEMTYPE":2,"ITEMTYPEEX":0,"PRODUCTTYPE":12,"LOTTYPE":"S1","EDITIONNOTYPE":0,"ISINVENTORY":2,"HOSTMANAGETYPE":-1,"SHIPMENTJUDGETYPE":null,"UNITVOLUME":1000,"TAREWEIGHT":0,"DEFAULTRECIPECODE":null,"ISQUALITYCHECK":1,"ISALLOCATEMORELOT":0,"ISINSERTLOTCHECK":-1,"RECEIPTVOLUMELIMIT":"10","RECEIPTZONECODE":"FS01","WAREHOUSEZONECODE":null,"SHIPMENTZONECODE":null,"LABELCODE":null,"THEME":null,"ARTICLENO":null,"ARTICLENAME":null,"ARTICLENAME_ID":null,"PACKAGEINDICATOR":null,"PRODUCTIONFLOWNO":null,"RECEIPTFLOWNO":null,"TESTFLOWNO":null,"RECORDREVIEWFLOWNO":null,"SHIPMENTGMPFLOWNO":null,"SHIPMENTGQPFLOWNO":null,"NOTE":null,"NOTE_ID":null,"STRENGTHUNITFLAG":0,"MEDICINEDIVISION":null,"EVALUATIONCLASS":"6200","SAMPLINGVALUEFORLIST":null,"STOCKVALUEFORLIST":null,"FILLINGVOLUME":null,"FILLINGVOLUMEMAXIMUMLIMIT":null,"FILLINGVOLUMEMINIMUMLIMIT":null,"FILTRATIONTYPE":null,"STERILIZATIONTYPE":null,"BOTTLEWASHING":null,"STRENGTHUNITFLAGEX":0,"DEFAULTMAKERCODE":"H75776","PRODUCTCODEFORBCD ":null,"STOCKCONDITIONCODE":5,"METALUSE":null,"AUTOALLOCFLAG":1,"INNERPACKAGEVOLUME ":1000,"AREACODE":"A1","STANDARDSTRENGTHUNITRATIO":1}',)
('{"ITEMCODE":"A00977760","DISPLAYREVISIONNO":1,"ITEMNAME":"EFGHIJKLMNOPQR","ITEMNAME_ID":"~0000PCJZR","ITEMTYPE":2,"ITEMTYPEEX":0,"PRODUCTTYPE":12,"LOTTYPE":"S1","EDITIONNOTYPE":0,"ISINVENTORY":2,"HOSTMANAGETYPE":-1,"SHIPMENTJUDGETYPE":null,"UNITVOLUME":1000,"TAREWEIGHT":0,"DEFAULTRECIPECODE":null,"ISQUALITYCHECK":1,"ISALLOCATEMORELOT":0,"ISINSERTLOTCHECK":-1,"RECEIPTVOLUMELIMIT":"10","RECEIPTZONECODE":"FS01","WAREHOUSEZONECODE":null,"SHIPMENTZONECODE":null,"LABELCODE":null,"THEME":null,"ARTICLENO":null,"ARTICLENAME":null,"ARTICLENAME_ID":null,"PACKAGEINDICATOR":null,"PRODUCTIONFLOWNO":null,"RECEIPTFLOWNO":null,"TESTFLOWNO":null,"RECORDREVIEWFLOWNO":null,"SHIPMENTGMPFLOWNO":null,"SHIPMENTGQPFLOWNO":null,"NOTE":null,"NOTE_ID":null,"STRENGTHUNITFLAG":0,"MEDICINEDIVISION":null,"EVALUATIONCLASS":"6200","SAMPLINGVALUEFORLIST":null,"STOCKVALUEFORLIST":null,"FILLINGVOLUME":null,"FILLINGVOLUMEMAXIMUMLIMIT":null,"FILLINGVOLUMEMINIMUMLIMIT":null,"FILTRATIONTYPE":null,"STERILIZATIONTYPE":null,"BOTTLEWASHING":null,"STRENGTHUNITFLAGEX":0,"DEFAULTMAKERCODE":"H75776","PRODUCTCODEFORBCD ":null,"STOCKCONDITIONCODE":5,"METALUSE":null,"AUTOALLOCFLAG":1,"INNERPACKAGEVOLUME ":1000,"AREACODE":"A1","STANDARDSTRENGTHUNITRATIO":1}',)
And if I could solve this with executing a SQL statement, I would be so happy.
Because our Application server with python and SQL Server are completely different machines. They are actually far apart each other.
Attempting the code below, I have run into an error. Could anyone give me some advice?
def jsonINSERT(_cn, _cur, jdata):
SQL = """
INSERT INTO TSTTBL VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
"""
_cur.executemany(SQL, jdata)
_cn.commit()
return
'The SQL contains 56 parameter markers, but 1 parameters were supplied', 'HY000'
I did like this eventually.
def jsonINSERT(_cn, _cur, sql, jdata):
_cur.executemany(sql, jdata)
_cn.commit()
return
#From Oracle
fetch_dat = cur.fetchall()
#To take tuple off
fetch_dat = [json.loads(t[0]) for t in fetch_dat]
#To declare columns for the query
columns = tuple(fetch_dat[0].keys())
params = ['?' for _ in range(len(columns))]
#To declare value for the query
data = [tuple(e.values()) for e in fetch_dat]
#Composing the sql!
sql = 'INSERT INTO TSTTBL ({}) VALUES ({})'.format(','.join(columns), ','.join(params))
jsonINSERT(mscn, mscur, sql, data)
Many ways to go about this, but I think an example will help, you may need something like this:
db.execute("INSERT INTO `t1` VALUES (%s)", json_vals)
OR essentially,
db.execute(yourQueryHere, json_vals)
I've created a database with an autoincremented field:
filecur.execute("CREATE TABLE IF NOT EXISTS File_Data (_FID INTEGER PRIMARY KEY AUTOINCREMENT, UID INTEGER(1000000), FileName varchar2 (15), Date_ varchar2 (15))")
filecur.execute("insert into File_Data values (?, ?, ?)", (uid, filename, today))
However, when I try to insert values in this table, I get an error that says that I've supplied only 3 values in a table with 4 fields. Isn't autoincrement supposed to work on its own, or am I missing something? Any help would be appreciated, thanks!
You have to specify which columns you're inserting, e.g.:
filecur.execute("insert into File_Data(UID, FileName, Date_) values(?, ?, ?)",
(uid, filename, today))
Note that you could insert the autoincrement field explicitly, e.g.:
filecur.execute("insert into File_Data(FID_, UID, FileName, Date_) values (?, ?, ?, ?)",
(fid, uid, filename, today))
But normally you wouldn't be doing that.
i have a script like below :
import psutil
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
result = c.execute("SELECT * FROM clientinfo WHERE IP = ?", ("192.168.10.111",))
if (len(result.fetchall()) > 0):
for x in DISK :
c.execute("UPDATE disk SET Device = ?, 'Mount Point' = ?, 'fstyle' = ?, 'opts' = ?, 'total' = ?, 'used' = ?, 'free' = ?, 'percent' = ? WHERE Client_IP = ?", (DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7], "192.168.10.111"))
else :
for x in DISK :
c.execute("INSERT INTO disk('Client_IP', 'Device', 'Mount Point', 'fstyle', 'opts', 'total', 'used', 'free', 'percent') VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", ("192.168.10.111", DISK[x][0], DISK[x][1], DISK[x][2], DISK[x][3], DISK[x][4], DISK[x][5], DISK[x][6], DISK[x][7]))
conn.commit()
conn.close()
Script will check in db if there is any record about IP "192.168.10.111". If db already have record about IP "192.168.10.111", Script will update data from dict DISK to table disk of db.
If db don't have record about IP "192.168.10.111", Script will create record to insert **DISK"" to database.
INSERT command work well but the UPDATE command don't work like i want. After INSERT command run, in table disk i will have two record about disk C and disk D with same value of column Client_IP (192.168.10.111).
After UPDATE, two record of IP "192.168.10.111" get same value on every column ehich is very wrong. One record must be contain information about disk C and another record cotain disk D information.
How can i make the UPDATE work right ? length of dict DISK depend on how many mounted devices the computer has. So i need to use for loop to UPDATE but not static UPDATE.
Please tell me how to fix this,
Many thanks,
Francis
Your SELECT query column value is incorrect based on your UPDATE and INSERT queries later on. Instead of IP, should't it be CLIENT_IP? Also, you need to change your string query formatting. Remove the single quotes form your column names. Lastely, you can shorten your code by using a simple list comprehension and cursor.executemany:
import sqlite3
DISK = {'1': ['C:\\', 'C:\\', 'NTFS', 'rw,fixed', '75.0Gb', '54.0Gb', '20.0Gb', '72.2%'], '2': ['D:\\', 'D:\\', 'NTFS', 'rw,fixed', '399.0Gb', '208.0Gb', '191.0Gb', '52.2%']}
conn = sqlite3.connect("Test.db")
c = conn.cursor()
if not list(c.execute('SELECT * FROM lientinfo WHERE CLIENT_IP = ?', ("192.168.10.111",))):
c.executemany("INSERT INTO disk (Client_IP, Device, Mount Point, fstyle, opts, total, used, free, percent) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", [["192.168.10.111", *i] for i in DISK.values()])
else:
c.executemany('UPDATE disk SET Device = ?, Mount Point = ?, fstyle = ?, opts = ?, total = ?, used = ?, free = ?, percent = ? WHERE Client_IP = ?', [[*i, "192.168.10.111"] for i in DISK.values()])