I am trying to connecting and inserting a data from json to sqlite. But it sucks me
I have tried to move and avoiding white spaces and copmleting with the databases
conn=sqlite3.connect('rosterdb.sqlite')
cur=conn.cursor()
cur.execute(
'''
DROP TABLE IF EXISTS USER;
DROP TABLE IF EXISTS Course;
DROP TABLE IF EXISTS Member;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
'''
)
fname=input('Enter file name')
if len(fname)<1:
fname=roster_data.json
str_data=open(fname).read()
json_data=json.loads(str_data)
for entry in json_data:
name=entry[0]
title=entry[1]
role=entry[2]
print(name,title,role)
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
conn.commit()
File "roster.py", line 51
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
^
IndentationError: unindent does not match any outer indentation level
You are probably mixing tabs and spaces
copy this to your editor:
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
And manually press space bar 4 times before cur.execute
And manually press space bar 8 times before VALUES
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
Please don't use tabs in combination with spaces
Related
i have an sql insert query that take values from user input and also insert the ID from another table as foreign key. for this is write the below query but it seems not working.
Status_type table
CREATE TABLE status_type (
ID int(5) NOT NULL,
status varchar(50) NOT NULL
);
info table
CREATE TABLE info (
ID int(11) NOT NULL,
name varchar(50), NULL
nickname varchar(50), NULL
mother_name varchar(50), NULL
birthdate date, NULL
status_type int <==this must be the foreign key for the status_type table
create_date date
);
for the user he has a dropdownlist that retrieve the value from the status_type table in order to select the value that he want to insert into the new record in the info table
where as the info table take int Type because i want to store the ID of the status_type and not the value
code:
query = '''
INSERT INTO info (ID,name,nickname,mother_name,birthdate,t1.status_type,created_date)
VALUES(?,?,?,?,?,?,?)
select t2.ID
from info as t1
INNER JOIN status_type as t2
ON t2.ID = t1.status_type
'''
args = (ID,name,nickname,mother_name,db,status_type,current_date)
cursor = con.cursor()
cursor.execute(query,args)
con.commit()
st.success('Record added Successfully')
the status_type field take an INT type (the ID of the value from another table ).
So when the user insert it insert the value.
What i need is to convert this value into its corresponding ID and store the ID
based on the answer of #Mostafa NZ I modified my query and it becomes like below :
query = '''
INSERT INTO info (ID,name,nickname,mother_name,birthdate,status_type,created_date)
VALUES(?,?,?,?,?,(select status_type.ID
from status_type
where status = ?),?)
'''
args = (ID,name,nickname,mother_name,db,status_type,current_date)
cursor = con.cursor()
cursor.execute(query,args)
con.commit()
st.success('Record added Successfully')
When creating a record, you can do one of these ways.
Receive as input from the user
Specify a default value for the field
INSERT INTO (...) VALUES (? ,? ,1 ,? ,?)
Use a select in the INSERT
INSERT INTO (...) VALUES (? ,? ,(SELECT TOP 1 ID FROM status_type ODER BY ID) ,? ,?)
When INSERT data, you can only enter the names of the destination table fields. t1.status_type is wrong in the following line
INSERT INTO info (ID,name,nickname,mother_name,birthdate,t1.status_type,created_date)
Since the title is quite confusing, allow me to clarify. In this instance, I am trying to select all parentemails of year 10 students. However, the year grade of the students are stored in another table, making the select statement rather tricky.
This is my attempt so far, I hope it highlights the roadblock I am at.
conn = sqlite3.connect('test.db')
c = conn.cursor()
# Makes tables
c.execute(
"""
CREATE TABLE IF NOT EXISTS student (
year INTEGER,
code INTEGER,
PRIMARY KEY (code)
)
""")
c.execute(
"""
CREATE TABLE IF NOT EXISTS studentcontact (
contactcode INTEGER,
studentcode INTEGER,
parentemail TEXT,
PRIMARY KEY (contactcode),
FOREIGN KEY (studentcode) REFERENCES student(code)
)
""")
c.execute("""
INSERT OR REPLACE INTO student (code, year) VALUES
(501, 9),
(502, 10),
(503, 10)
""")
c.execute("""
INSERT OR REPLACE INTO studentcontact (contactcode, studentcode, parentemail) VALUES
(401, 501, "bobjones#email.com"),
(402, 502, "billwilliams#email.com"),
(403, 503, "sallydavidson#email.com")
""")
### -- QUERY HERE -- ##
# My attempt so far
query = """
SELECT code FROM student WHERE year ='10'
SELECT parentemail FROM studentcontact WHERE studentcode = *results from select statement above*
"""
One way to do this is:
SELECT parentemail
FROM studentcontact
WHERE studentcode IN (
SELECT code
FROM student
WHERE year='10'
)
If I understand correctly, you just want join:
SELECT sc.parentemail
FROM student s JOIN
studentcontact sc
ON s.code = sc.studentcode
WHERE s.year = 10
I would like to ask why below statement gave an disk I/O message if commit was executed inside for loop ? This would not give an error if commit was executed outside the for loop.
Error message:
Traceback (most recent call last):
File "D:\Dropbox\Public\EBOOK\Python\Learn\SQLITE_DB\ParsedJSON\ParsedJSON.py"
, line 71, in <module>
conn.commit()
sqlite3.OperationalError: disk I/O error
Code
for entry in json_data:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
conn.commit()
Complete code
import json
import sqlite3
conn = sqlite3.connect('rosterdb.sqlite')
cur = conn.cursor()
# Do some setup
cur.executescript('''
DROP TABLE IF EXISTS User;
DROP TABLE IF EXISTS Member;
DROP TABLE IF EXISTS Course;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'roster_data.json'
str_data = open(fname).read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
conn.commit()
I was curious how much impact the commit I/O actually had, so I ran a few tests with some mocked-up JSON matching the following format:
{
"folks": [
{
"name": "Foghorn Leghorn",
"title": "Principal",
"role": "Administration"
}
]
}
My results were enlightening. These are averaged times over 10 runs of the test:
with commit() outside loop:
8.960046267508 seconds for 50 Records
and
with commit() outside loop:
0.3031771421432 for 50 Records
I ran the test with ~100,000 records.
With the .commit() outside the loop:
15.2660858631 seconds for 102150 records
With the .commit() inside the loop:
23.81681369933333 MINUTES for 102150 records
Understand that with the .commit() outside the loop, this example also involves 102150 writes to disk. When the commit() is outside, it defers writing to your database until all of your operations are complete and buffered. Inside, it immediately writes to the database file after each iteration completes.
In addition, between each commit(), sqlite is creating a journal file (in this case, 'rosterdb.sqlite-journal'), so you're also creating and deleting this additional file for every commit(), which multiplies the impact on your hardware and your performance. (If you're curious, you can watch this file appear and disappear each iteration in whichever directory your database exists.)
So putting the commit() outside is way, way faster and easier on your hardware. As to why it is returning Disk I/O errors, I'd say it's tied to the speed and frequency of those commits().
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
This application will read roster data in JSON format, parse the file, and then produce an SQLite database that contains a User, Course, and Member table and populate the tables from the data file.
This code is incomplete as I need to modify the program to store the role column in the Member table to complete the problem. And I cannot understand how to do it.
import json
import sqlite3
conn = sqlite3.connect('rosterdb.sqlite')
cur = conn.cursor()
cur.executescript('''
DROP TABLE IF EXISTS User;
DROP TABLE IF EXISTS Member;
DROP TABLE IF EXISTS Course;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'roster_data.json'
# [
# [ "Charley", "si110", 1 ],
# [ "Mea", "si110", 0 ],
str_data = open(fname).read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0];
title = entry[1];
print name, title
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id) VALUES ( ?, ? )''',
( user_id, course_id ) )
conn.commit()
Once the necessary changes are made to the program and it has been run successfully reading the given JSON data, run the following SQL command:
SELECT hex(User.name || Course.title || Member.role ) AS X FROM
User JOIN Member JOIN Course
ON User.id = Member.user_id AND Member.course_id = Course.id
ORDER BY X
Find the first row in the resulting record set and enter the long string that looks like 53656C696E613333.
On one hand, there are two changes you need to make in your code:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
and
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
but on the other hand, if you're having problems with your Coursera homework, you should really be posting to the class discussion forums about it (after reading what people have already posted about it there, in case that answers your questions) rather than to Stack Overflow. Considering that your question is about an assignment that was due some time ago rather than a current one, I don't feel so bad about talking about it here, but yeah, in the future use the discussion forums instead, and look at what I posted here to understand what is going on.
I am a bit confused about the output I am getting in the command window after running my code. I am have reviewed each section a few different times, and I am not sure what is going wrong.
Why is the dictionary count printing, but the rest of the print statements are not showing up? I also checked the table in the SQLite file, and none of my data is in the tables, just headings.
Have I overlooked something? There is no error showing up, but it is not giving the output or data in the tables, even with the insert, select, and fetchone statements.
Thank you in advance for your comments!
Here is what I have been playing around with:
import xml.etree.ElementTree as ET
import sqlite3
conn = sqlite3.connect('trackdb.sqlite')
cur = conn.cursor()
cur.excutescript('''
DROP TABLE IF EXISTS Artist;
DROP TABLE IF EXISTS Album;
DROP TABLE IF EXISTS Track;
DROP TABLE IF EXISTS Genre;
CREATE TABLE Artist (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Genre (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Album (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
artist_id INTEGER,
title TEXT UNIQUE
);
CREATE TABLE Track (
id INTEGER NOT NULL PRIMARY KEY
AUTOINCREMENT UNIQUE,
title TEXT UNIQUE,
album_id INTEGER,
genre_id INTEGER,
len INTEGER, rating INTEGER, count INTEGER
);
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'Library.xml'
# <key>Track ID</key><integer>369</integer>
# <key>Name</key><string>Another One Bites The Dust</string>
# <key>Artist</key><string>Queen</string>
def lookup(d, key):
found = False
for child in d:
if found : return child.text
if child.tag == 'key' and child.text == key :
found = True
return None
stuff = ET.parse(fname)
all = stuff.findall('dict/dict/dict')
print 'Dict count:', len(all)
for entry in all:
if ( lookup(entry, 'Track ID') is None ) : continue
name = lookup(entry, 'Name')
artist = lookup(entry, 'Artist')
album = lookup(entry, 'Album')
count = lookup(entry, 'Play Count')
rating = lookup(entry, 'Rating')
length = lookup(entry, 'Total Time')
genre = lookup(entry, 'Genre')
if name is None or artist is None or album or genre is None : continue
print name, artist, album, genre, count, rating, length
cur.execute('''INSERT OR IGNORE INTO Artist (name)
VALUES ( ? )''', ( artist, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (artist, ))
artist_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Album (title, artist_id)
VALUES ( ?, ? )''', ( album, artist_id ) )
cur.execute('SELECT id FROM Album WHERE title = ? ', (album, ))
album_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Genre (name)
VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Genre WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Track
(title, album_id, genre_id, len, rating, count)
VALUES ( ?, ?, ?, ?, ?, ? )''',
( name, album_id, genre_id, length, rating, count ) )
conn.commit()
Am I missing a SELECT statement? I tried to apply a JOIN statement to have the desired table with the track, genre, album and artists names exposed, but I can not really figure how to incorporate it. I was getting an error when I tried genre_id = cur.fetchone()[0] If anyone can give me some insight, I would be most grateful.
Here is the link to the XML data:
http://www.pythonlearn.com/code/tracks.zip
It basically shows dictionaries within dictionaries of the different key/value pairs for the track information.
Thanks again.
I find four errors that need to be corrected to make this program run.
1. In the fifth line of code:
cur.excutescript('''
should be:
cur.executescript('''
2. In your CREATE TABLE for Track you need to add the artist_id column:
CREATE TABLE Track (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE,
album_id INTEGER,
artist_id INTEGER,
genre_id INTEGER);
artist_id will also need to be added to your INSERT OR REPLACE statement for the Track table:
cur.execute('''INSERT OR REPLACE INTO Track
(title, album_id, artist_id, genre_id)
VALUES ( ?, ?, ?, ?)''',
( name, album_id, artist_id, genre_id) )
You have references to count, rating and length, but they aren't part of your tables, so you can remove those from your code. These are in your lookup statements, your print statement, and your INSERT OR REPLACE INTO for the Tracks table.
4. In this line:
if name is None or artist is None or album or genre is None : continue
You're missing is None after album:
if name is None or artist is None or album is None or genre is None: continue
With those changes, the code works, displays the info from the print statement, and builds your db correctly.
Consider the following adjustment that does the following:
Fixes your if line inside loop, even suggests a list comprehension replacement commented out.
Avoids the multiple SELECT statement fetches as it revises both the Album and Track table inserts with an INSERT INTO...SELECT using implicit joins (i.e., only WHERE clauses)
Adds conn.commit() to each execute statement since the preceding append queries are needed for Album and Track table and must be committed to database beforehand and not at the end.
Updated Code
stuff = ET.parse(fname)
all = stuff.findall('dict/dict/dict')
print 'Dict count:', len(all)
for entry in all:
if ( lookup(entry, 'Track ID') is None ) : continue
name = lookup(entry, 'Name')
artist = lookup(entry, 'Artist')
album = lookup(entry, 'Album')
count = int(lookup(entry, 'Play Count')) # NUMERIC CONVERSION
rating = float(lookup(entry, 'Rating')) # NUMERIC CONVERSION
length = float(lookup(entry, 'Total Time')) # NUMERIC CONVERSION
genre = lookup(entry, 'Genre')
if (name is None) or (artist is None) or (album is None) or (genre is None) : continue
#if (i for i in (name, artist, album, genre) if i is None) : continue
print name, artist, album, genre, count, rating, length
cur.execute('''INSERT OR IGNORE INTO Artist (name)
VALUES ( ? )''', ( artist, ) )
conn.commit()
cur.execute('''INSERT OR IGNORE INTO Album (title, artist_id)
SELECT ? As title, Artist.artist_id
FROM Artist
WHERE Artist.name = ?''',
( album, artist ) )
conn.commit()
cur.execute('''INSERT OR IGNORE INTO Genre (name)
VALUES ( ? )''', ( genre, ) )
conn.commit()
cur.execute('''INSERT OR REPLACE INTO Track (title, album_id, genre_id,
len, rating, count)
SELECT ? As name, Album.album_id, Genre.genre_id,
? As len, ? As rating, ? As count
FROM Album, Genre
WHERE Album.title = ? AND Genre.name = ?;''',
( name, length, rating, count, album, genre ))
conn.commit()