Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
This application will read roster data in JSON format, parse the file, and then produce an SQLite database that contains a User, Course, and Member table and populate the tables from the data file.
This code is incomplete as I need to modify the program to store the role column in the Member table to complete the problem. And I cannot understand how to do it.
import json
import sqlite3
conn = sqlite3.connect('rosterdb.sqlite')
cur = conn.cursor()
cur.executescript('''
DROP TABLE IF EXISTS User;
DROP TABLE IF EXISTS Member;
DROP TABLE IF EXISTS Course;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'roster_data.json'
# [
# [ "Charley", "si110", 1 ],
# [ "Mea", "si110", 0 ],
str_data = open(fname).read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0];
title = entry[1];
print name, title
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id) VALUES ( ?, ? )''',
( user_id, course_id ) )
conn.commit()
Once the necessary changes are made to the program and it has been run successfully reading the given JSON data, run the following SQL command:
SELECT hex(User.name || Course.title || Member.role ) AS X FROM
User JOIN Member JOIN Course
ON User.id = Member.user_id AND Member.course_id = Course.id
ORDER BY X
Find the first row in the resulting record set and enter the long string that looks like 53656C696E613333.
On one hand, there are two changes you need to make in your code:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
and
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
but on the other hand, if you're having problems with your Coursera homework, you should really be posting to the class discussion forums about it (after reading what people have already posted about it there, in case that answers your questions) rather than to Stack Overflow. Considering that your question is about an assignment that was due some time ago rather than a current one, I don't feel so bad about talking about it here, but yeah, in the future use the discussion forums instead, and look at what I posted here to understand what is going on.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
example
conn= sqlite3.connect("test.db")
ID INT PRIMARY KEY NOT NULL,
SEX TEXT PRIMARY KEY NOT NULL,
after inserting the data example
1 male
2 male
3 male
4 male
5 female
6 male
7 male
I want the user(input) write the id then I print the SEX from database if ID valid
print("what id you want his sex")
x=input()#4 example
I want the sqlite code to take x(id) and from x he give me the sex(male)-
It looks like there are a few issues with 1. What you are trying to do, and 2. how you are trying to do it.
First thing to note, is that you cannot have multiple Primary Keys (PK) in 1 SQL table.
I'm not sure what you are trying to achieve with the below statement:
conn= sqlite3.connect("test.db") ID INT PRIMARY KEY NOT NULL, SEX TEXT PRIMARY KEY NOT NULL,
conn=sqlite3.connect("test.db") is used to connect to a database that already exists called test.db, if it does not exist it will create it.
If you are trying to create a new table with the cols you have provided, this could be done by:
conn=sqlite3.connect("test.db")
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXIST users(
ID int PRIMARY_KEY NOT NULL,
sex text)'''
conn.commit()
Once the table is initialised, if you are trying to insert:
c.execute('''INSERT INTO users (id, Sex)
SELECT
1,
'male''''
conn.commit()
Now, if you are trying to access that data, you can:
c.execute('''SELECT sex FROM users WHERE ID = 1'''
result = [dict(row) for row in c.fetchall()]
print(result)
Again, from your question it is very unclear what you are trying to do... but I thought i'd put some SQLITE3 basics down anyway.
EDIT based on clarification in comments:
The below script is an example of how you could create a database, create a table in that database, and then insert some dummy data - the function get_user_by_id executes the sql select statement. This function is then called in main where it uses the id input by the user as the parameter in the query:
import sqlite3
# Connect to db
conn = sqlite3.connect("test.db")
c = conn.cursor()
# Create table for storing data
c.execute('''CREATE TABLE IF NOT EXISTS users(
ID INT PRIMARY_KEY NOT NULL,
sex text)''')
# commit the changes to the database
conn.commit()
# inserting dummy data
c.execute('''INSERT INTO users (ID, sex)
SELECT
1,
"male"''')
conn.commit()
def get_user_by_id(id):
# conn = sqlite3.connect("test.db")
# c = conn.cursor()
c.execute("SELECT sex FROM users WHERE id = ?", (id,))
result = c.fetchone()
return result
def main():
id = int(input("Enter User ID: "))
sex = get_user_by_id(id)
print(sex[0])
if __name__ == '__main__':
main()
Just a note to the poster - please in future try to be clear and concise in your questions to help us help you.
Since the title is quite confusing, allow me to clarify. In this instance, I am trying to select all parentemails of year 10 students. However, the year grade of the students are stored in another table, making the select statement rather tricky.
This is my attempt so far, I hope it highlights the roadblock I am at.
conn = sqlite3.connect('test.db')
c = conn.cursor()
# Makes tables
c.execute(
"""
CREATE TABLE IF NOT EXISTS student (
year INTEGER,
code INTEGER,
PRIMARY KEY (code)
)
""")
c.execute(
"""
CREATE TABLE IF NOT EXISTS studentcontact (
contactcode INTEGER,
studentcode INTEGER,
parentemail TEXT,
PRIMARY KEY (contactcode),
FOREIGN KEY (studentcode) REFERENCES student(code)
)
""")
c.execute("""
INSERT OR REPLACE INTO student (code, year) VALUES
(501, 9),
(502, 10),
(503, 10)
""")
c.execute("""
INSERT OR REPLACE INTO studentcontact (contactcode, studentcode, parentemail) VALUES
(401, 501, "bobjones#email.com"),
(402, 502, "billwilliams#email.com"),
(403, 503, "sallydavidson#email.com")
""")
### -- QUERY HERE -- ##
# My attempt so far
query = """
SELECT code FROM student WHERE year ='10'
SELECT parentemail FROM studentcontact WHERE studentcode = *results from select statement above*
"""
One way to do this is:
SELECT parentemail
FROM studentcontact
WHERE studentcode IN (
SELECT code
FROM student
WHERE year='10'
)
If I understand correctly, you just want join:
SELECT sc.parentemail
FROM student s JOIN
studentcontact sc
ON s.code = sc.studentcode
WHERE s.year = 10
I am trying to connecting and inserting a data from json to sqlite. But it sucks me
I have tried to move and avoiding white spaces and copmleting with the databases
conn=sqlite3.connect('rosterdb.sqlite')
cur=conn.cursor()
cur.execute(
'''
DROP TABLE IF EXISTS USER;
DROP TABLE IF EXISTS Course;
DROP TABLE IF EXISTS Member;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
'''
)
fname=input('Enter file name')
if len(fname)<1:
fname=roster_data.json
str_data=open(fname).read()
json_data=json.loads(str_data)
for entry in json_data:
name=entry[0]
title=entry[1]
role=entry[2]
print(name,title,role)
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
conn.commit()
File "roster.py", line 51
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
^
IndentationError: unindent does not match any outer indentation level
You are probably mixing tabs and spaces
copy this to your editor:
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
And manually press space bar 4 times before cur.execute
And manually press space bar 8 times before VALUES
cur.execute('''INSERT OR IGNORE INTO Member(user_id,course_id,role)
VALUES(?,?,?)''',(user_id,course_id,role))
Please don't use tabs in combination with spaces
I would like to ask why below statement gave an disk I/O message if commit was executed inside for loop ? This would not give an error if commit was executed outside the for loop.
Error message:
Traceback (most recent call last):
File "D:\Dropbox\Public\EBOOK\Python\Learn\SQLITE_DB\ParsedJSON\ParsedJSON.py"
, line 71, in <module>
conn.commit()
sqlite3.OperationalError: disk I/O error
Code
for entry in json_data:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
conn.commit()
Complete code
import json
import sqlite3
conn = sqlite3.connect('rosterdb.sqlite')
cur = conn.cursor()
# Do some setup
cur.executescript('''
DROP TABLE IF EXISTS User;
DROP TABLE IF EXISTS Member;
DROP TABLE IF EXISTS Course;
CREATE TABLE User (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Course (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE
);
CREATE TABLE Member (
user_id INTEGER,
course_id INTEGER,
role INTEGER,
PRIMARY KEY (user_id, course_id)
)
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'roster_data.json'
str_data = open(fname).read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0];
title = entry[1];
role = entry[2];
print name, title, role
cur.execute('''INSERT OR IGNORE INTO User (name)
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
conn.commit()
I was curious how much impact the commit I/O actually had, so I ran a few tests with some mocked-up JSON matching the following format:
{
"folks": [
{
"name": "Foghorn Leghorn",
"title": "Principal",
"role": "Administration"
}
]
}
My results were enlightening. These are averaged times over 10 runs of the test:
with commit() outside loop:
8.960046267508 seconds for 50 Records
and
with commit() outside loop:
0.3031771421432 for 50 Records
I ran the test with ~100,000 records.
With the .commit() outside the loop:
15.2660858631 seconds for 102150 records
With the .commit() inside the loop:
23.81681369933333 MINUTES for 102150 records
Understand that with the .commit() outside the loop, this example also involves 102150 writes to disk. When the commit() is outside, it defers writing to your database until all of your operations are complete and buffered. Inside, it immediately writes to the database file after each iteration completes.
In addition, between each commit(), sqlite is creating a journal file (in this case, 'rosterdb.sqlite-journal'), so you're also creating and deleting this additional file for every commit(), which multiplies the impact on your hardware and your performance. (If you're curious, you can watch this file appear and disappear each iteration in whichever directory your database exists.)
So putting the commit() outside is way, way faster and easier on your hardware. As to why it is returning Disk I/O errors, I'd say it's tied to the speed and frequency of those commits().
I am a bit confused about the output I am getting in the command window after running my code. I am have reviewed each section a few different times, and I am not sure what is going wrong.
Why is the dictionary count printing, but the rest of the print statements are not showing up? I also checked the table in the SQLite file, and none of my data is in the tables, just headings.
Have I overlooked something? There is no error showing up, but it is not giving the output or data in the tables, even with the insert, select, and fetchone statements.
Thank you in advance for your comments!
Here is what I have been playing around with:
import xml.etree.ElementTree as ET
import sqlite3
conn = sqlite3.connect('trackdb.sqlite')
cur = conn.cursor()
cur.excutescript('''
DROP TABLE IF EXISTS Artist;
DROP TABLE IF EXISTS Album;
DROP TABLE IF EXISTS Track;
DROP TABLE IF EXISTS Genre;
CREATE TABLE Artist (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Genre (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
CREATE TABLE Album (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
artist_id INTEGER,
title TEXT UNIQUE
);
CREATE TABLE Track (
id INTEGER NOT NULL PRIMARY KEY
AUTOINCREMENT UNIQUE,
title TEXT UNIQUE,
album_id INTEGER,
genre_id INTEGER,
len INTEGER, rating INTEGER, count INTEGER
);
''')
fname = raw_input('Enter file name: ')
if ( len(fname) < 1 ) : fname = 'Library.xml'
# <key>Track ID</key><integer>369</integer>
# <key>Name</key><string>Another One Bites The Dust</string>
# <key>Artist</key><string>Queen</string>
def lookup(d, key):
found = False
for child in d:
if found : return child.text
if child.tag == 'key' and child.text == key :
found = True
return None
stuff = ET.parse(fname)
all = stuff.findall('dict/dict/dict')
print 'Dict count:', len(all)
for entry in all:
if ( lookup(entry, 'Track ID') is None ) : continue
name = lookup(entry, 'Name')
artist = lookup(entry, 'Artist')
album = lookup(entry, 'Album')
count = lookup(entry, 'Play Count')
rating = lookup(entry, 'Rating')
length = lookup(entry, 'Total Time')
genre = lookup(entry, 'Genre')
if name is None or artist is None or album or genre is None : continue
print name, artist, album, genre, count, rating, length
cur.execute('''INSERT OR IGNORE INTO Artist (name)
VALUES ( ? )''', ( artist, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (artist, ))
artist_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Album (title, artist_id)
VALUES ( ?, ? )''', ( album, artist_id ) )
cur.execute('SELECT id FROM Album WHERE title = ? ', (album, ))
album_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Genre (name)
VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Genre WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Track
(title, album_id, genre_id, len, rating, count)
VALUES ( ?, ?, ?, ?, ?, ? )''',
( name, album_id, genre_id, length, rating, count ) )
conn.commit()
Am I missing a SELECT statement? I tried to apply a JOIN statement to have the desired table with the track, genre, album and artists names exposed, but I can not really figure how to incorporate it. I was getting an error when I tried genre_id = cur.fetchone()[0] If anyone can give me some insight, I would be most grateful.
Here is the link to the XML data:
http://www.pythonlearn.com/code/tracks.zip
It basically shows dictionaries within dictionaries of the different key/value pairs for the track information.
Thanks again.
I find four errors that need to be corrected to make this program run.
1. In the fifth line of code:
cur.excutescript('''
should be:
cur.executescript('''
2. In your CREATE TABLE for Track you need to add the artist_id column:
CREATE TABLE Track (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
title TEXT UNIQUE,
album_id INTEGER,
artist_id INTEGER,
genre_id INTEGER);
artist_id will also need to be added to your INSERT OR REPLACE statement for the Track table:
cur.execute('''INSERT OR REPLACE INTO Track
(title, album_id, artist_id, genre_id)
VALUES ( ?, ?, ?, ?)''',
( name, album_id, artist_id, genre_id) )
You have references to count, rating and length, but they aren't part of your tables, so you can remove those from your code. These are in your lookup statements, your print statement, and your INSERT OR REPLACE INTO for the Tracks table.
4. In this line:
if name is None or artist is None or album or genre is None : continue
You're missing is None after album:
if name is None or artist is None or album is None or genre is None: continue
With those changes, the code works, displays the info from the print statement, and builds your db correctly.
Consider the following adjustment that does the following:
Fixes your if line inside loop, even suggests a list comprehension replacement commented out.
Avoids the multiple SELECT statement fetches as it revises both the Album and Track table inserts with an INSERT INTO...SELECT using implicit joins (i.e., only WHERE clauses)
Adds conn.commit() to each execute statement since the preceding append queries are needed for Album and Track table and must be committed to database beforehand and not at the end.
Updated Code
stuff = ET.parse(fname)
all = stuff.findall('dict/dict/dict')
print 'Dict count:', len(all)
for entry in all:
if ( lookup(entry, 'Track ID') is None ) : continue
name = lookup(entry, 'Name')
artist = lookup(entry, 'Artist')
album = lookup(entry, 'Album')
count = int(lookup(entry, 'Play Count')) # NUMERIC CONVERSION
rating = float(lookup(entry, 'Rating')) # NUMERIC CONVERSION
length = float(lookup(entry, 'Total Time')) # NUMERIC CONVERSION
genre = lookup(entry, 'Genre')
if (name is None) or (artist is None) or (album is None) or (genre is None) : continue
#if (i for i in (name, artist, album, genre) if i is None) : continue
print name, artist, album, genre, count, rating, length
cur.execute('''INSERT OR IGNORE INTO Artist (name)
VALUES ( ? )''', ( artist, ) )
conn.commit()
cur.execute('''INSERT OR IGNORE INTO Album (title, artist_id)
SELECT ? As title, Artist.artist_id
FROM Artist
WHERE Artist.name = ?''',
( album, artist ) )
conn.commit()
cur.execute('''INSERT OR IGNORE INTO Genre (name)
VALUES ( ? )''', ( genre, ) )
conn.commit()
cur.execute('''INSERT OR REPLACE INTO Track (title, album_id, genre_id,
len, rating, count)
SELECT ? As name, Album.album_id, Genre.genre_id,
? As len, ? As rating, ? As count
FROM Album, Genre
WHERE Album.title = ? AND Genre.name = ?;''',
( name, length, rating, count, album, genre ))
conn.commit()