Let's say i have two simple .csv files which i want to import into a SQLite database:
id
state
1
State1
2
State2
3
State3
id
district
values
state_fk
1
District1
123
1
2
District2
456
2
3
District3
789
3
I receive an updated version of the district file every 2 weeks. I simply read the csv files and import/update them via df.to_sql('table', con, if_exists='replace').
Is there a way in pandas to set a PK-FK relation between states.id and districts.state_fk?
I would like to regulary update the district table with new values and don't set the relation manually again after each update.
I don't think it's possible to create foreign keys with pandas. That is not what the module is for.
Instead of replacing the database table each time could you not first empty/truncate it using the sqlite3 module and then import your data using the if_exists='append' option ? Would look something like that :
import sqlite3
con = sqlite3.connect('example.db')
cur = con.cursor()
cur.execute('''TRUNCATE TABLE table''')
con.commit()
con.close()
df.to_sql('table', con, if_exists='append')
I imported a csv file into Python and tried using SQLLite. But when I created the table, the columns and values were changed. For example, in the overall column the values "asin" appeared. The reviewerID column received the values "overall" and so on.
How to fix it?
class csvrd(object):
def csvFile(self):
self.readFile('reviews.csv')
def readFile(self, filename):
conn = sqlite3.connect('amazonReviews.db')
cur = conn.cursor()
cur.execute("""CREATE TABLE IF NOT EXISTS amazonReviews(reviewerID varchar, asin INT,reviewerName varchar,reviewText varchar, overall INT,summary varchar,unixReviewTime INTEGER,reviewTime INTEGER,helpful INT,total INT)""")
filename.encode('utf-8')
print("Amazon Reviews table executed")
with open(filename) as f:
reader = csv.reader(f)
for field in reader:
cur.execute("INSERT INTO amazonReviews VALUES (?,?,?,?,?,?,?,?,?,?);", field)
print("CSV Loaded into SQLite")
conn.commit()
conn.close()
c = csvrd().csvFile()
con = sqlite3.connect('amazonReviews.db')
pd.read_sql_query("SELECT * FROM amazonReviews LIMIT 5", con)
EXPECTED:
reviewerID asin reviewerName reviewText overall summary unixReviewTime reviewTime helpful total
0 A1EE2E3N7PW666 B000GFDAUG Aaron L. Allen "Orgazmo1009" CA Lewsi' review should be removed. he's revie... 5 Stupid 1202256000 02 6, 2008 0 0
1 AGZ8SM1BGK3CK B000GFDAUG Mind's Clay I truly love the humor of South Park. It's soc... 5 "More Moist Than Should Be" Humor 1198195200 12 21, 2007 1 1
ACTUAL:
asin overall reviewText reviewTime reviewerID reviewerName summary unixReviewTime helpful total
0 A1EE2E3N7PW666 B000GFDAUG Aaron L. Allen "Orgazmo1009" CA Lewsi' review should be removed. he's revie... 5 Stupid 1202256000 02 6, 2008 0 0
1 AGZ8SM1BGK3CK B000GFDAUG Mind's Clay I truly love the humor of South Park. It's soc... 5 "More Moist Than Should Be" Humor 1198195200 12 21, 2007 1 1[enter image description here][1]
This would happen if the table already exists with the columns in the "ACTUAL" result order. Table will not be created because of "IF NOT EXISTS" in the "CREATE" sql. The data will be loaded without "type" complaint because of manifest typing.
I am learning python and trying to replicate what online tutorials do. I am trying to create a python desktop app where data is store in Postgresql. code is added below,
`cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY, title text, author text, year integer, isbn integer)")`
problem is with (id INTEGER PRIMARY KEY), when i execute the code its showing none in place of 1st index. i want to show numbers.
please help
this is for Python 3.7.3, psycopg2==2.8.3,
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,
title text, author text, year integer, isbn integer)")
conn.commit()
conn.close()
the result I am expecting is auto increment of numbers in 1st index where as presently it shows NONE.
below is the present and expected result again.
none title auther year isbn
01 title auther year isbn
Trying to use a Cursor execute will not work for a CREATE statement and hence the NONE. See below for an example.
Re Indexes :-
There will be no specific index as column_name INTEGER PRIMARY KEY is special in that it defines the column as an alias of the rowid column which is a special intrinsic index using the underlying B-tree storage engine.
When a row is inserted then if no value is specified for the column (e.g. INSERT INTO book (title,author, year, isbn) VALUES ('book1','The Author','1999','1234567890') then id will be 1 and typically (but not certainly) the next row inserted will have an id of 2 and so on.
If after adding some rows you use SELECT * FROM book, then the rows will be ordered according to the id as no other index is specified/used.
Perhaps have a look at Rowid Tables.
Example
Perhaps consider the following example :-
DROP TABLE IF EXISTS book;
CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY, title text, author text, year integer, isbn integer);
INSERT INTO book (title,author, year, isbn) VALUES
('book1','The Author','1999','1234567890'),
('book2','Author 2','1899','2234567890'),
('book3','Author 3','1799','3234567890')
;
INSERT INTO book VALUES (100,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID
INSERT INTO book (title,author, year, isbn) VALUES
('book11','Author 11','1999','1234567890'),
('book12','Author 12','1899','2234567890'),
('book13','Author 13','1799','3234567890')
;
INSERT INTO book VALUES (10,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID
SELECT * FROM book;
This :-
DROPs the book table (to make it easily re-runable)
CREATEs the book table.
INSERTs 3 books with the id not specified (typpical)
INSERTs a fourth book but with a specific id of 100
INSERTs another 3 books (not that these will be 101-103 as 100 is the highest id before the inserts)
INSERTs a last row BUT with a specific id of 10.
SELECTs all rows with all columns from the book table ordered, as no ORDER BY has been specified, according to the hidden index based upon the id. NOTE although id 10 was the last inserted it is the 4th row.
Result
In Python :-
conn = sqlite3.connect("books.db")
conn.execute("DROP TABLE IF EXISTS book")
conn.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,title text, author text, year integer, isbn integer)")
conn.execute("INSERT INTO book (title,author, year, isbn) "
"VALUES('book1','The Author','1999','1234567890'), "
"('book2','Author 2','1899','2234567890'), "
"('book3','Author 3','1799','3234567890');")
conn.execute("INSERT INTO book VALUES (100,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID")
conn.execute("INSERT INTO book (title,author, year, isbn) VALUES ('book11','Author 11','1999','1234567890'),('book12','Author 12','1899','2234567890'),('book13','Author 13','1799','3234567890');")
conn.execute("INSERT INTO book VALUES (10,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID")
cur = conn.cursor()
cur.execute("SELECT * FROM book")
for each in cur:
print("{0:<20} {1:<20} {2:<20} {3:<20} {4:<20}".format(each[0],each[1],each[2],each[3],each[4]))
conn.commit()
conn.close()
The results in :-
1 book1 The Author 1999 1234567890
2 book2 Author 2 1899 2234567890
3 book3 Author 3 1799 3234567890
10 book10 Author 10 1999 4234567890
100 book10 Author 10 1999 4234567890
101 book11 Author 11 1999 1234567890
102 book12 Author 12 1899 2234567890
103 book13 Author 13 1799 3234567890
because you say I am inserting the data manually then instead of
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,
title text, author text, year integer, isbn integer)")
conn.commit()
conn.close()
Try to use
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("SELECT * FROM books")
for row in cur:
print(row[0],row[1],row[2],row[3],row[4])
conn.commit()
conn.close()
I am very new to Mysql and dumping a file in db using python. I'm having 2 tables
The file format is:
id name sports
1 john baseball
2 mary Football
like student & Sports
Student table
id name
1 John
2 Mary
here id is primary key
& in sports table
stu_id sports_title
1 Baseball
2 Football
and here stu_id is foreign key reference with student table
and my problem is
query="insert into sports (stu_id,name)VALUES (%d,%s)"
("select id from student where id=%d,%s")
#words[0]=1 ,words[2]=Baseball
args=(words[0],words[2])
cursor.execute(query,args)
upon executing this code, I'm facing
"Not all parameters were used in the SQL statement")
ProgrammingError: Not all parameters were used in the SQL statement
You can't use both VALUES and SELECT as the source of the data in INSERT. You use VALUES if the data is literals, you use SELECT if you're getting it from another table. If it's a mix of both, you use SELECT and put the literals into the SELECT list.
query = """
INSERT INTO sports (stu_id, sports_title)
SELECT id, %s
FROM student
WHERE name = %s
"""
args = (words[2], words[1])
cursor.execute(args)
Or, since the file contains the student ID, you don't need SELECT at all.
query = "INSERT INTO sports(stu_id, sports_title) VALUES (%d, %s)"
args = (words[0], words[1])
cursor.execute(args)
I'm using Python and SQLite to manipulate a database.
I have a SQLite table Movies in database Data that looks like this:
| ID | Country
+----------------+-------------
| 1 | USA, Germany, Mexico
| 2 | Brazil, Peru
| 3 | Peru
I have a table Countries in the same database that looks like this
| ID | Country
+----------------+-------------
| 1 | USA
| 1 | Germany
| 1 | Mexico
| 2 | Brazil
| 2 | Peru
| 3 | Peru
I want to insert from database Data all movies from Peru into a new database PeruData that looks like this
| ID | Country
+----------------+-------------
| 2 | Peru
| 3 | Peru
I'm new to SQL and having trouble programming the right query.
Here's my attempt:
con = sqlite3.connect("PeruData.db")
cur = con.cursor()
cur.execute("CREATE TABLE Movies (ID, Country);")
cur.execute("ATTACH DATABASE 'Data.db' AS other;")
cur.execute("\
INSERT INTO Movies \
(ID, Country) \
SELECT ID, Country
FROM other.Movies CROSS JOIN other.Countries\
WHERE other.Movies.ID = other.Countries.ID AND other.Countries.Country = 'Peru'\
con.commit()
con.close()
Clearly, I'm doing something wrong because I get the error
sqlite3.OperationalError: no such table: other.Countries
Here's a workaround which successfully got the result you wanted.
Instead of having to write con = sqlite3.connect("data.db") and then having to write con.commit() and con.close(), you can shorten your code to written like this:
with sqlite3.connect("Data.db") as connection:
c = connection.cursor()
This way you won't have to commit the changes and close each time you're working with a database. Just a nifty shortcut I learned. Now onto your code...
Personally, I'm unfamiliar with the SQL statement ATTACH DATABASE. I would incorporate your new database at the end of your program instead that way you can avoid any conflicts that you aren't knowledgeable of handling (such as your OperationalError given). So first I would begin to get the desired result and then insert that into your new table. Your third execution statement can be rewritten like so:
c.execute("""SELECT DISTINCT Movies.ID, Countries.Country
FROM Movies
CROSS JOIN Countries
WHERE Movies.ID = Countries.ID AND Countries.Country = 'Peru'
""")
This does the job, but you need to use fetchall() to return your result set in a list of tuples which can then be inserted into your new table. So you'd type this:
rows = c.fetchall()
Now you can open a new connection by creating the "PeruData.db" database, creating the table, and inserting the values.
with sqlite3.connect("PeruData.db") as connection:
c = connection.cursor()
c.execute("CREATE TABLE Movies (ID INT, Country TEXT)")
c.executemany("INSERT INTO Movies VALUES(?, ?)", rows)
That's it.
Hope I was able to answer your question!
The current error is probably caused by a typo or other minor problem. I could create the databases described here and successfully do the insert after fixing minor errors: a missing backslash at an end of line and adding qualifiers for the selected columns.
But I also advise your to use aliases for the tables in multi-table selects. The code that works in my test is:
cur.execute("\
INSERT INTO Movies \
(ID, Country) \
SELECT m.ID, c.Country\
FROM other.Movies m CROSS JOIN other.Countries c \
WHERE m.ID = c.ID AND c.Country = 'Peru'")