Develop Reference

Python: Save the previously scraped data in database (already created)

I have done the scraping with Selenium correctly and the results are printed perfectly. I don't know how to save them in a database I have already created the tables and fields with Sqlite. In general I know how to save the data in a database, but I don't know how to save the scraping data. What I don't know is how to write the code to save the scraping data.
Now I would just like to add the scraped data into the database and save. The database table name is TableExample1. The fields are: ID, Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB.
I have 2 problems
The text that is saved in the database like this: [<selenium.webdriver.firefox.webelement.FirefoxWebElement (session = "226bc9f3
Only one (1) row is added to the database. Instead, dozens or hundreds of them should be added (each like this: Product_Name, Product_Description, Vendor, Price), depending on the scraped data.
The Python code with the data printed by the scraping is as follows:
#Name of the scraped data
Product_Name = (driver.find_element_by_class_name ("tablet-desktop-only"). Text)
Product_Description = (driver.find_element_by_class_name ("h-text-left"). Text)
Vendor = (driver.find_element_by_class_name ("in-vendor"). Text)
Price = (driver.find_element_by_class_name ("h-text-center"). Text)
for seq in Product_Name + Product_Description + Vendor + Price:
print(seq.text)
UPDATE N.1
Current code, following "Jeremy Kahan"'s answers to this question
and in THIS (it is different, I asked to print in the console all the results of the scraping and not just 1)
This is the current most stable and functioning code. Scraping works fine, but that Num_Groups and for i in range (Num_Groups) you suggested in the other question, print only one group, not all.
I still have the same 2 problems
The text that is saved in the database like this: [<selenium.webdriver.firefox.webelement.FirefoxWebElement (session = "226bc9f3
Only one (1) row is added to the database. Instead, dozens or hundreds of them should be added (each like this: Product_Name, Product_Description, Vendor, Price), depending on the scraped data.
import sqlite3
from datetime import datetime
#SCRAPING
Product_Name=driver.find_elements_by_class_name("tablet-desktop-only")
Product_Description=driver.find_elements_by_class_name("h-text-left")
Vendor=driver.find_elements_by_class_name("in-match")
Price=driver.find_elements_by_class_name("h-text-center")
# How do I print the other data always with the same html name?
# This is one row data. This is the code you wrote me in the
# other question. Print only one group. What to write to print
# all groups?
Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))
for i in range(Num_Groups):
print(Product_Name[i].text)
print(Product_Description[i].text)
print(Vendor[i].text)
print(Price[i].text)
#INSERT DATA IN DATABASE
con = sqlite3.connect('/home/mypc/Desktop/aaaaa/Database.db')
cursor = con.cursor()
ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID},'{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
sqlite_insert_query = 'INSERT INTO TableExample1 (ID, Product_Name, Product_Description, Vendor, Price)' + Values
count = cursor.execute(sqlite_insert_query)
con.commit()
print("Record inserted successfully ", cursor.rowcount)
cursor.close()
2° UPLOAD (final?)
PROBLEM: found 12 groups, Record inserted successfully 1, Added a total of 1 records How do I get 12 staves inserted in the database?
Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))
records_added = 0
for i in range(Num_Groups):
print(Product_Name[i].text)
print(Product_Description[i].text)
print(Vendor[i].text)
print(Price[i].text)
con = sqlite3.connect('/home/mypc/Scrivania/aaaa/Database.db')
cursor = con.cursor()
Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values
print("Qui, all'interno del ciclo, eseguiresti" + sqlite_insert_query)
count = cursor.execute(sqlite_insert_query)
con.commit()
print("Record inserted successfully ", cursor.rowcount)
records_added = records_added + 1
cursor.close()
print("")
print(f'Added a total of {records_added} records')
print(f"found {Num_Groups} groups") # should be more than 1

Okay. So you are telling me you are good with connecting to the database, getting a cursor, executing sql with the cursor and the connection, and committing changes and closing the cursor. What you need is the sql string to execute.
Values = f"VALUES('{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
sqlite_insert_query = 'INSERT INTO TableExample1(Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) ' + Values
Then you should be able to, assuming cursor is the cursor set up:
count = cursor.execute(sqlite_insert_query)
I am assuming you set things up so ID is a unique key that will be generated if you do not specify it.
EDIT: So it sounds like the database was struggling with the ID and not making more than one new one. Assuming your IDs just need to be sequential (and only you are using it and scraping takes more than a second), you could handle IDs yourself as follows. After from datetime import datetime earlier
ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID},'{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
sqlite_insert_query = 'INSERT INTO TableExample1(ID, Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) ' + Values
Also, you mentioned getting the text describing the element in Product_Name as opposed to just the Product_Name (and similarly for the other fields). You need:
Product_Name = driver.find_element_by_class_name("tablet-desktop-only").text
(or to put (Product_Name.text) in my f-string, but that seems confusing)
2ND Edit:
What #Prophet was trying to say at https://stackoverflow.com/questions/68110293/problems-saving-scraped-data-to-database-python/68111109?noredirect=1#comment120399103_68111109 is as follows.
When you say:
`Product_Name = driver.find_element_by_class_name("tablet-desktop-only").text`
Product_Name is a string, and your for loop on Product_Name + Product_Description + Vendor + Price
is looping over a single concatenated string, and seq then has the characters of the string, taken one at a time. Then seq.text fails, as you experienced. That is why I commented out the print command over there, and put one in later to print the Values string instead. That is an approach that should work.
If you leave things the way you had them originally
`Product_Name = driver.find_element_by_class_name("tablet-desktop-only")
Product_Name is an element, which when merged into the Values string, gets converted to a string representation of the element, which is why you see all that irrelevant text in the database. I understand your leaving it so that the for loop will work, but then you should do something like this:
ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID},'{Product_Name.text}','{Product_Description.text}','{Vendor.text}','{Price.text}')"
or
Product_Name_Text = Product_Name.Text
Product_Description_Text = Product_Description.Text
Vendor_Text = Vendor.Text
Price_Text = Price.Text
and then
Values = f"VALUES({ID},'{Product_Name_Text}','{Product_Description_Text}','{Vendor_Text}','{Price_Text}')"
or
ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID}"
for seq in Product_Name + Product_Description + Vendor + Price:
print(seq.text)
Values = Values + "," + "'" + seq.text + "'"
Values = Values + ")"
I would definitely recommend, at least for debugging, printing Values or sqlite_insert_query. If you share those results back with us, we may be able to help, if it's still not working.
In the above options you can leave out my assigning the ID if that turns out not be an issue after all.
I do not see any loop in your code over different groups of elements, so I am not sure what you are expecting in terms of inserting more than one new entry.
3rd (final?) Edit:
There are 2 issues you are having. That you are getting extra stuff in the database is because you are putting the description of each element into the Values part of the SQL as opposed to the text. I have illustrated how to fix that by formmating the text into the Values string.
The second issue you are having is that your locator is finding only one thing when you say find_elements (I cannot debug your locators because I don't know the page). But the code here should tell you how many things matched. To test that hypothesis, I wrote my own version below, that works (if the page is permanent) grabbing data from Amazon on cosmetics. The details are not important, but the code should illustrate what has to happen. Also, I believe what I did earlier with IDs in unnecessary, the database will handle that.
Since I am not actually a database, I commented that out. You will uncomment it.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# launch and go to site, yours will vary
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.maximize_window()
wait = WebDriverWait(driver, 15)
driver.get("https://www.amazon.com/b?node=18505451011&pd_rd_w=gVvMZ&pf_rd_p=b6363b44-58dd-4354-979f-1446a1c45f7a&pf_rd_r=5FYAS41AJR7GPQ5Q74J4&pd_rd_r=9bfd1639-2256-4c17-928c-99cf03e00d63&pd_rd_wg=UPUHV")
wait.until(EC.presence_of_element_located((By.LINK_TEXT, "See all results")))
# SCRAPING
# you will want to change back to your locators
Product_Name = driver.find_elements_by_class_name("apb-line-clamp-3")
# actually number of reviewers, picks up see all at the bottom, too, but it's ok because of min
Product_Description = driver.find_elements_by_class_name("a-color-link")
Vendor = driver.find_elements_by_class_name(
"apb-browse-searchresults-product-byline")
Price = driver.find_elements_by_class_name("a-price") # prices
# if the locators work well, these match multiple groups of products not just one
Num_Groups = min(len(Product_Name), len(
Product_Description), len(Vendor), len(Price))
print(f"found {Num_Groups} groups") # should be more than 1
# I have commented out the database code, because I am not actually using one
#con = sqlite3.connect('/home/mypc/Desktop/aaaaa/Database.db')
# connect to database outside the loop, not for each item
# I removed the code about generating ID's, which the database should handle
records_added = 0
for i in range(Num_Groups): # should cause i to count from 0 up to and including Num_Groups-1
print("") # skip a line between stuff
print(Product_Name[i].text)
print(Product_Description[i].text)
print(Vendor[i].text)
print(Price[i].text)
# note below I need to format the .text into the values string, not the text description of the element
Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values
print("Here, inside the loop, you would execute " + sqlite_insert_query)
#cursor = con.cursor()
#count = cursor.execute(sqlite_insert_query)
# con.commit()
#print("Record inserted successfully ", cursor.rowcount)
records_added = records_added + 1
# cursor.close()
print("")
print(f'Added a total of {records_added} records')
output was:
found 12 groups
Crest 3D White Professional Effects Whitestrips 20 Treatments + Crest 3D White 1 Hour Express Whitestrips 2 Treatments - Teeth Whitening Kit
46,020
by Crest
$47
88
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Professional Effects Whitestrips 20 Treatments + Crest 3D White 1 Hour Express Whitestrips 2 Treatments - Teeth Whitening Kit','46,020','by Crest','$47
88')
REVLON One-Step Hair Dryer And Volumizer Hot Air Brush, Black, Packaging May Vary
274,196
by REVLON
$41
99
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('REVLON One-Step Hair Dryer And Volumizer Hot Air Brush, Black, Packaging May Vary','274,196','by REVLON','$41
99')
Waterpik WP-660 Water Flosser Electric Dental Countertop Professional Oral Irrigator For Teeth, Aquarius, White
72,786
by Waterpik
$59.99
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Waterpik WP-660 Water Flosser Electric Dental Countertop Professional Oral Irrigator For Teeth, Aquarius, White','72,786','by Waterpik','$59.99')
Schick Hydro Silk Touch-Up Multipurpose Exfoliating Dermaplaning Tool, Eyebrow Razor, and Facial Razor with Precision Cover, 3 Count (Packaging May Vary)
113,269
by Schick Hydro Silk
$68
27
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Schick Hydro Silk Touch-Up Multipurpose Exfoliating Dermaplaning Tool, Eyebrow Razor, and Facial Razor with Precision Cover, 3 Count (Packaging May Vary)','113,269','by Schick Hydro Silk','$68
27')
Neutrogena Makeup Remover Cleansing Face Wipes, Daily Cleansing Facial Towelettes to Remove Waterproof Makeup and Mascara, Alcohol-Free, Value Twin Pack, 25 Count, 2 Pack
62,881
by Neutrogena
$4
99
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Makeup Remover Cleansing Face Wipes, Daily Cleansing Facial Towelettes to Remove Waterproof Makeup and Mascara, Alcohol-Free, Value Twin Pack, 25 Count, 2 Pack','62,881','by Neutrogena','$4
99')
Gillette Fusion Power Men's Razor Blades - 8 Refills
32,435
by Gillette
$6.99
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Gillette Fusion Power Men's Razor Blades - 8 Refills','32,435','by Gillette','$6.99')
Softsoap Moisturizing Liquid Hand Soap, Soothing Clean Aloe Vera - 7.5 Fluid Ounces (6 Pack)
47,238
by Softsoap
$8
12
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Softsoap Moisturizing Liquid Hand Soap, Soothing Clean Aloe Vera - 7.5 Fluid Ounces (6 Pack)','47,238','by Softsoap','$8
12')
Neutrogena Lightweight Body Oil for Dry Skin, Sheer Body Moisturizer in Light Sesame Formula, 16 fl. oz
13,011
by Neutrogena
$11.96
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Lightweight Body Oil for Dry Skin, Sheer Body Moisturizer in Light Sesame Formula, 16 fl. oz','13,011','by Neutrogena','$11.96')
Crest 3D White Whitestrips with Light, Teeth Whitening Strips Kit, 10 Treatments, 20 Individual Strips (Packaging May Vary)
9,172
by Crest
$23
91
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Whitestrips with Light, Teeth Whitening Strips Kit, 10 Treatments, 20 Individual Strips (Packaging May Vary)','9,172','by Crest','$23
91')
Neutrogena Hydro Boost Hyaluronic Acid Hydrating Water Gel Daily Face Moisturizer for Dry Skin, Oil-Free, Non-Comedogenic Face Lotion, 1.7 fl. oz
51,417
by Neutrogena
$32.51
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Hydro Boost Hyaluronic Acid Hydrating Water Gel Daily Face Moisturizer for Dry Skin, Oil-Free, Non-Comedogenic Face Lotion, 1.7 fl. oz','51,417','by Neutrogena','$32.51')
CHI 44 Iron Guard Thermal Protection Spray 8 Fl Oz
17,660
by CHI
$5
33
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('CHI 44 Iron Guard Thermal Protection Spray 8 Fl Oz','17,660','by CHI','$5
33')
Crest 3D White Toothpaste Radiant Mint (3 Count of 4.1 oz Tubes), 12.3 oz (Packaging May Vary)
40,354
by Crest
$13.99
Here, inside the loop, you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Toothpaste Radiant Mint (3 Count of 4.1 oz Tubes), 12.3 oz (Packaging May Vary)','40,354','by Crest','$13.99')
added a total of 12 records
REVISION
Here is what your code needs to look like.
Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))
con = sqlite3.connect('/home/mypc/Scrivania/aaaa/Database.db')
cursor = con.cursor()
records_added = 0
for i in range(Num_Groups):
print(Product_Name[i].text)
print(Product_Description[i].text)
print(Vendor[i].text)
print(Price[i].text)
Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values
print("Qui, all'interno del ciclo, eseguiresti" + sqlite_insert_query)
count = cursor.execute(sqlite_insert_query)
con.commit()
print("Record inserted successfully ", cursor.rowcount)
records_added = records_added + 1
cursor.close()
The body of code indented under the for: is executed for each value of i (in the case you mentioned, from 0 to 11). Right now, because your database insert is outside this loop, it is only executed once.
Really the last edit:
We will use a parameterized query to let the database engine handle the apostrophe in the values (and anything else I may have missed). From what I am reading, that is a safer approach anyway to help prevent SQL injection attacks.
Change the Values= line (keeping the same indentation) to
Values = (Product_Name[i].text, Product_Description[i].text, Vendor[i].text, Price[i].text)
Change the sqlite_insert_query = line to say (keeping the same indentation as now):
sqlite_insert_query = 'INSERT INTO TableExample1 (Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) VALUES (?, ?, ?, ?);'
change the count=cursor.execute line to say
count = cursor.execute(sqlite_insert_query, Values)
That should work better and be safer. If it works, there is some optional cleaning up you could do. For example you could set sqlite_insert_query outside the loop, back where you connect to the database and initialize variables. You could also stop printing sqlite_insert_query and print Values instead (or just not print any of it since you have the 4 lines earlier that print the values).

Trying to create a record for each item in an order with SQLite Database and Python

I believe that I need to to expand the sql statement as the current one I have:
SELECT OrderID FROM Orders WHERE UserID = (?)
is likely to cause problems in the future when there are users with multiple orders. Is there a way that I can select and use the autoincrementing orderID in my orders table in combination?
For reference, my DB looks like this:
And the python I currently have is like this however I feel that it may have to be rewritten entirely.
results[0][0] is the ID of the current user from the Users table
elif userinput == "5":
basket = []
print(pd.read_sql_query('SELECT * FROM Products', conn))
shopping = True
while shopping:
itemToAdd = input("Please enter the ID of the item to add to the basket: ")
basket.append(itemToAdd)
print(basket)
continueShop = input("Continue shopping?(y/n): ")
if continueShop == "n":
conn.execute("INSERT INTO Orders (UserID) VALUES (?)", (results[0][0],))
conn.commit()
counter = 0
for items in basket:
createOrderItems = "INSERT INTO OrderItems (OrderID, ProductID) VALUES (?,?)"
currentOrder = ("SELECT OrderID FROM Orders WHERE UserID = (?) ", (results[0][0]))
conn.execute(createOrderItems, (currentOrder[0], basket[counter]))
counter = +1
conn.commit()
Because the database is built using the DB Browser application, I really have no clue how to access the parts of it that I need to ensure I am selecting the right record. Any help is appreciated, thanks.
Also, since the primary key of each table is autoincrementing I need a way to select only the record that was created just now using
conn.execute("INSERT INTO Orders (UserID) VALUES (?)",
However, I cannot think of a way to do so even though the record is unique in the orders table, as there is no way of knowing which order is the one that needs to be looked at as far as I can tell

You can use the lastrowid attribute of the cursor object to find the id of the inserted order. Then use a compound WHERE in the select statement (ie WHERE userid = something and orderid = something_else).

ID tag is not auto incrementing the numbers in python

I am learning python and trying to replicate what online tutorials do. I am trying to create a python desktop app where data is store in Postgresql. code is added below,
`cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY, title text, author text, year integer, isbn integer)")`
problem is with (id INTEGER PRIMARY KEY), when i execute the code its showing none in place of 1st index. i want to show numbers.
please help
this is for Python 3.7.3, psycopg2==2.8.3,
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,
title text, author text, year integer, isbn integer)")
conn.commit()
conn.close()
the result I am expecting is auto increment of numbers in 1st index where as presently it shows NONE.
below is the present and expected result again.
none title auther year isbn
01 title auther year isbn

Trying to use a Cursor execute will not work for a CREATE statement and hence the NONE. See below for an example.
Re Indexes :-
There will be no specific index as column_name INTEGER PRIMARY KEY is special in that it defines the column as an alias of the rowid column which is a special intrinsic index using the underlying B-tree storage engine.
When a row is inserted then if no value is specified for the column (e.g. INSERT INTO book (title,author, year, isbn) VALUES ('book1','The Author','1999','1234567890') then id will be 1 and typically (but not certainly) the next row inserted will have an id of 2 and so on.
If after adding some rows you use SELECT * FROM book, then the rows will be ordered according to the id as no other index is specified/used.
Perhaps have a look at Rowid Tables.
Example
Perhaps consider the following example :-
DROP TABLE IF EXISTS book;
CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY, title text, author text, year integer, isbn integer);
INSERT INTO book (title,author, year, isbn) VALUES
('book1','The Author','1999','1234567890'),
('book2','Author 2','1899','2234567890'),
('book3','Author 3','1799','3234567890')
;
INSERT INTO book VALUES (100,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID
INSERT INTO book (title,author, year, isbn) VALUES
('book11','Author 11','1999','1234567890'),
('book12','Author 12','1899','2234567890'),
('book13','Author 13','1799','3234567890')
;
INSERT INTO book VALUES (10,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID
SELECT * FROM book;
This :-
DROPs the book table (to make it easily re-runable)
CREATEs the book table.
INSERTs 3 books with the id not specified (typpical)
INSERTs a fourth book but with a specific id of 100
INSERTs another 3 books (not that these will be 101-103 as 100 is the highest id before the inserts)
INSERTs a last row BUT with a specific id of 10.
SELECTs all rows with all columns from the book table ordered, as no ORDER BY has been specified, according to the hidden index based upon the id. NOTE although id 10 was the last inserted it is the 4th row.
Result
In Python :-
conn = sqlite3.connect("books.db")
conn.execute("DROP TABLE IF EXISTS book")
conn.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,title text, author text, year integer, isbn integer)")
conn.execute("INSERT INTO book (title,author, year, isbn) "
"VALUES('book1','The Author','1999','1234567890'), "
"('book2','Author 2','1899','2234567890'), "
"('book3','Author 3','1799','3234567890');")
conn.execute("INSERT INTO book VALUES (100,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID")
conn.execute("INSERT INTO book (title,author, year, isbn) VALUES ('book11','Author 11','1999','1234567890'),('book12','Author 12','1899','2234567890'),('book13','Author 13','1799','3234567890');")
conn.execute("INSERT INTO book VALUES (10,'book10','Author 10','1999','4234567890'); --<<<<<<<<<< specific ID")
cur = conn.cursor()
cur.execute("SELECT * FROM book")
for each in cur:
print("{0:<20} {1:<20} {2:<20} {3:<20} {4:<20}".format(each[0],each[1],each[2],each[3],each[4]))
conn.commit()
conn.close()
The results in :-
1 book1 The Author 1999 1234567890
2 book2 Author 2 1899 2234567890
3 book3 Author 3 1799 3234567890
10 book10 Author 10 1999 4234567890
100 book10 Author 10 1999 4234567890
101 book11 Author 11 1999 1234567890
102 book12 Author 12 1899 2234567890
103 book13 Author 13 1799 3234567890

because you say I am inserting the data manually then instead of
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS book (id INTEGER PRIMARY KEY,
title text, author text, year integer, isbn integer)")
conn.commit()
conn.close()
Try to use
def connect():
conn=sqlite3.connect("books.db")
cur=conn.cursor()
cur.execute("SELECT * FROM books")
for row in cur:
print(row[0],row[1],row[2],row[3],row[4])
conn.commit()
conn.close()

How to Write a Value Into a Chosen Row and Column in SQLite using Python

So I'm trying to work with an SQLite database at the moment and was hoping to get some input on the best way of writing a value to a particular row and column (so cell) of a database. I know how to write to the database row by row, so basically appending a row to the end of the database each time, but instead I would like to write the data into the database non sequentially.
I have put together an arbitrary example below to illustrate what I'm trying to do by using apples. In this case I create two tables in a database. The first table will be my ID table and is called apples. This will contain a primary key and two columns for the apple name and the farm it is grown in. The second table keyFeatures will contain the primary key again which refers back to the ID of the apple in the apple table, but also a column for the taste, texture and the colour.
In the example below I have only the taste, texture and colour of the apple Pink Lady from farm 3. I now want to write that information into row 3 of the table keyFeature in the relevant columns before any of the other rows are populated. For the life of me I can't work out how to do this. I assume I need to position the cursor to the correct row, but from the documentation I am not clear on how to achieve this. I'm sure that this is a trivial problem and if anyone can point me in the right direction I would greatly appreciate it!
import sqlite3
dbName = 'test.db'
################# Create the Database File ###################
# Connecting to the database file
conn = sqlite3.connect(dbName)
c = conn.cursor()
#Create the identification table with names and origin
c.execute('''CREATE TABLE apples(appleID INT PRIMARY KEY, Name TEXT,
farmGrown TEXT)''')
#Create the table with key data
c.execute('''CREATE TABLE keyFeature(appleID INT PRIMARY KEY, taste INT,
texture INT, Colour TEXT)''')
#Populate apples table and id in keyFeature table
AppleName = ['Granny Smith', 'Golden Delicious', 'Pink Lady', 'Russet']
appleFarmGrown = ['Farm 1', 'Farm 2', 'Farm 3', 'Farm 4']
id = []
for i in range(len(AppleName)):
id.append(i+1)
c = conn.cursor()
c.execute('''INSERT INTO apples(appleID, Name, farmGrown)
VALUES(?,?,?)''', (id[i], AppleName[i], appleFarmGrown[i]))
c.execute('''INSERT INTO keyFeature(appleID)
VALUES(?)''', (id[i],))
#Current Apple to populate row in keyFeature
appleCurrent = (('Pink Lady','Farm 3'))
tasteCurrent = 4
textureCurrent = 5
colourCurrent = 'red'
#Find ID and write into the database
c.execute("SELECT appleID FROM apples")
appleIDDB = c.fetchall()
c.execute("SELECT name FROM apples")
nameDB = c.fetchall()
c.execute("SELECT farmGrown FROM apples")
farmGrownDB = c.fetchall()
conn.commit()
conn.close()
# I assume that if I close the connection the cursor whould be positioned at the
# first row again but this doesn't appear to be the case
conn = sqlite3.connect(dbName)
for i in range(len(appleIDDB)):
c = conn.cursor()
if ((nameDB[i][0] == appleCurrent[0]) and (farmGrownDB[i][0] == appleCurrent[1])):
idCurrent = appleIDDB[i][0]
print("This apple matches the apple stored with id number " +str(idCurrent))
# This writes into the fifth row of the table
c.execute('''INSERT INTO keyFeature(taste, texture, Colour)
VALUES(?,?,?)''', (tasteCurrent, textureCurrent, colourCurrent))
conn.commit()
conn.close()

You're almost there.
A relational database such as sqlite isn't quite like a table in a speadsheet. Instead of a list of rows with a certain order you just have "a big bag of rows" (technically called a set of tuples) and you can sort them any way you want.
The way we solve your problem, as you've already identified when you created the table, is to make sure every row has a key that allows us to identify it (like your apple ID). When we want this key to represent an ID in another table, that's called a foreign key. So we just need to add a foreign key (called appleID) to the keyFeature table, and use that whenever we add a row to it.
First, get rid of this from your first for loop, we don't need it at this stage, the table can sit empty.
c.execute('''INSERT INTO keyFeature(appleID)
VALUES(?)''', (id[i],))
Next, you don't need to get the whole table to find the apple you want, you can just select the one you are interested in:
c.execute("SELECT appleID FROM apples WHERE name=? AND farm=?",("Pink Lady", "Farm 3"))
idCurrent = c.fetchone()[0]
The real trick is, when adding data to keyFeature, we have to insert all the data in one statement. This way a new tuple (row) is created with the ID and all the other information all at once. As if it were in "the right place" in the table.
c.execute('''INSERT INTO keyFeature(appleID, taste, texture, Colour)
VALUES(?,?,?,?)''', (idCurrent, tasteCurrent, textureCurrent, colourCurrent))
Now we can retrieve information from the keyFeature table using the ID of the apple we're interested in.
c.execute("SELECT taste, texture, Colour FROM keyFeature WHERE apple_id=?", (my_apple_id,))
Final complete code:
import sqlite3
dbName = 'test.db'
################# Create the Database File ###################
# Connecting to the database file
conn = sqlite3.connect(dbName)
c = conn.cursor()
#Create the identification table with names and origin
c.execute('''CREATE TABLE apples(appleID INT PRIMARY KEY, Name TEXT,
farmGrown TEXT)''')
#Create the table with key data
c.execute('''CREATE TABLE keyFeature(appleID INT PRIMARY KEY, taste INT,
texture INT, Colour TEXT)''')
#Populate apples table and id in keyFeature table
AppleName = ['Granny Smith', 'Golden Delicious', 'Pink Lady', 'Russet']
appleFarmGrown = ['Farm 1', 'Farm 2', 'Farm 3', 'Farm 4']
id = []
for i in range(len(AppleName)):
id.append(i+1)
c = conn.cursor()
c.execute('''INSERT INTO apples(appleID, Name, farmGrown)
VALUES(?,?,?)''', (id[i], AppleName[i], appleFarmGrown[i]))
#Current Apple to populate row in keyFeature
appleCurrent = ('Pink Lady','Farm 3')
tasteCurrent = 4
textureCurrent = 5
colourCurrent = 'red'
#Find ID and write into the database
c.execute("SELECT appleID FROM apples WHERE name=? AND farm=?",(appleCurrent[0], appleCurrent[1]))
idCurrent = c.fetchone()[0]
c.execute('''INSERT INTO keyFeature(appleID, taste, texture, Colour)
VALUES(?,?,?,?)''', (idCurrent, tasteCurrent, textureCurrent, colourCurrent))
conn.commit()
conn.close()

Query to insert data in mysql table contains foreign key

I am very new to Mysql and dumping a file in db using python. I'm having 2 tables
The file format is:
id name sports
1 john baseball
2 mary Football
like student & Sports
Student table
id name
1 John
2 Mary
here id is primary key
& in sports table
stu_id sports_title
1 Baseball
2 Football
and here stu_id is foreign key reference with student table
and my problem is
query="insert into sports (stu_id,name)VALUES (%d,%s)"
("select id from student where id=%d,%s")
#words[0]=1 ,words[2]=Baseball
args=(words[0],words[2])
cursor.execute(query,args)
upon executing this code, I'm facing
"Not all parameters were used in the SQL statement")
ProgrammingError: Not all parameters were used in the SQL statement

You can't use both VALUES and SELECT as the source of the data in INSERT. You use VALUES if the data is literals, you use SELECT if you're getting it from another table. If it's a mix of both, you use SELECT and put the literals into the SELECT list.
query = """
INSERT INTO sports (stu_id, sports_title)
SELECT id, %s
FROM student
WHERE name = %s
"""
args = (words[2], words[1])
cursor.execute(args)
Or, since the file contains the student ID, you don't need SELECT at all.
query = "INSERT INTO sports(stu_id, sports_title) VALUES (%d, %s)"
args = (words[0], words[1])
cursor.execute(args)