Develop Reference

Trouble inserting JSON data into sqlite3 database

I created an sqlite3 database using python to store data as shown in the code below
import sqlite3
conn = sqlite3.connect('tweets_data.sqlite')
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS tweets')
cur.execute('''
CREATE TABLE tweets (
id INTEGER PRIMARY KEY, created_at TEXT, full_text TEXT,
favourite_count INTEGER, retweet_count INTEGER)
''')
With this table, i want to store data from a JSON file (parts of the code are screenshotted and attached as images), which i have loaded as seen below
import json
with open('tweets.json') as f:
data = json.load(f)
After that, i tried inserting the data into the table using a for loop to pull out all unique tweet id and its following information. The code below is what i tried doing
for records in data:
cur.execute('INSERT INTO tweets (id, created_at, full_text, favourite_count, retweet_count) VALUES (?, ?, ?, ?, ?)',
(records['id'], records['created_at'], records['full_text'], records['user']['favourites_count'], records['retweet_count']))
conn.commit()
print(cur.fetchall())
conn.close()
However when i did a print (cur.fetchall()), the output was only an empty list. Nothing was inserted into the table. Thank you if anybody is able to help!
json file page 1
json file page 2

cur.fetchall() returns the result of the last query, and INSERT yields no result. You need a SELECT-query first:
cur.execute('SELECT * FROM tweets')
rows = cut.fetchall()

MySQL Insert / update from python, data from excel spreadsheet

Upsert to MySQL using python and data from excel.
Im working on populating a MySQL DB, using python.
The data is stored on excel sheets.
Because the DB is suppossed to be used for monitoring "projects", there's a posibility for repeated pk, so in that case it need to be updated instead of insert, because a project can have many stages.
Also, there's a value to be inserted in the DB table, that can't be added from the spreadsheet. So i'm wondering if in that case, the insert of this value, most be done using a separated query for it or if theres a way to insert it in the same query. The value is the supplier ID and needs to be inserted between id_ops and cif_store.
And to finish, I need to perform an inner join, to import the store_id using the store_cif, from another table called store. I know how do it, but im wondering if it also must be executed from a sepparated query or can be performed at the sameone.
So far, i have done this.
import xlrd
import MySQLdb
def insert():
book = xlrd.open_workbook(r"C:\Users\DevEnviroment\Desktop\OPERACIONES.xlsx")
sheet = book.sheet_by_name("Sheet1")
database = MySQLdb.connect (host="localhost", user = "pytest", passwd = "password", db = "opstest1")
cursor = database.cursor()
query = """INSERT INTO operation (id_ops, cif_store, date, client,
time_resp, id_area_service) VALUES (%s, %s, %s, %s, %s, %s)"""
for r in range(1, sheet.nrows):
id_ops = sheet.cell(r,0).value
cif_store = sheet.cell(r,1).value
date = sheet.cell(r,2).value
client = sheet.cell(r,3).value
time_resp = sheet.cell(r,4).value
id_area_service = sheet.cell(r,5).value
values = (id_ops, cif_store, date, client, time_resp, id_area_service)
cursor.execute(query, values)
# Close the cursor
cursor.close()
# Commit the transaction
database.commit()
# Close the database connection
database.close()
# Print results
print ("")
print ("")
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print ("Imported", columns,"columns and", rows, "rows. All Done!")
insert()

What you are looking for is INSERT ... ON DUPLICATE KEY UPDATE ...
Take a look here https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html
Regarding the extraneous data, if its a static value for all rows you can just hard code it right into the INSERT query. If it's dynamic you'll have to write some additional logic.
For example:
query = """INSERT INTO operation (id_ops, hard_coded_value, cif_store, date, client,
time_resp, id_area_service) VALUES (%s, "my hard coded value", %s, %s, %s, %s, %s)"""

Creating database in Python using sqlite3

I got stuck when I was trying to create database in Python using sqlite3. The below is what I did. When I tried to run, it kept telling me the tables already exist. I couldn't figure out why. Thanks!
import sqlite3
variables = (data)
functions = (data)
var_func = (data)
conn = sqlite3.connect('python_database')
c = conn.cursor()
#create table
c.execute(''' CREATE table variable_table (
id integer,
name text,
module text,
type text,
desc text) ''')
c.execute(''' CREATE table function_table (
id integer,
name text) ''')
c.execute(''' CREATE table var_func_table (
variable_id integer,
function_id integer,
type text) ''')
#fill tables with data
for row in variables:
c.execute ('insert into variable_table values (?,?,?,?,?)', row )
for row in functions:
c.execute ('insert into function_table values (?,?)', row)
for row in var_func:
c.execute ('insert into var_func_table values (?,?,?)', row)
# Save (commit) the change
conn.commit
conn.close

MySQL inserts stop from Python

I am trying to pull records from the a database, do some processing on one of the fields, then insert the data back into the database. However, I notice that the script stops after one successful insert, even though there are many more records beyond what is inserted into the database.
cur = db.cursor()
# Pull records from the database, process each one, and store back into the database
SQLQuery = """SELECT * FROM cybersecurity4.hackhoundposts"""
cur.execute(SQLQuery)
for row in cur:
postID = row[0]
threadID = row[1]
authorID = row[2]
url = row[3]
subforum = row[4]
threadTitle = row[5]
authorName = row[6]
postDateTime = row[7]
postSequence = row[8]
flatcontent = row[9]
userTitle = row[11]
hasAttachment = row[12]
print (postID)
# #process the flatcontent
flatcontent = rmvSpecialCharsandCamelCase(flatcontent)
# insert into the database
try:
string = """INSERT INTO cybersecurity4.cleanedhackhoundposts
(postid, threadid, authorid, URL, subforum, threadTitle, authorName, postdatetime, postSequence, flatcontent, usertitle, hasattachment)
VALUES ('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s','%s', '%s', '%s', '%s');""" % \
(postID ,threadID,authorID,url,subforum,threadTitle,authorName,postDateTime,postSequence,flatcontent,userTitle,hasAttachment)
print (string)
cur.execute(string)
print ("Successfully inserted post " + str(postID))
except:
int("Failed to insert post" + str(postID))

As i understand your code, you have nothing repeating the insertion, so you are retrieving N records, but only inserting 1 record. Some sudo code of what you need to achieve is.
While(cursor hasnext):
select row
manipulate
insert
You are close to the result, but missing the last steps.
Edit:
You can not insert ids on the same id as you select you must let the db handle the id or simply update the row. So removing id from the insert query would probably fix the problem. Next time also check the database log it tells alot.

Insert data into MySQL table from Python script

I have a MySQL Table named TBLTEST with two columns ID and qSQL. Each qSQL has SQL queries in it.
I have another table FACTRESTTBL.
There are 10 rows in the table TBLTEST.
For example, On TBLTEST lets take id =4 and qSQL ="select id, city, state from ABC".
How can I insert into the FACTRESTTBL from TBLTEST using python, may be using dictionary?
Thx!

You can use MySQLdb for Python.
Sample code (you'll need to debug it as I have no way of running it here):
#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Select qSQL with id=4.
cursor.execute("SELECT qSQL FROM TBLTEST WHERE id = 4")
# Fetch a single row using fetchone() method.
results = cursor.fetchone()
qSQL = results[0]
cursor.execute(qSQL)
# Fetch all the rows in a list of lists.
qSQLresults = cursor.fetchall()
for row in qSQLresults:
id = row[0]
city = row[1]
#SQL query to INSERT a record into the table FACTRESTTBL.
cursor.execute('''INSERT into FACTRESTTBL (id, city)
values (%s, %s)''',
(id, city))
# Commit your changes in the database
db.commit()
# disconnect from server
db.close()