This question already has answers here:
How to UPSERT (MERGE, INSERT ... ON DUPLICATE UPDATE) in PostgreSQL?
(7 answers)
Closed 7 years ago.
I have a python script that is using the Psycopg adapter; I am parsing a JSON Array and inserting into my PostgreSQL database.
for item in data["SchoolJSONData"]:
mId = item.get("Id")
mNumofRooms = item.get("NumofRooms")
mFloors = item.get("Floors")
con = None
con = psycopg2.connect("dbname='database' user='dbuser'")
cur = con.cursor()
cur.execute('INSERT INTO Schools(Id, NumofRooms, Floors)VALUES(%s, %s, %s)',(mId, mNumofRooms, mFloors))
con.commit()
Everytime I run the script again, I get the following:
psycopg2.IntegrityError: duplicate key value violates unique constraint "schools_pkey"
How can I run the insert script so that it will ignore existing entries in the database?
EDIT: Thanks for the replies all... I am trying to NOT overwrite any data, only ADD (if the PK is not already in the table), and ignore any errors. In my case I will only be adding new entries, never updating data.
There is no single one way to solve this problem. As well this problem has little to do with python. It is valid exception generated by the database ( not just postgre all databases will do the same ).
But you can try - catch this exception and continue smoothly later.
OR
you can use "select count(*) where id = mId" to ensure it is not existing already.
Related
This question already has answers here:
Python mySQL Update, Working but not updating table
(3 answers)
Closed 3 months ago.
I have a MySQL database instance hosted on GCP, and I am connecting to it using the pymysql python package. I would like to delete some rows from a database table called Basic.
The code I have written to do this is included below. The variable conf contains the connection details to the database instance.
import pymysql
import pandas as pd
# Establish connection.
connection = pymysql.connect(**conf)
cursor = connection.cursor()
# Delete rows of table.
try:
cursor.execute("DELETE FROM Basic WHERE Date = '2022-11-25 04:00:00';")
except Exception as ex:
print("Exception occured: %s" %ex)
finally:
connection.close()
# Check the table to see if deletion has occurred.
connection = pymysql.connect(**conf)
cursor = connection.cursor()
cursor.execute("SELECT * FROM Basic WHERE Date = '2022-11-25 04:00:00'")
connection.close()
df = pd.DataFrame(cursor.fetchall(), columns = ["Date", "State", "Price", "Demand"])
Clearly one would expect the dataframe defined here to have no rows, but this is not the case. This shows that the SELECT statement included in the code above produces the expected result, but the corresponding DELETE statement does not.
Why is this the case?
The code above requires the addition of the following line, in order to commit the DELETE statement.
connection.commit()
The commit method should be called after every transaction that modifies data, such as this one.
This question already has answers here:
How to use variables in SQL statement in Python?
(5 answers)
Closed 10 months ago.
I've recently started learning and trying to apply SQL into my python code and came across this problem; I can't figure out a way to properly insert variables into a table using the set up I have. This setup came from a SQL library tutorial on realpython.com (https://realpython.com/python-sql-libraries/), and judging by the different formats of sqlite setups, I feel like it isn't too efficient.
import sqlite3
from sqlite3 import Error
# - - - - SQL DATABASE SETUP - - - - #
def create_connection(path):
connection = None
try:
connection = sqlite3.connect(path)
print("Connection to SQLite DB successful")
except Error as e:
print(f"The error '{e}' occurred")
return connection
connection = create_connection("databse.sqlite") # Creates the connection
def execute_query(connection, query):
cursor = connection.cursor()
try:
cursor.execute(query)
connection.commit()
print("Query executed successfully")
except Error as e:
print(f"The error '{e}' occurred")
create_users_table = """
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
age INTEGER,
grade INTEGER,
gender TEXT,
nationality TEXT
);
"""
execute_query(connection, create_users_table)
# - - - - ^ SQL DATABSE SETUP ^ - - - - #
I believe I understand everything going on in these two functions, although I noticed that not many use this way, and I think that's why I couldn't find an answer to my problem, I would appreciate any advice on more efficient setups.
I tried using a solution that used the q-mark style in order to insert variables, but it didn't work for me. I think it could be because of the triple quotes strings, and the query statement being a parameter of a function within a function.
First attempt looked like this
name = str(input("Hi! What is your name?\n"))
store_name = ("""
INSERT INTO users (name)
VALUES (?)
""", name)
execute_query(connection, store_name)
Which gave me the error "ValueError: operation parameter must be str"
I later tried to remove the variable and just plug in the query as a single line statement, enclosed within a parameter
execute_query(connection, ("INSERT INTO users (name) VALUES (?)", name))
Same error, might be a simple mistake on my end that I'm not seeing.
And again if this setup isn't efficient, I would appreciate any advice.
you need to use same query as MySQL.
For example - "INSERT INTO table (column1,column2 ,..)
VALUES( value1, value2 ,...)"
This question already has answers here:
Sqlite insert query not working with python?
(2 answers)
Closed 1 year ago.
I am currently learning SQL for one of my projects and the site, that I learn from, advised me to use DB Browser to see my Database Content. However, I can't see the data inside the SQL. This is how my code looks like. I'm creating a table and then trying to write some values in it. It creates the DB successfully but the data doesn't show up.
import sqlite3 as sql
connection = sql.connect("points.db")
cursor = connection.cursor()
cursor.execute("CREATE TABLE IF NOT EXISTS servers (server_id TEXT, name TEXT, exp INTEGER)")
cursor.execute("INSERT INTO servers VALUES ('848117357214040104', 'brknarsy', 20)")
Can you check that your data is inserted?
Something like this in the end:
cursor.execute("SELECT * FROM servers")
r = cursor.fetchall()
for i in r:
print(r)
Perhaps SQLite browser just needs a refresh
This question already has answers here:
SQLite not saving data between uses
(1 answer)
python sqlite3, how often do I have to commit?
(1 answer)
Closed 1 year ago.
I have a database in python sqlite3 and I'm trying to delete rows from it depending on a value that I selected.
After the execution of this code, I got the result that I want when I'm in this function. But the problem is that when I'm out of this function and try to print the database, the deletion query did not work but the one to add values did. Can anyone understand why?
def datamanip():
selected = SecTree.focus()
values = SecTree.item(selected, 'text')
conn = sqlite3.connect('DataStore.db')
c = conn.cursor()
query='DELETE FROM Limits WHERE TypeCard=(?)'
c.execute(query,(values,))
c.execute("INSERT INTO Limits VALUES (:TypeCard,:CreaseMaxC,:CreaseMinC,:CreaseMaxA,:CreaseMinA,:WidthMaxC,:WidthMinC,:WidthMaxA,:WidthMinA)",
{'TypeCard':values,
'CreaseMaxC': w2data,
'CreaseMinC': wdata,
'CreaseMaxA': w4data,
'CreaseMinA': w3data,
'WidthMaxC': w6data,
'WidthMinC': w5data,
'WidthMaxA': w8data,
'WidthMinA': w7data
}
)
c.execute('SELECT * FROM Limits')
records= c.fetchall()
print(records)
EDIT:
The connection must be commited after making the deletion for the database.
conn.commit()
solved the problem.
This question already has answers here:
Executing multiple statements with Postgresql via SQLAlchemy does not persist changes
(3 answers)
Closed 3 years ago.
I am trying to execute a sql query from a file using sqlalchemy.
When I run the queries, I get a result saying that it affected x amount of rows, but when I check the DB it doesn't actually insert anything to the tables.
Here is my current code:
def import_to_db(df, table_name):
df.to_sql(
table_name,
con=engine,
schema='staging',
if_exists='replace',
index= False,
method= 'multi'
)
print('imported data to staging.{}'.format(table_name))
with open('/home/kyle/projects/data_pipelines/ahc/sql/etl_{}.sql'.format(table_name)) as fp:
etl = fp.read()
result = engine.execute(etl)
print('moved {} rows to public.{}'.format(result.rowcount, table_name))
When I run the .sql scripts manually, they work fine. I even tried making stored procedures but that didn't work either. Here is an example of one of the sql files im executing:
--Delete Id's in prod table that are in current staging table
DELETE
FROM public.table
WHERE key IN
(SELECT key FROM staging.table);
--Insert new/old id's into prod table and do any cleaning
INSERT INTO
public.table
SELECT columna, columnb, columnc
FROM staging.table;
Found a solution, although I don't fully understand it.
I added BEGIN; at the top of my script, and COMMIT; at the bottom.
This works, but my row count now say -1 so it doesn't help me much for logging.