Develop Reference

How to update or ignore row if primary key already exist using python?

I have read documentation and have seeing a lot of examples, but still don't understand what I need to do...
I have few parsers and each make me a csv file, I made a script to connect to my database and add data from csv to pgsql, but don't know how to avoid error, when id (primary key) already exists.
I need to update this row or just ignore.
I have find similar questions, but code in answers doesn't help me or did nothing.
Like create a temporary table and copy from that, but don't understand why it's not working for me.
Please, could some one help me?
This is my code using sqlalchemy:
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine('postgresql://postgres:root#localhost:5432/test_DB')
connection = engine.connect()
with open(r'csv\ss.csv', 'r') as file:
data_df = pd.read_csv(file)
data_df.to_sql('car', con=engine, index=False, if_exists='append')
connection.close()
Or this using psycopg2:
conn = psycopg2.connect(
host=hostname,
dbname=database,
user=uesername,
password=password,
port=port_id)
cursor = conn.cursor()
my_file = open(r'csv\ss.csv')
SQL_STATEMENT = """
COPY car FROM STDIN WITH
CSV
HEADER
DELIMITER AS ','
"""
cursor.copy_expert(sql=SQL_STATEMENT, file=my_file)
conn.commit()
cursor.close()

How to put multi csv files into one sqlite or mysql Database in Python?

For only one cvs file, I can import it into sqlite as follows:
conn = sqlite3.connect("data.sqlite")
df = pd.read_csv('data.csv')
df.to_sql('data', conn, if_exists='append', index=False)
conn.close()
What if I have multi cvs files? How to ingest all the tables from cvs files into one sqlite or mysql database?

Assuming all files need to be written to different tables. Assuming the names of the tables should be set as file names, without the extension.
file_names = [...]
conn = sqlite3.connect("data.sqlite")
for file_name in file_names:
table_name = file_name.split('.')[0]
df = pd.read_csv('data.csv')
df.to_sql(table_name, conn, if_exists='append', index=False)
conn.close()
this would create or open data.sqlite and for each file in file_names it will create a pandas df and write it to a new or append to an existing table in the same SQLite DB
This method will not work with MySQL, you will need SQLAlchemy connection to write to MySQL
more on it: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html

Python Error - no viable alternative input when trying to insert values from file

I'm trying to insert some values from a csv file through Python but I'm getting a no viable alternative at input error. When I specify the values instead of %s the code works but when I try to use %s it fails. This is my code:
import jaydebeapi
import jpype
import pyodbc
import pandas as pd
import csv
conn = pyodbc.connect("myconnection")
cursor = conn.cursor()
with open('/Users/user/Desktop/TEST.csv') as f:
reader = csv.reader(f)
for row in reader:
cursor.execute("INSERT INTO mytable (user_id, email) VALUES(%s,%s)", row)
#close the connection to the database.
mydb.commit()
cursor.close()

How to speed up pandas read_sql for MySQL data using temp file?

I have got solution for Postgres, but I don't know how to achieve this functionality in MySQL?
import tempfile
import pandas
def read_sql_tmpfile(query, db_engine):
with tempfile.TemporaryFile() as tmpfile:
copy_sql = "COPY ({query}) TO STDOUT WITH CSV {head}".format(
query=query, head="HEADER"
)
conn = db_engine.raw_connection()
cur = conn.cursor()
cur.copy_expert(copy_sql, tmpfile)
tmpfile.seek(0)
df = pandas.read_csv(tmpfile)
return df
The problem here is that for PostgreSQL , the db_engine is created by using the library psycopg2. And for MySQL I can use pyodbc, sqlalchemy and mysql.connector. But any of these doesn't have the copy_expert method available.

how to render data from postgresql to csv in python flask app?

I am new in python and trying to write a code in it. I am trying to run a select query but i am not able to to render a data to csv file ?
this is the psql query :
# \copy (
# SELECT
# sr.imei,
# sensors.label,sr.created_at,
# sr.received_at,
# sr.type_id,
#
but How to write it in python to render it to csv file ?
thanking you,
Vikas

sql = "COPY (SELECT * FROM sensor_readings WHERE reading=blahblahblah) TO STDOUT WITH CSV DELIMITER ';'"
with open("/tmp/sensor_readings.csv", "w") as file:
cur.copy_expert(sql, file)
I think you just need to change the sql for your use, and it should work.

Install psycopg2 via pip install psycopg2 than you need something like this
import csv
import psycopg2
query = """
SELECT
sr.imei,
sensors.label,sr.created_at,
sr.received_at,
sr.type_id,
sr.data FROM sensor_readings as sr LEFT JOIN sensors on sr.imei = sensors.imei
WHERE sr.imei not like 'test%' AND sr.created_at > '2019-02-01'
ORDER BY sr.received_at desc
"""
conn = psycopg2.connect(database="routing_template", user="postgres", host="localhost", password="xxxx")
cur = conn.cursor()
cur.execute(query)
with open('result.csv', 'w') as f:
writer = csv.writer(f, delimiter=',')
for row in cur.fetchall():
writer.writerow(row)
cur.close()
conn.close()