I am trying to format a insert query string for multiple rows but also with ON CONFLICT. I got all mixed up with formatting arguments.
ses_crud_sql = """INSERT INTO session_crud(orgid, appid, sessionid, userid, customlabel, uploadedon)
VALUES (%s, %s, %s, %s, %s, %s)
ON CONFLICT (orgid, appid)
DO UPDATE SET customlabel = array_append(customlabel, '%(label_name)s') WHERE sessionid=sessionid
"""
ses_crud_rows = [(org_id, app_id, sessionid, userid, str({label_name}), datetime.strftime(current_time, '%Y-%m-%d %H:%M:%S'))
for sessionid in session_ids]
cursor.executemany(ses_crud_sql, ses_crud_rows)
I need to insert multiple rows for every session in session_ids list.
So I also want to add %(label_name)s but this gives me
psycopg2.ProgrammingError: argument formats can't be mixed
Related
I have a table in mysql and i am inserting data in it from a python client.
I am using a insert query to insert data into the table
code
sql_insert_query = """ INSERT INTO Data
(`deviceID`,`date`,`timestamp`,`counter`,`rssi`,
`CO2 Sensor Value`,
`Supply DPT`,
`block`,
`floor`)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"""
connection = mysql.connector.connect(host='localhost',database='minniedb',user='',
password='',auth_plugin='mysql_native_password')
cursor = connection.cursor()
"""
create 'insert_tuple' based on some api calls
"""
cursor.execute(sql_insert_query,insert_tuple)
connection.commit()
cursor.close()
connection.close()
print('inserted in db')
This works fine when everything is static. I have a case when the number of columns in my table is around 60-70 and the parameters I get from api is a subset of columns(around 10-15) and these parameters can change every time. The api returns the column name and the value.
Sample return from api can be of form
{
'deviceID':20,
'counter' :61,
'block' :'A'
}
or it can be
{
'deviceID' :25,
'CO2 Sensor Value':600,
'floor' : 5
}
How do i write a query in such case to insert whatever data i received from api in the respective columns and have others as null.
You can try like this :
sensor_data = {
'deviceID':20,
'counter' :61,
'block' :'A'
}
sql_insert_query = """ INSERT INTO Data {} VALUES {}""".format(tuple(sensor_data.keys()), tuple(sensor_data.values()))
P.S: For sensor data I'll suggest using google Firebase :)
When I insert a date from a file that has it formatted "dd/mm/yy" into my database table with the date formatted "yy/mm/dd" the date is wrong:
Instead of getting 2019:04:11 I get 2011:04:19.
I want to keep the database format ("yy/mm/dd")
I have tried:
actualdate = DATE_FORMAT(j[0], '%y-%m-%d')
cursor.execute(actualdate)
but it tells me error: name 'DATE_FORMAT' is not defined
import mysql.connector
sql = mysql.connector.connect(host='',user='',password='',db='')
cursor = sql.cursor()
f = open("C:\Cumulus\data\Apr19log.txt","r")
st=[i.strip().split(',') for i in f.readlines()]
actualdate = DATE_FORMAT(j[0], '%y-%m-%d')
cursor.execute(actualdate)
sqllist = "INSERT INTO station_fenelon (variable, date, time,
outside_temp, outside_humidity) VALUES (%s, %s, %s, %s, %s)"
record = [(i+1, j[0], j[1], j[2], j[3]) for i, j in enumerate(st)]
cursor.executemany(sqllist, record)
sql.commit()
error: name 'DATE_FORMAT' is not defined
DATE_FORMAT() is a MySQL function and you call it directly in your python script so you get an error message that is not defined.You should remove this line.
You could use STR_TO_DATE to convert your string to a date
sqllist = "INSERT INTO station_fenelon (variable, date, time,
outside_temp, outside_humidity) VALUES (%s, STR_TO_DATE(%s,'%d/%m/%Y'), %s, %s, %s)"
I have thousands of related CSVs and I want to write their contents to a Postgres table in a way that includes metadata about where each row came from.
I am not clear on how to write the variables I created near the top of my script into the table.
Can anyone advise?
target_directory = Path(sys.argv[1]).resolve()
# FOR THE WAC AND RAC DATASETS
for file in target_directory.rglob('*.csv'):
print(str(file.stem).split('_'))
state = str(file.stem).split('_')[0]
data_category = str(file.stem).split('_')[1]
workforce_segment = str(file.stem).split('_')[2] # THIS IS DIFFERENT FROM THE O-D DATASETS
job_type = str(file.stem).split('_')[3]
year = str(file.stem).split('_')[4]
print('Writing: ' + str(file.name))
# MAKE SURE THIS IS THE RIGHT TABLE FOR THE FILES
cur.execute(create_table_WAC)
with open(file,'r') as file_in:
# INSERT THE DATA IN USING THE COLUMN NAMES....SO YOU CAN ADD YOUR SPLIT STRING INFO ABOVE.....
# MAKE SURE THIS HAS THE RIGHT TABLE NAME IN THE COPY STATEMENT
cur.execute("INSERT INTO opendata_uscensus_usa_lodes_wac (serial_id, state_name, data_category, workforce_segment, job_type, year, w_geocode, C000, CA01, CA02, CA03, CE01, CE02) \
VALUES (%s, state_name, data_category, workforce_segment, job_type, year, %s, %s, %s, %s, %s, %s)")
conn.commit()
conn.close()
As per PEP-249 (Python Database API Specification) which most DB-APIs adhere to including pymssql, cx_oracle, ibm_db, pymysql, sqlite3, and pyodbc, in psycopg2 variables to be binded as parameters in prepared statements would go into the second argument of cur.execute(query, params).
Specifically, combine your file level variables with CSV variables during iteration and pass them as a list or tuple of parameters into execution call. Below uses the csv.DictReader method that builds a dictionary of every row from csv data.
NOTE: below query leaves out primary key, serial_id, which should populate via a sequence in Postgres table.
for file in target_directory.rglob('*.csv'):
print(str(file.stem).split('_'))
# FILE LEVEL VARIABLES
state_name = str(file.stem).split('_')[0]
data_category = str(file.stem).split('_')[1]
workforce_segment = str(file.stem).split('_')[2]
job_type = str(file.stem).split('_')[3]
year = str(file.stem).split('_')[4]
# PREPARED STATEMENT
sql = """INSERT INTO opendata_uscensus_usa_lodes_wac
(state_name, data_category, workforce_segment,
job_type, year, w_geocode, C000, CA01, CA02, CA03, CE01, CE02)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
with open(file,'r') as file_in:
# ITERATE THROUGH FOR CSV VARIABLES
reader = csv.DictReader(file_in)
for row in reader:
cur.execute(sql, (state,data_category,workforce_segment,job_type,year,
row['w_geocode'], row['C000'], row['CA01'],
row['CA02'], row['CA03'], row['CE01'], row['CE02'])
)
conn.commit()
I am trying to insert data to the table that was created earlier using python script. Here is the code I am trying to execute. I want to insert data into table with date as well.
date_today = dt.date.today()
conn = psycopg2.connect(host = serverip, port = port, database = database, user = uid, password = pwd)
cursor = conn.cursor()
cursor.execute("INSERT INTO My_TABLE (Date, Class, Total_students, failed_students, Percent_passed_students) VALUES (date_today, 'Class Name', int1, int2, int3)")
print "Data Inserted successfully"
conn.commit()
conn.close()
Here is the error I see from my job. what am i missing here?
psycopg2.ProgrammingError: column "date_today" does not exist
I created the table using different job with the following query:
cursor.execute("""CREATE TABLE MY_TABL(Date date, Lob varchar(30), Total_Students int, failed_students int, Percent_passed_students int)""")
And the table is created with above five columns.
This line:
cursor.execute("INSERT INTO My_TABLE (Date, Class, Total_students, failed_students, Percent_passed_students) VALUES (date_today, 'Class Name', int1, int2, int3)")
Is the incorrect way to dynamically insert values into a database.
Here's a functional and correct example:
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
And applying it in your case...
cursor.execute("INSERT INTO My_TABLE VALUES (%s, %s, %s, %s, %s)", (date_today, 'Class Name', int1, int2, int3))
I am scraping a website and getting the companies details from it, Now I trying to store the data into database. But I am getting some error like
raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1054, "Unknown column 'companyaddress' in 'field list'")
Here is my code
for d in companydetail:
lis = d.find_all('li')
companyname = lis[0].get_text().strip()
companyaddress = lis[1].get_text().strip()
companycity = lis[2].get_text().strip()
try:
companypostalcode = lis[3].get_text().strip()
companypostalcode = companypostalcode.replace(",","")
except:
companypostalcode = lis[3].get_text().strip()
try:
companywebsite = lis[4].get_text().strip()
except IndexError:
companywebsite = 'null'
print (companyname)
print (companyaddress)
print (companycity)
print (companypostalcode)
print (companywebsite)
try:
with connection.cursor() as cursor:
print ('saving to db')
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
connection.commit()
connection.close()
I am getting my data which I want but it I am not able to store data into database.
The result which I get while print (companyname) and print (campanyaddress) is :
NINGBO BOIGLE DIGITAL TECHNOLOGY CO.,LTD.
TIANYUAN INDUSTRIAL ZONE CIXI NINGBO
ZHEJIANGNINGBO
315325
http://www.boigle.com.cn
You cannot simply use variable names inside a query string as you do:
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
Instead, pass your variables into the query making it parameterized:
params = (companyname, companyaddress, companycity, companypostalcode, companywebsite)
cursor.execute("""
INSERT INTO
company
(companyname, address, city, pincode, website)
VALUES
(%s, %s, %s, %s, %s)
""", params)
In
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
the values in the second bracket are interpreted as table fields, rather than as python variables. Try
cursor.execute("""INSERT INTO company(
companyname,address,city,pincode,website)
VALUES (%s, %s, %s, %s, %s)""",
(companyname, companyaddress, companycity,
companypostalcode, companywebsite))
instead. You may also want to consult the docs on that.