I am scraping a website and getting the companies details from it, Now I trying to store the data into database. But I am getting some error like
raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1054, "Unknown column 'companyaddress' in 'field list'")
Here is my code
for d in companydetail:
lis = d.find_all('li')
companyname = lis[0].get_text().strip()
companyaddress = lis[1].get_text().strip()
companycity = lis[2].get_text().strip()
try:
companypostalcode = lis[3].get_text().strip()
companypostalcode = companypostalcode.replace(",","")
except:
companypostalcode = lis[3].get_text().strip()
try:
companywebsite = lis[4].get_text().strip()
except IndexError:
companywebsite = 'null'
print (companyname)
print (companyaddress)
print (companycity)
print (companypostalcode)
print (companywebsite)
try:
with connection.cursor() as cursor:
print ('saving to db')
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
connection.commit()
connection.close()
I am getting my data which I want but it I am not able to store data into database.
The result which I get while print (companyname) and print (campanyaddress) is :
NINGBO BOIGLE DIGITAL TECHNOLOGY CO.,LTD.
TIANYUAN INDUSTRIAL ZONE CIXI NINGBO
ZHEJIANGNINGBO
315325
http://www.boigle.com.cn
You cannot simply use variable names inside a query string as you do:
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
Instead, pass your variables into the query making it parameterized:
params = (companyname, companyaddress, companycity, companypostalcode, companywebsite)
cursor.execute("""
INSERT INTO
company
(companyname, address, city, pincode, website)
VALUES
(%s, %s, %s, %s, %s)
""", params)
In
cursor.execute("INSERT INTO company(companyname,address,city,pincode,website) VALUES (companyname,companyaddress,companycity,companypostalcode,companywebsite)")
the values in the second bracket are interpreted as table fields, rather than as python variables. Try
cursor.execute("""INSERT INTO company(
companyname,address,city,pincode,website)
VALUES (%s, %s, %s, %s, %s)""",
(companyname, companyaddress, companycity,
companypostalcode, companywebsite))
instead. You may also want to consult the docs on that.
Related
I'm trying to import some CSV files to a table on a MySQL database. The CSV files are updated daily and my intention is to use this program in python to automate the process.
The idea is: if the information already exists, I want to update it. If the information doesn't exist, I want to insert the data.
But I'm having this error:
AttributeError
'DictCursor' object has no attribute 'update'
Thanks in advance.
csv_data = csv.reader(open('ATEG_REGIONAL_MG_DADOS_TB_ATIVIDADE.csv', encoding='ISO-8859-15'), delimiter=';')
next(csv_data)
for row in csv_data:
for i, l in enumerate(row):
if row[i] == '':
row[i] = None
cursor.execute('SELECT * FROM atividade WHERE CD_ATIVIDADE=%s', row[0])
if cursor.fetchall():
cursor.update('UPDATE atividade WHERE CD_ATIVIDADE = row[0]'),
else:
cursor.execute('INSERT INTO atividade (CD_ATIVIDADE, NM_ATIVIDADE, ST_ATIVO, COD_USUARIO_INCLUSAO, COD_USUARIO_ALTERACAO, DAT_INCLUSAO, DAT_ALTERACAO, CO_ATIVIDADE_REZOLVE, ROWID, FLG_SAFRA, FLG_PRODUTIVO, FLG_TIPO_ATIVIDADE, FLG_INDICADOR_ISA) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)', row)
# close the connection to the database.
db.commit()
cursor.close()
print("Imported!")
If you are using psycopg2, there is no cursor.update() function present. Try cursor.execute() instead.
Also row[0] is considered as a string in your query. So, change it to:
cursor.execute('UPDATE atividade WHERE CD_ATIVIDADE = ' + row[0])
Seems like you are confusing two different libraries.
import MySQLdb and from flask_mysqldb import MySQL are two different libraries.
Since you are using flask adding this line app.config['MYSQL_CURSORCLASS'] = 'DictCursor' and then calling the cursor cursor=db.connection.cursor() should solve your problem.
Taken from the official git page
import mysql.connector
def add_features_to_db(stockname, timeframe, date, feature):
try:
conn = mysql.connector.connect(
user='root', password='', host='localhost', database='fx003')
cursor = conn.cursor()
dbtable = stockname + timeframe
mySql_insert_query = """INSERT INTO `%s` (date, trend) VALUES ( `%s`, `%s` )"""
record = (dbtable, date, feature)
cursor.execute(mySql_insert_query, record)
conn.commit()
print("Record inserted successfully")
except mysql.connector.Error as error:
print("Failed to insert into MySQL table {}".format(error))
finally:
if conn.is_connected():
cursor.close()
conn.close()
print("MySQL connection is closed")
add_features_to_db("aud-cad", "_30mins", "2021-09-24 21:00:00", "Short")
I have the code above and giving me the below error:
Failed to insert into MySQL table 1146 (42S02): Table 'fx003.'aud-cad_30mins'' doesn't exist
aud-cad_30mins table does exist and an insert query like below doing its job:
mySql_insert_query = """INSERT INTO aud-cad_30mins (date, trend) VALUES ( "2021-09-24 21:00:00","Short" )"""
So when I try to use variables in the query, it gives the error. Why the table name getting unwanted quotes? Checked several tutorials but couldn't find a solution, any ideas?
The table name should be hardcoded in the query string instead of having it there as a placeholder %s, which is meant for the values to be inserted. So if you have the table name in the variable, you can replace it via format() before calling cursor.execute()
dbtable = stockname + timeframe
mySql_insert_query = """INSERT INTO {} (date, trend) VALUES ( %s, %s )""".format(dbtable)
see the examples in the docs
edit: as Bill mentioned in the comment, dont add the backticks around the %s placeholders.
I have thousands of related CSVs and I want to write their contents to a Postgres table in a way that includes metadata about where each row came from.
I am not clear on how to write the variables I created near the top of my script into the table.
Can anyone advise?
target_directory = Path(sys.argv[1]).resolve()
# FOR THE WAC AND RAC DATASETS
for file in target_directory.rglob('*.csv'):
print(str(file.stem).split('_'))
state = str(file.stem).split('_')[0]
data_category = str(file.stem).split('_')[1]
workforce_segment = str(file.stem).split('_')[2] # THIS IS DIFFERENT FROM THE O-D DATASETS
job_type = str(file.stem).split('_')[3]
year = str(file.stem).split('_')[4]
print('Writing: ' + str(file.name))
# MAKE SURE THIS IS THE RIGHT TABLE FOR THE FILES
cur.execute(create_table_WAC)
with open(file,'r') as file_in:
# INSERT THE DATA IN USING THE COLUMN NAMES....SO YOU CAN ADD YOUR SPLIT STRING INFO ABOVE.....
# MAKE SURE THIS HAS THE RIGHT TABLE NAME IN THE COPY STATEMENT
cur.execute("INSERT INTO opendata_uscensus_usa_lodes_wac (serial_id, state_name, data_category, workforce_segment, job_type, year, w_geocode, C000, CA01, CA02, CA03, CE01, CE02) \
VALUES (%s, state_name, data_category, workforce_segment, job_type, year, %s, %s, %s, %s, %s, %s)")
conn.commit()
conn.close()
As per PEP-249 (Python Database API Specification) which most DB-APIs adhere to including pymssql, cx_oracle, ibm_db, pymysql, sqlite3, and pyodbc, in psycopg2 variables to be binded as parameters in prepared statements would go into the second argument of cur.execute(query, params).
Specifically, combine your file level variables with CSV variables during iteration and pass them as a list or tuple of parameters into execution call. Below uses the csv.DictReader method that builds a dictionary of every row from csv data.
NOTE: below query leaves out primary key, serial_id, which should populate via a sequence in Postgres table.
for file in target_directory.rglob('*.csv'):
print(str(file.stem).split('_'))
# FILE LEVEL VARIABLES
state_name = str(file.stem).split('_')[0]
data_category = str(file.stem).split('_')[1]
workforce_segment = str(file.stem).split('_')[2]
job_type = str(file.stem).split('_')[3]
year = str(file.stem).split('_')[4]
# PREPARED STATEMENT
sql = """INSERT INTO opendata_uscensus_usa_lodes_wac
(state_name, data_category, workforce_segment,
job_type, year, w_geocode, C000, CA01, CA02, CA03, CE01, CE02)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
with open(file,'r') as file_in:
# ITERATE THROUGH FOR CSV VARIABLES
reader = csv.DictReader(file_in)
for row in reader:
cur.execute(sql, (state,data_category,workforce_segment,job_type,year,
row['w_geocode'], row['C000'], row['CA01'],
row['CA02'], row['CA03'], row['CE01'], row['CE02'])
)
conn.commit()
I am new to python and working on using the psycopg2 to insert data in postgres database. I am trying to insert items but get the error message
"Psycopg2.ProgrammingError: syntax error at or near "cup"
LINE 1: INSERT INTO store VALUES(7,10.5,coffee cup)
with the ^ next to coffee cup. I am assuming the order is wrong but i thought you could enter it this way as long as you specified the values.
Here is the code.
import psycopg2
def create_table():
conn=psycopg2.connect("dbname='db1' user='postgres' password='postgress123' host='localhost' port='5432'")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS store (item TEXT, quantity INTEGER, price REAL)")
conn.commit()
conn.close()
def insert(quantity, price, item):
conn=psycopg2.connect("dbname='db1' user='postgres' password='postgress123' host='localhost' port='5432'")
cur=conn.cursor()
cur.execute("INSERT INTO store VALUES(%s,%s,%s)" % (quantity, price, item))
conn.commit()
conn.close()
create_table()
insert(7, 10.5, 'coffee cup')
Remember to always use the second argument of the execute command to pass the variables, as stated here.
Also, use the name of the fields in your syntax:
cur.execute("INSERT INTO store (item, quantity, price) VALUES (%s, %s, %s);", (item, quantity, price))
That should do the trick.
Problem in your case is coffee cup parameter value is considered as string but psycopg2 accept the value in single quote.
Basically as per my understanding when we create SQL query for psycopg2 it ask for single quote for data parameters [if you have given double quote for query start and end]
In your case you have given double quote for Query Start and end so you need to give single quote for the parameters.
My Observation is you provide single quote for each data paramater in psycopg2
import psycopg2
def create_table():
conn=psycopg2.connect("dbname='db1' user='postgres' password='postgress123' host='localhost' port='5432'")
cur=conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS store (item TEXT, quantity INTEGER, price REAL)")
conn.commit()
conn.close()
def insert(quantity, price, item):
conn=psycopg2.connect("dbname='db1' user='postgres' password='postgress123' host='localhost' port='5432'")
cur=conn.cursor()
#cur.execute("INSERT INTO store VALUES(%s,%s,%s)" % (quantity, price, item))
cur.execute("INSERT INTO store VALUES('%s','%s','%s')" % (quantity, price, item))
conn.commit()
conn.close()
create_table()
insert(7, 10.5, 'coffee cup')
I also faced the very same problem, and after a while troubleshooting the code, I found that I forgot to add commas(,) in the Insert query.
The code that causes the error:
data['query'] = 'insert into contacts (name, contact_no, alternate_contact_no, email_id, address)' \
'values (%s %s %s %s %s)'
As you can see in above code, I forgot to add commas after every '%s'.
The correct code:
data['query'] = 'insert into contacts (name, contact_no, alternate_contact_no, email_id, address)' \
'values (%s, %s, %s, %s, %s)'
Hope, It helps!
I am trying to insert data to the table that was created earlier using python script. Here is the code I am trying to execute. I want to insert data into table with date as well.
date_today = dt.date.today()
conn = psycopg2.connect(host = serverip, port = port, database = database, user = uid, password = pwd)
cursor = conn.cursor()
cursor.execute("INSERT INTO My_TABLE (Date, Class, Total_students, failed_students, Percent_passed_students) VALUES (date_today, 'Class Name', int1, int2, int3)")
print "Data Inserted successfully"
conn.commit()
conn.close()
Here is the error I see from my job. what am i missing here?
psycopg2.ProgrammingError: column "date_today" does not exist
I created the table using different job with the following query:
cursor.execute("""CREATE TABLE MY_TABL(Date date, Lob varchar(30), Total_Students int, failed_students int, Percent_passed_students int)""")
And the table is created with above five columns.
This line:
cursor.execute("INSERT INTO My_TABLE (Date, Class, Total_students, failed_students, Percent_passed_students) VALUES (date_today, 'Class Name', int1, int2, int3)")
Is the incorrect way to dynamically insert values into a database.
Here's a functional and correct example:
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
And applying it in your case...
cursor.execute("INSERT INTO My_TABLE VALUES (%s, %s, %s, %s, %s)", (date_today, 'Class Name', int1, int2, int3))