All,
I am using mysql.connector to connect and perform DB operation from my python code. I started getting the below error the moment I added ON DUPLICATE KEY UPDATE query.
Failed to insert into MySQL table 1064 (42000): You have an error in
your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near '''),
Assigned_to=values(''), Assignment_group=values('Updated'),
Company=values(' at line 1
My Code that performs the operation is
mycursor = connection.cursor()
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=values(%s), Assigned_to=values(%s), Assignment_group=values(%s), Company=values(%s), Serial_number=values(%s)"
val = (
data[0]["Asset tag"],
data[0]["Assigned to"],
"Temp",
data[0]["Company"],
data[0]["CPU core count"],
data[0]["CPU count"],
data[0]["CPU manufacturer"],
data[0]["CPU speed (MHz)"],
data[0]["CPU type"],
data[0]["Disk space (GB)"],
data[0]["DNS Domain"],
data[0]["Manufacturer"],
data[0]["Name"],
data[0]["Operating System"],
data[0]["OS Domain"],
data[0]["OS Service Pack"],
data[0]["OS Version"],
data[0]["RAM (MB)"],
data[0]["Serial number"],
data[0]["Description"],
data[0]["Asset tag"],
data[0]["Assigned to"],
"Updated",
data[0]["Company"],
data[0]["Serial number"],
)
print(val)
mycursor.execute(sql, (val))
connection.commit()
Update:
The below also doesn't work
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(%s), Assigned_to=VALUES(%s), Assignment_group=VALUES(%s), Company=VALUES(%s), Serial_number=VALUES(%s)"
Referred to these link but didn't help:
https://topherpedersen.blog/2019/12/10/how-to-use-on-duplicate-key-update-with-python-mysql-and-mysql-connector/
MySQL Python ON DUPLICATE KEY UPDATE VALUES
try this in sql:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(Asset_tag), Assigned_to=VALUES(Assigned_to), Assignment_group=VALUES(Assignment_group), Company=VALUES(Company), Serial_number=VALUES(Serial_number);"
Note: Remove extra keys you are sending in val or you can skip use of VALUES(col_name) and directly assign value..
Refer to this for proper use of ON DUPLICATE KEY: https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html
In assignment value expressions in the ON DUPLICATE KEY UPDATE clause, you can use the VALUES(col_name)
The value placeholders need to be separated with commas too, not just spaces.
There is an issue with the way VALUES function is used in ON DUPLICATE KEY UPDATE. Here is a snippet from documentation explaining the same.
In assignment value expressions in the ON DUPLICATE KEY UPDATE clause, you can use the VALUES(col_name) function to refer to column values from the INSERT portion of the INSERT ... ON DUPLICATE KEY UPDATE statement. In other words, VALUES(col_name) in the ON DUPLICATE KEY UPDATE clause refers to the value of col_name that would be inserted, had no duplicate-key conflict occurred. This function is especially useful in multiple-row inserts.
Link to documentation
Solutions:
We can either avoid the usage of VALUES function and update the values directly as below:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=%s, Assigned_to=%s, Assignment_group=%s, Company=%s, Serial_number=%s"
Pass on the exact field names from insert statement as below:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(Asset_tag), Assigned_to=VALUES(Assigned_to), Assignment_group=%s, Company=VALUES(Company), Serial_number=VALUES(Serial_number)"
Related
In python, I am trying to insert some data into my table;
try:
cursor.execute("INSERT INTO covid_testtable (pid, age, state, city, notes, backnotes, type, nationality, status, announced, changed) " +
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, STR_TO_DATE(%s, '%%d/%%m/%%Y'), STR_TO_DATE(%s, '%%d/%%m/%%Y'))",
(pid, age, item["detectedstate"], item["detectedcity"], item["notes"], item["backupnotes"], item["typeoftransmission"],
item["nationality"], item["currentstatus"], dateannounced, statuschanged));
except ValueError as verr:
print(item["dateannounced"], verr);
break;
This works till a point where one of the date columns have a empty entry.
pymysql.err.InternalError: (1411, "Incorrect datetime value: '' for function str_to_date"
I tried to change the value of the empty date to "00/00/0000" however it seems the issue is with the str_to_date() function.
I am fine with inserting NULL value however, not managing to insert.
The valid values which are succesfully inserted are of this format "30/01/2020" -> "%d/%m/%Y"
My table schema is as below:
create table covid_testtable ( pid int NOT NULL, age int, state VARCHAR(255),
city VARCHAR(255), notes VARCHAR(255), backnotes VARCHAR(255), type VARCHAR(255),
nationality VARCHAR(255), status VARCHAR(255), announced DATE default NULL,
changed DATE default NULL, PRIMARY KEY (pid));
EDIT 2: Selected Answer solved this issue:
def validate_date(val):
try:
return datetime.strptime(val, '%d/%m/%Y').strftime('%Y-%m-%d');
except ValueError as verr:
return None;
dateannounced = validate_date(item["dateannounced"]);
statuschanged = validate_date(item["statuschangedate"]);
try:
cursor.execute("INSERT INTO covid_testtable (pid, age, state, city, notes, backnotes, type, nationality, status, announced, changed) " +
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)",
(pid, age, item["detectedstate"], item["detectedcity"],
item["notes"], item["backupnotes"], item["typeoftransmission"],
item["nationality"], item["currentstatus"], dateannounced,
statuschanged));
Rather than processing the dates in the query, you could pre-process them in your python code. For example:
from datetime import datetime
dstr = ''
try:
dateannounced = datetime.strptime(dstr, '%d/%m/%Y').strftime('%Y-%m-%d')
except ValueError:
dateannounced = None
Then your query simply becomes
cursor.execute("INSERT INTO covid_testtable (pid, age, state, city, notes, backnotes, type, nationality, status, announced, changed) " +
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)",
(pid, age, item["detectedstate"], item["detectedcity"],
item["notes"], item["backupnotes"], item["typeoftransmission"],
item["nationality"], item["currentstatus"], dateannounced,
statuschanged));
Check whether the string is empty with if().
cursor.execute("""
INSERT INTO covid_testtable (pid, age, state, city, notes, backnotes, type, nationality, status, announced, changed)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s,
IF(%s = '', NULL, STR_TO_DATE(%s, '%%d/%%m/%%Y')),
IF(%s = '', NULL, STR_TO_DATE(%s, '%%d/%%m/%%Y')))""",
(pid, age, item["detectedstate"], item["detectedcity"], item["notes"], item["backupnotes"], item["typeoftransmission"],
item["nationality"], item["currentstatus"], dateannounced, dateannounced, statuschanged, statuschanged));
Note that I had to repeat dateannounced and statuschanged in the parameter list, since I'm substituting them into the IF functions in addition to the STR_TO_DATE functions.
I'm an Mechanical Engineer with a little experience in C and C++ languages and no experience in Python or SQL labguages.
Recently, I started to work on stock prices analyses in order to optimise my portfolio.
I started with an Excel file and several VBA macros. It works quite good but is very slow.
So, I'm now trying to step up and set up a proper "stock prices" database on my server (based on this post).
In the "stock_prices" database, there is an 'exchange' table that stores all the markets identification codes (MIC) and related information.
In order to update the exchange table, a python script will be launched once a month and it includes the below Python / SQL statements.
import pymysql.cursors
conn = pymysql.connect(host='localhost', user='xxx', password='yyy', database='stock_prices')
cursor = conn.cursor()
mic_data = pd.read_excel('https://www.iso20022.org/sites/default/files/'+ str(date.year) + '-' + str(format(date.month, '02d')) + '/ISO10383_MIC.xls', sheet_name='MICs List by Country', na_filter=False)
mic_data.columns = ['country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acronym', 'city', 'website', 'status_date', 'status', 'creation_date', 'comments']
for row in mic_data.itertuples(index=False):
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE ", row)
Unfortunately, the "Insert INTO" statement returns an error :
1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1"
I have read several posts on this error (which seems quite common) and have tried the following modifications that all returned the same error:
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE", row)
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE;", row)
cursor.execute("""INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE""", row)
I have also tried this modification :
cursor.execute("INSERT INTO exchange('country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acronym', 'city', 'website', 'status_date', 'status', 'creation_date', 'comments') VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE", row)
with a slightly different result :
1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acro' at line 1"
However, I don't see anything special near 'country'.
Could anybody suggest modifications to the "INSERT INTO" statement that I could try ?
Best Regards,
Edit 31/03/2020 : Statement after correction
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE country=%s, iso_country_code=%s, mic=%s, mic_type=%s, name=%s, acronym=%s, city=%s, website=%s, status_date=%s, status=%s, creation_date=%s, comments=%s", (row.country, row.iso_country_code, row.mic, row.operating_mic, row.mic_type, row.name, row.acronym, row.city, row.website, row.status_date, row.status, row.creation_date, row.comments,row.country, row.iso_country_code, row.mic, row.mic_type, row.name, row.acronym, row.city, row.website, row.status_date, row.status, row.creation_date, row.comments))
There is no issue with the code I checked it the only issue you have is that ON DUPLICATE KEY UPDATE requires a condition take a look here
Edit: you cannot have strings regarding column fields you can use (`) backtick or just leave them as they are in the query below
Your query should look like this now:
INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments)
VALUES('ALBANIA', 'AL', 'XALS', 'XALS', 'O', 'ALBANIA SECURITIES EXCHANGE', 'ALSE', 'TIRANA', 'WWW.ALSE.AL', 'APRIL 2019', 'ACTIVE', 'APRIL 2019', 'ALL CLASSES OF SECURITIES.') ON DUPLICATE KEY UPDATE operating_mic='XALN'
Python:
cursor.execute("INSERT INTO exchange(`country`, `iso_country_code`, `mic`, `operating_mic`, `mic_type`, `name`, `acronym`, `city`, `website`, `status_date`, `status`, `creation_date`, `comments`) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE operating_mic='XALN'", row)
OR
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE operating_mic='XALN'", row)
I have the python code below that generically inserts values in to a mysql db. It checks the length of my list field_split and builds a variable string var_string. Where I am stuck is that I want to modify this to support ON DUPLICATE KEY UPDATE user=user_id_var. If I was building a long string I would know how to do this but couldn't figure out a way to do it by passing two values to cursor.execute.
var_string = ', '.join(['%s'] * len(field_split))
query_string = 'INSERT into ' + table_name
query_string = query_string + ' VALUES (%s);' % var_string #query_string = 'INSERT INTO tbl_rtp_GET_FBA_ESTIMATED_FBA_FEES_TXT_DATA VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);'
cursor.execute(query_string, field_split)
db.commit()
Just construct the query and pass all the parameters as a single sequence to cursor.execute(). You can also use str.format() to avoid multiple string concatenations, which can be slow:
var_string = ', '.join(['%s'] * len(field_split))
query_string = 'INSERT into {} VALUES ({}) ON DUPLICATE KEY UPDATE user=%s'.format(
table_name, var_string)
cursor.execute(query_string, field_split + [user_id_var])
db.commit()
I am trying to write the data frame into the SQL Server Table. My code:
conn = pymssql.connect(host="Dev02", database="DEVDb")
cur = conn.cursor()
query = "INSERT INTO dbo.SCORE_TABLE VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
cur.executemany(query, df_sql)
conn.commit()
cur.close()
conn.close()
The dimension of the df_sql is (5860, 20) i.e. the number of columns in the data frame is same as the number of columns in the SQL Server Table. Still I am getting following error:
ValueError: more placeholders in sql than params available
UPDATED BELOW
As per one of the comments, I tried using turbodbc as below:
conn = turbodbc.connect(driver="{SQL Server}", server="Dev02", Database="DEVDb")
conn.use_async_io = True
cur = conn.cursor()
query = "INSERT INTO dbo.STG_CONTACTABILITY_SCORE VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
cur.executemany(query, df_sql.values)
cur.commit()
cur.close()
conn.close()
I am getting following error:
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()
I don't get it. What is wrong here. I see df_sql.values and I don't find anything wrong.
The first row of ndarray is as below:
[nan 'DUSTIN HOPKINS' 'SOUTHEAST MISSOURI STATE UNIVERSITY' 13.0
'5736512217' None None 'Monday' '8:00AM' '9:00AM' 'Summer' None None None
None '2017-12-22 10:39:30.626331' 'Completed' None '1-11KUFFZ'
'Central Time Zone']
I think you just need to specify each column name and don't forget the table must have the id field to charge the data frame index:
conn = pymssql.connect(host="Dev02", database="DEVDb")
cur = conn.cursor()
query = """INSERT INTO dbo.SCORE_TABLE(index, column1, column2, ..., column20)
VALUES (?, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s, %s)"""
cur.executemany(query, df_sql)
conn.commit()
cur.close()
conn.close()
Ok I have been using pandas and I exported the last data frame to csv like:
df.to_csv('new_file_name.csv', sep=',', encoding='utf-8')
Then I just used pyobdc and BULK INSERT Transact-SQL like:
import pyodbc
conn = pyodbc.connect(DRIVER='{SQL Server}', Server='server_name', Database='Database_name', trusted_connection='yes')
cur = conn.cursor()
cur.execute("""BULK INSERT table_name
FROM 'C:\\Users\\folders path\\new_file_name.csv'
WITH
(
CODEPAGE = 'ACP',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)""")
conn.commit()
cur.close()
conn.close()
It was a second to charge 15314 rows into SQL Server. I hope this gives you an idea.
If i understand correctly you want to use DataFrame.to_sql() method:
df_sql.to_sql('dbo.SCORE_TABLE', conn, index=False, if_exists='append')
Possibly executemany treats each row in the ndarray from your df.values call as one item since there are no comma separators between values. Hence, the placeholders outnumber actual binded values and you receive the mismatch error.
Consider converting array to a tuple of tuples (or lists of lists/tuple of lists/list of tuples) and then pass that object into executemany:
query = "INTO dbo.SCORE_TABLE VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
sql_data = tuple(map(tuple, df.values))
cur.executemany(query, sql_data)
cur.commit()
This works for me-
insert_query = """INSERT INTO dbo.temptable(CHECK_TIME, DEVICE, METRIC, VALUE, TOWER, LOCATION, ANOMALY, ANOMALY_SCORE, ANOMALY_SEVERITY)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"""
write_data = tuple(map(tuple, data_frame.values))
cursor.executemany(insert_query, write_data)
con.commit()
cursor.close()
con.close()
I have a csv file that I am loading which you can see linked below as well as the error output I am getting. I cannot figure out why this error is happening. Any help is appreciated.
def url_store():
run_urlcrazy()
url_file = open('url_csv')
csv_reader = csv.reader(url_file)
cursor = db.cursor()
for row in csv_reader:
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", str(row))
db.commit()
cursor.close()
Error Screenshot
CSV file
The query expects 8 parameters (indicated by (%s, %s, %s, %s, %s, %s, %s, %s)), however you only provide a single parameter, namely str(row).
If you're sure that row contains 8 string values, you can use
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", *row)
or just go with
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", (row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7]))