I am trying to write the data frame into the SQL Server Table. My code:
conn = pymssql.connect(host="Dev02", database="DEVDb")
cur = conn.cursor()
query = "INSERT INTO dbo.SCORE_TABLE VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
cur.executemany(query, df_sql)
conn.commit()
cur.close()
conn.close()
The dimension of the df_sql is (5860, 20) i.e. the number of columns in the data frame is same as the number of columns in the SQL Server Table. Still I am getting following error:
ValueError: more placeholders in sql than params available
UPDATED BELOW
As per one of the comments, I tried using turbodbc as below:
conn = turbodbc.connect(driver="{SQL Server}", server="Dev02", Database="DEVDb")
conn.use_async_io = True
cur = conn.cursor()
query = "INSERT INTO dbo.STG_CONTACTABILITY_SCORE VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
cur.executemany(query, df_sql.values)
cur.commit()
cur.close()
conn.close()
I am getting following error:
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()
I don't get it. What is wrong here. I see df_sql.values and I don't find anything wrong.
The first row of ndarray is as below:
[nan 'DUSTIN HOPKINS' 'SOUTHEAST MISSOURI STATE UNIVERSITY' 13.0
'5736512217' None None 'Monday' '8:00AM' '9:00AM' 'Summer' None None None
None '2017-12-22 10:39:30.626331' 'Completed' None '1-11KUFFZ'
'Central Time Zone']
I think you just need to specify each column name and don't forget the table must have the id field to charge the data frame index:
conn = pymssql.connect(host="Dev02", database="DEVDb")
cur = conn.cursor()
query = """INSERT INTO dbo.SCORE_TABLE(index, column1, column2, ..., column20)
VALUES (?, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s, %s)"""
cur.executemany(query, df_sql)
conn.commit()
cur.close()
conn.close()
Ok I have been using pandas and I exported the last data frame to csv like:
df.to_csv('new_file_name.csv', sep=',', encoding='utf-8')
Then I just used pyobdc and BULK INSERT Transact-SQL like:
import pyodbc
conn = pyodbc.connect(DRIVER='{SQL Server}', Server='server_name', Database='Database_name', trusted_connection='yes')
cur = conn.cursor()
cur.execute("""BULK INSERT table_name
FROM 'C:\\Users\\folders path\\new_file_name.csv'
WITH
(
CODEPAGE = 'ACP',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)""")
conn.commit()
cur.close()
conn.close()
It was a second to charge 15314 rows into SQL Server. I hope this gives you an idea.
If i understand correctly you want to use DataFrame.to_sql() method:
df_sql.to_sql('dbo.SCORE_TABLE', conn, index=False, if_exists='append')
Possibly executemany treats each row in the ndarray from your df.values call as one item since there are no comma separators between values. Hence, the placeholders outnumber actual binded values and you receive the mismatch error.
Consider converting array to a tuple of tuples (or lists of lists/tuple of lists/list of tuples) and then pass that object into executemany:
query = "INTO dbo.SCORE_TABLE VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
sql_data = tuple(map(tuple, df.values))
cur.executemany(query, sql_data)
cur.commit()
This works for me-
insert_query = """INSERT INTO dbo.temptable(CHECK_TIME, DEVICE, METRIC, VALUE, TOWER, LOCATION, ANOMALY, ANOMALY_SCORE, ANOMALY_SEVERITY)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"""
write_data = tuple(map(tuple, data_frame.values))
cursor.executemany(insert_query, write_data)
con.commit()
cursor.close()
con.close()
Related
All,
I am using mysql.connector to connect and perform DB operation from my python code. I started getting the below error the moment I added ON DUPLICATE KEY UPDATE query.
Failed to insert into MySQL table 1064 (42000): You have an error in
your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near '''),
Assigned_to=values(''), Assignment_group=values('Updated'),
Company=values(' at line 1
My Code that performs the operation is
mycursor = connection.cursor()
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=values(%s), Assigned_to=values(%s), Assignment_group=values(%s), Company=values(%s), Serial_number=values(%s)"
val = (
data[0]["Asset tag"],
data[0]["Assigned to"],
"Temp",
data[0]["Company"],
data[0]["CPU core count"],
data[0]["CPU count"],
data[0]["CPU manufacturer"],
data[0]["CPU speed (MHz)"],
data[0]["CPU type"],
data[0]["Disk space (GB)"],
data[0]["DNS Domain"],
data[0]["Manufacturer"],
data[0]["Name"],
data[0]["Operating System"],
data[0]["OS Domain"],
data[0]["OS Service Pack"],
data[0]["OS Version"],
data[0]["RAM (MB)"],
data[0]["Serial number"],
data[0]["Description"],
data[0]["Asset tag"],
data[0]["Assigned to"],
"Updated",
data[0]["Company"],
data[0]["Serial number"],
)
print(val)
mycursor.execute(sql, (val))
connection.commit()
Update:
The below also doesn't work
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(%s), Assigned_to=VALUES(%s), Assignment_group=VALUES(%s), Company=VALUES(%s), Serial_number=VALUES(%s)"
Referred to these link but didn't help:
https://topherpedersen.blog/2019/12/10/how-to-use-on-duplicate-key-update-with-python-mysql-and-mysql-connector/
MySQL Python ON DUPLICATE KEY UPDATE VALUES
try this in sql:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(Asset_tag), Assigned_to=VALUES(Assigned_to), Assignment_group=VALUES(Assignment_group), Company=VALUES(Company), Serial_number=VALUES(Serial_number);"
Note: Remove extra keys you are sending in val or you can skip use of VALUES(col_name) and directly assign value..
Refer to this for proper use of ON DUPLICATE KEY: https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html
In assignment value expressions in the ON DUPLICATE KEY UPDATE clause, you can use the VALUES(col_name)
The value placeholders need to be separated with commas too, not just spaces.
There is an issue with the way VALUES function is used in ON DUPLICATE KEY UPDATE. Here is a snippet from documentation explaining the same.
In assignment value expressions in the ON DUPLICATE KEY UPDATE clause, you can use the VALUES(col_name) function to refer to column values from the INSERT portion of the INSERT ... ON DUPLICATE KEY UPDATE statement. In other words, VALUES(col_name) in the ON DUPLICATE KEY UPDATE clause refers to the value of col_name that would be inserted, had no duplicate-key conflict occurred. This function is especially useful in multiple-row inserts.
Link to documentation
Solutions:
We can either avoid the usage of VALUES function and update the values directly as below:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=%s, Assigned_to=%s, Assignment_group=%s, Company=%s, Serial_number=%s"
Pass on the exact field names from insert statement as below:
sql = "INSERT INTO server(Asset_tag,Assigned_to,Assignment_group,Company,CPU_core_count,CPU_count,CPU_manufacturer,CPU_speed,CPU_type,Diskspace,DNS_Domain,Manufacturer,Name,Operating_System,OS_Domain,OS_Service_Pack,OS_Version,RAM,Serial_number,Description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) \
ON DUPLICATE KEY \
UPDATE Asset_tag=VALUES(Asset_tag), Assigned_to=VALUES(Assigned_to), Assignment_group=%s, Company=VALUES(Company), Serial_number=VALUES(Serial_number)"
I'm an Mechanical Engineer with a little experience in C and C++ languages and no experience in Python or SQL labguages.
Recently, I started to work on stock prices analyses in order to optimise my portfolio.
I started with an Excel file and several VBA macros. It works quite good but is very slow.
So, I'm now trying to step up and set up a proper "stock prices" database on my server (based on this post).
In the "stock_prices" database, there is an 'exchange' table that stores all the markets identification codes (MIC) and related information.
In order to update the exchange table, a python script will be launched once a month and it includes the below Python / SQL statements.
import pymysql.cursors
conn = pymysql.connect(host='localhost', user='xxx', password='yyy', database='stock_prices')
cursor = conn.cursor()
mic_data = pd.read_excel('https://www.iso20022.org/sites/default/files/'+ str(date.year) + '-' + str(format(date.month, '02d')) + '/ISO10383_MIC.xls', sheet_name='MICs List by Country', na_filter=False)
mic_data.columns = ['country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acronym', 'city', 'website', 'status_date', 'status', 'creation_date', 'comments']
for row in mic_data.itertuples(index=False):
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE ", row)
Unfortunately, the "Insert INTO" statement returns an error :
1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1"
I have read several posts on this error (which seems quite common) and have tried the following modifications that all returned the same error:
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE", row)
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE;", row)
cursor.execute("""INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE""", row)
I have also tried this modification :
cursor.execute("INSERT INTO exchange('country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acronym', 'city', 'website', 'status_date', 'status', 'creation_date', 'comments') VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE", row)
with a slightly different result :
1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''country', 'iso_country_code', 'mic', 'operating_mic', 'mic_type', 'name', 'acro' at line 1"
However, I don't see anything special near 'country'.
Could anybody suggest modifications to the "INSERT INTO" statement that I could try ?
Best Regards,
Edit 31/03/2020 : Statement after correction
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE country=%s, iso_country_code=%s, mic=%s, mic_type=%s, name=%s, acronym=%s, city=%s, website=%s, status_date=%s, status=%s, creation_date=%s, comments=%s", (row.country, row.iso_country_code, row.mic, row.operating_mic, row.mic_type, row.name, row.acronym, row.city, row.website, row.status_date, row.status, row.creation_date, row.comments,row.country, row.iso_country_code, row.mic, row.mic_type, row.name, row.acronym, row.city, row.website, row.status_date, row.status, row.creation_date, row.comments))
There is no issue with the code I checked it the only issue you have is that ON DUPLICATE KEY UPDATE requires a condition take a look here
Edit: you cannot have strings regarding column fields you can use (`) backtick or just leave them as they are in the query below
Your query should look like this now:
INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments)
VALUES('ALBANIA', 'AL', 'XALS', 'XALS', 'O', 'ALBANIA SECURITIES EXCHANGE', 'ALSE', 'TIRANA', 'WWW.ALSE.AL', 'APRIL 2019', 'ACTIVE', 'APRIL 2019', 'ALL CLASSES OF SECURITIES.') ON DUPLICATE KEY UPDATE operating_mic='XALN'
Python:
cursor.execute("INSERT INTO exchange(`country`, `iso_country_code`, `mic`, `operating_mic`, `mic_type`, `name`, `acronym`, `city`, `website`, `status_date`, `status`, `creation_date`, `comments`) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE operating_mic='XALN'", row)
OR
cursor.execute("INSERT INTO exchange(country, iso_country_code, mic, operating_mic, mic_type, name, acronym, city, website, status_date, status, creation_date, comments) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) ON DUPLICATE KEY UPDATE operating_mic='XALN'", row)
I have the python code below that generically inserts values in to a mysql db. It checks the length of my list field_split and builds a variable string var_string. Where I am stuck is that I want to modify this to support ON DUPLICATE KEY UPDATE user=user_id_var. If I was building a long string I would know how to do this but couldn't figure out a way to do it by passing two values to cursor.execute.
var_string = ', '.join(['%s'] * len(field_split))
query_string = 'INSERT into ' + table_name
query_string = query_string + ' VALUES (%s);' % var_string #query_string = 'INSERT INTO tbl_rtp_GET_FBA_ESTIMATED_FBA_FEES_TXT_DATA VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);'
cursor.execute(query_string, field_split)
db.commit()
Just construct the query and pass all the parameters as a single sequence to cursor.execute(). You can also use str.format() to avoid multiple string concatenations, which can be slow:
var_string = ', '.join(['%s'] * len(field_split))
query_string = 'INSERT into {} VALUES ({}) ON DUPLICATE KEY UPDATE user=%s'.format(
table_name, var_string)
cursor.execute(query_string, field_split + [user_id_var])
db.commit()
I have a csv file that I am loading which you can see linked below as well as the error output I am getting. I cannot figure out why this error is happening. Any help is appreciated.
def url_store():
run_urlcrazy()
url_file = open('url_csv')
csv_reader = csv.reader(url_file)
cursor = db.cursor()
for row in csv_reader:
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", str(row))
db.commit()
cursor.close()
Error Screenshot
CSV file
The query expects 8 parameters (indicated by (%s, %s, %s, %s, %s, %s, %s, %s)), however you only provide a single parameter, namely str(row).
If you're sure that row contains 8 string values, you can use
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", *row)
or just go with
cursor.execute("INSERT INTO scanresults(typotype,squatdomain, ip, id, domaincontact, mx, originaldomain, ipcontact) \
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", (row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7]))
My Sounds table has 7 columns: Start_Time, End_Time, Salience_Label, Class_label, Bitrate, Bitdepth, Samplerate.
I want to insert some values into this table with the command
cursor.execute("INSERT INTO Sounds VALUES (%s, %s, %s, %s, %s, %s, %s)",(start_time, end_time, salience_label, class_label, samplerate, bitrate, bitdepth))
try:
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
.... doing staff for getting values for my variables ...
cursor.execute("INSERT INTO Sounds VALUES (%s, %s, %s, %s, %s, %s, %s)",(start_time, end_time, salience_label, class_label, samplerate, bitrate, bitdepth))
print "Connected!\n"
except:
print "I am unable to connect to the database"
cursor.close()
conn.close()
print('Close conection')
While testing do not catch exceptions. Make the parameters a single tuple as Psycopg will adapt it to a record. Use mogrify to check what is being sent to the server:
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
insert = "insert into Sounds values %s"
parameters = (
start_time, end_time, salience_label,
class_label, samplerate, bitrate, bitdepth
)
print cursor.mogrify(insert, (parameters,))
cursor.execute(insert, (parameters,))
conn.commit()
cursor.close()
conn.close()
BTW, the good practice is to name the columns which will receive the data like in:
insert into t (col_a, col_b) values (1, 'a')
That will avoid some problems.