I am trying to executemany in python with on duplicate key update, with the following script:
# data from a previous query (returns 4 integers in each row)
rows = first_cursor.fetchall()
query="""
INSERT INTO data (a, b, c)
VALUES (%s,%s,%s) ON DUPLICATE KEY UPDATE a=%s
"""
second_cursor.executemany(query,rows)
I'm getting this error:
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 212, in executemany
self.errorhandler(self, TypeError, msg)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
TypeError: not all arguments converted during string formatting
Is this even possible without creating my own loop?
This is a bug in MySQLdb due to the regex that MySQLdb uses to parse INSERT statements:
In /usr/lib/pymodules/python2.7/MySQLdb/cursors.py:
restr = (r"\svalues\s*"
r"(\(((?<!\\)'[^\)]*?\)[^\)]*(?<!\\)?'"
r"|[^\(\)]|"
r"(?:\([^\)]*\))"
r")+\))")
insert_values= re.compile(restr)
Although there have been numerous bug reports about this problem that have been closed as fixed, I was able to reproduce the error in MySQLdb version 1.2.3. (Note the latest version of MySQLdb at the moment is 1.2.4b4.)
Maybe this bug is fixable, I don't really know. But I think it is just the tip of the iceberg -- it points to much more trouble lurking just a little deeper. You could have for instance an INSERT ... SELECT statement with nested SELECT statements with WHERE conditions and parameters sprinkled all about... Making the regex more and more complicated to handle these cases seems to me like a losing battle.
You could use oursql; it does not use regex or string formating. It passes parametrized queries and arguments to the server separately.
When you write sql like following:
sql = insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=%s, count=count+%s'
You will get the following error: TypeError: not all arguments converted during string formatting.
So when you use "ON DUPLICATE KEY UPDATE" in python, you need to write sql like this:
sql = 'insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=values(last_date),count=count+values(count)'
found:
on duplicate key update col1=VALUES(col1), col2=VALUES(col2)
https://hardforum.com/threads/python-mysql-not-all-arguments-converted-during-string-formatting.1367039/
It is a bug of mysqldb as ubuntu said, sightly change the sql then it works:
insert into tb_name(col1, col2) select 1,2 on duplicate key update col1=1
Related
This is the pymssql query I am working with
query = 'INSERT INTO [dbo].[helios_devops_data_curr] ("iipm.l3_it_org", "iipm.it_custodian","iipm.it_executive") VALUES ({}{}{})'.format("'Innovation and Technology'", "'bob tom'", "'bob tom'")
I'm using these values as an example, they are not the real values I'm trying to upload. However the errors are the same:
109, b'There are more columns in the INSERT statement than values specified in the VALUES clause. The number of values in the VALUES clause must match the number of columns specified in the INSERT statement.DB-Lib error message 20018, severity 15:\nGeneral SQL Server error: Check messages from the SQL Server\n')
I'm not sure why these error is occurring as there's clearly 3 columns and 3 values being inserted.
Any help on this would be appreciated
Your format placeholders might need commas between them:
query = 'INSERT INTO [dbo].[helios_devops_data_curr] ("iipm.l3_it_org", "iipm.it_custodian","iipm.it_executive") VALUES ({}, {}, {})'.format("'Innovation and Technology'", "'bob tom'", "'bob tom'")
Looking at the docs, it appears that pymssql might prefer you use C-style string-formatting symbols; maybe something like this:
query = "INSERT INTO dbo.helios_devops_data_curr(iipm.l3_it_org, iipm.it_custodian, iipm.it_executive) VALUES (%s, %s, %s)"
params = ("Innovation and Technology", "bob tom", "bob tom")
cursor.execute(query, params)
Here is another SO question with an example.
I want to read a csv file to insert data into postgres SQL with Python
but I have these error:
cursor.execute(passdata)
psycopg2.IntegrityError: duplicate key value violates unique constraint "prk_constraint_project"
DETAIL: Key (project_code)=(%s) already exists.
My code is:
clinicalCSVINSERT = open(clinicalname, 'r')
reader = csv.reader(clinicalCSVINSERT, delimiter='\t')
passdata = "INSERT INTO project (project_code, program_name ) VALUES ('%s', '%s')";
cursor.execute(passdata)
conn.commit()`
What does this error mean?
Is it possible to have a working script?
The immediate problem with your code is that you are trying to include the literal %s. Since you probably did run it more than once you already have a literal %s in that unique column hence the exception.
It is necessary to pass the values wrapped in an iterable as parameters to the execute method. The %s is just a value place holder.
passdata = """
INSERT INTO project (project_code, program_name )
VALUES (%s, %s)
"""
cursor.execute(passdata, (the_project_code, the_program_name))
Do not quote the %s. Psycopg will do it if necessary.
As your code does not include a loop it will only insert one row from the csv. There are some patterns to insert the whole file. If the requirements allow just use copy_from which is simpler and faster.
A Python API is giving back u"'HOPPE'S No. 9'" as a value for a particular product attribute. I'm then looking to insert it into the DB, also using Python (python-mysqldb), with the following query:
INSERT INTO mytable (rating, Name) VALUES('5.0 (7)', 'HOPPE'S No. 9';
MySQL rejects this, and the suggested approach to handling a single quote in MySQL is to escape it first. This I need to do in Python, so I try:
In [5]: u"'HOPPE'S No. 9'".replace("'", "\'")
Out[5]: u"'HOPPE'S No. 9'"
When I incorporate this in my program, MySQL still rejects it. So I double-escape the apostrophe, and then an insert happens successfully. Thing is, it contains the escape character (so what gets written is 'HOPPE\'S No. 9').
If I need the second escape character, but when I add it gets left in, then how can I handle the escaping without having the escape character included in the string that gets inserted?
Edit: Based on theBjorn's suggestion, tried:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
#cur.execute(queryString)
cur.execute(actualSQL,
(configData["table"], sqlFieldMappingString, sqlFieldValuesString))
but it looks like I'm back to where I was when I was trying to escape using the single escape with .replace():
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''mytable' ('rating, Name, Image, mfg, price, URL') VALUES('\'5.0 (3)\', \'AR-1' at line 1
You should never construct sql that way. Use parameterized code instead:
cursor.execute(
"insert into mytable (rating, name) values (%s, %s);",
("5.0 (7)", "HOPPE'S No. 9")
)
your latest problem is due to the misconception that this is string interpolation, which it isn't (the use of %s is confusing), thus:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
will be wrong. It is possible to construct your sql string, but probably easier to do so in two steps so we don't trip over sql parameter markers looking like string interpolation markers. Assuming you have the values in a tuple named field_values:
params = ["%s"] * len(field_values) # create a list with the correct number of parameter markers
sql = "insert into %s (%s) values (%s)" % ( # here we're using string interpolation, but not with the values
configData["table"],
sqlFieldMappingString,
', '.join(params)
)
if you print sql it should look like my example above. Now you can execute it with:
cursor.execute(sql, field_values)
I am trying to use SQL with prepared statements in Python. Python doesn't have its own mechanism for this so I try to use SQL directly:
sql = "PREPARE stmt FROM ' INSERT INTO {} (date, time, tag, power) VALUES (?, ?, ?, ?)'".format(self.db_scan_table)
self.cursor.execute(sql)
Then later, in the loop:
sql = "EXECUTE stmt USING \'{}\', \'{}\', {}, {};".format(d, t, tag, power)
self.cursor.execute(sql)
And in the loop I get:
MySQL Error [1064]: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''2014-12-25', '12:31:46', 88000000, -6.64' at line 1
What's going on?
Using prepared statements with MySQL in Python is explained e.g at http://zetcode.com/db/mysqlpython/ -- look within that page for Prepared statements.
In your case, that would be, e.g:
sql = ('INSERT INTO {} (date, time, tag, power) VALUES '
'(%s, %s, %s, %s)'.format(self.db_scan_table))
and later, "in the loop" as you put it:
self.cursor.execute(sql, (d, t, tag, power))
with no further string formatting -- the MySQLdb module does the prepare and execute parts on your behalf (and may cache things to avoid repeating work needlessly, etc, etc).
Do consider, depending on the nature of "the loop" you mention, that it's possible that a single call to .execute_many (with a sequence of tuples as the second argument) could take the place of the whole loop (unless you need more processing within that loop beyond just the insertion of data into the DB).
Added: a better alternative nowadays may be to use mysql's own Connector/Python and the explicit prepare=True option in the .cursor() factory -- see http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html . This lets you have a specific cursor on which statements are prepared (with the "more efficient than using PREPARE and EXECUTE" binary protocol, according to that mysql.com page) and another one for statements that are better not prepared; "explicit is better than implicit" is after all one of the principles in "The Zen of Python" (import this from an interactive prompt to read all those principles). mysqldb doing things implicitly (and it seems the current open-source version doesn't use prepared statements) can't be as good an architecture as Connector/Python's more explicit one.
import mysql.connector
db_con=mysql.connector.connect(host='',
database='',
user='',
password='')
cursor = db_con.cursor(prepared=True,)
#cursor = db_con.cursor(prepared=True)#IT MAY HAVE PROBLEM
sql = """INSERT INTO table (xy,zy) VALUES (%s, %s)"""
input=(1,2)
cursor.execute(sql , input)
db_con.commit()
SELECT STMT
sql = """SELECT * FROM TABLE WHERE XY=%s ORDER BY id DESC LIMIT 1 """
ID=1
input=(ID,)
#input=(ID)# IT MAY HAS PROBLEM
cursor.execute(sql, input)
data = cursor.fetchall()
rowsNumber=cursor.rowcount
Python does support prepared statements:
sql = "INSERT INTO {} (date, time, tag, power) VALUES (%s, %s, %s, %s);"
sql = sql.format(self.db_scan_table)
self.cursor.execute(sql, (d, t, tag, power))
(You should ensure self.db_scan_table is not vulnerable to SQL injection)
This assumes your paramstyle is 'format', which it should be for MySQL.
I am having trouble in formatting the list for insertion using psycopg.
Here is a sample of code i am trying to do.
Basically I am just reading data from one table and trying to insert it into another table.
Code:
cur.execute("""select data from \"Table1\" where lat=-20.004189 and lon=-63.848004""")
rows = cur.fetchall()
print rows
cur.execute("""INSERT INTO \"%s\" (data) VALUES (ARRAY%s)""" % (args.tableName,rows)))
The result returned by first select query is like this:
[([6193, 3975, 4960, 5286, 3380, 970, 3328, 3173, 2897, 2457, 2443, 2674, 2172, 2740, 3738, 4907, 3691, 4234, 3651, 3215],)]
When I try to insert this into another table I get the following format error.
cur.execute(cur.mogrify("""INSERT INTO \"%s\" (data) VALUES (%s)""" % (args.tableName,rows)))
psycopg2.ProgrammingError: syntax error at or near "["
LINE 1: INSERT INTO "DUMMY1km" (data) VALUES ([([6193, 3975, 4960, 5...
I tried cur.mogrify, but it does not seem to help.
Please let me know if anyone has a work around for this issue.
Thanks
Adi
I don't think mogrify is needed here. Use executemany and pass rows as the second argument.
cur.executemany(
"""INSERT INTO "%s" (data) VALUES (%%s)""" % (args.tableName),rows)
Using parametrized arguments helps prevent SQL injection.
The table name can not be parametrized, so we do have to use string interpolation to place the table name in the SQL query. %%s gets escapes the percent sign and becomes %s after string interpolation.
By the way, (as a_horse_with_no_name has already pointed out) you can use the INSERT INTO ... SELECT form of INSERT to perform both SQL queries as one:
cur.execute(
"""INSERT INTO %s (data)
SELECT data FROM Table1
WHERE lat=-20.004189 AND lon=-63.848004""" % (args.tableName))
Per the question in the comments, if there are multiple fields, then the SQL becomes:
cur.executemany(
"""INSERT INTO {t} (lat,lon,data1,data2)
VALUES (%s,%s,%s,%s)""".format(t=args.tableName),rows)
(If you use the format method, then you don't have to escape all the other %ss.)