Develop Reference

Successful connection to MS SQL DB on Azure using pymssql, INSERT statement executing with no error message but 0 rows inserted

I'm using pymssql to connect to a MS SQL DB on Azure and insert records from a CSV file. I've verified that the connection is successful, and that the list used for the executemany parameter contains all of the data in the correct format and with the correct number of values. However, when I run the script 0 rows are inserted - but no error is thrown.
I looked around and it seems like most others that have experienced something similar were missing the commit(), but that isn't the issue here.
Here is the code. Any help is greatly appreciated.
with open('file.csv') as csvfile:
data = csv.reader(csvfile)
next(data)
dicts = ({'col1': line[0], 'col1': line[1], 'col3': line[2], 'col4': int(line[3]), 'col5': int(line[4]), 'col6': float(line[5])} for line in data)
to_db = ((i['col1'], i['col2'], i['col3'], i['col4'], i['col5'], i['col6']) for i in dicts)
cursor.executemany(
'INSERT INTO myTable VALUES (%s, %s, %s, %d, %d, %f)',
to_db)
print str(cursor.rowcount) + " rows inserted"
conn.commit()
Edit: If I execute the query using cursor.execute() and include the values explicitly in the query then I can successfully insert rows into the database (see below for example).
cursor.execute("INSERT INTO myTable VALUES ('4/18/2016','test','test',0,0,0.0)")
But if I user the cursor.executemany(operation,parameters) syntax and pass a list of the values as the parameter then it results in an incorrect syntax error.
cursor.executemany("INSERT INTO myTable VALUES(%s,%s,%s,%d,%d,%f)",list_of_values)
I was just reading in the module reference that only %s and %d are supported. So I'm thinking that might be the issue. But how do I pass a float?

Using the float placeholder (%f) was in fact the issue. Only %s and %d are supported, but are purely placeholders and do not have any impact on formatting the way that they typically do in python, so really only %s is needed. The working code is as follows:
cursor.executemany("INSERT INTO myTable VALUES(%s,%s,%s,%s,%s,%s)",list_of_values)

How to handle apostrophes in MySQL-Python?

A Python API is giving back u"'HOPPE'S No. 9'" as a value for a particular product attribute. I'm then looking to insert it into the DB, also using Python (python-mysqldb), with the following query:
INSERT INTO mytable (rating, Name) VALUES('5.0 (7)', 'HOPPE'S No. 9';
MySQL rejects this, and the suggested approach to handling a single quote in MySQL is to escape it first. This I need to do in Python, so I try:
In [5]: u"'HOPPE'S No. 9'".replace("'", "\'")
Out[5]: u"'HOPPE'S No. 9'"
When I incorporate this in my program, MySQL still rejects it. So I double-escape the apostrophe, and then an insert happens successfully. Thing is, it contains the escape character (so what gets written is 'HOPPE\'S No. 9').
If I need the second escape character, but when I add it gets left in, then how can I handle the escaping without having the escape character included in the string that gets inserted?
Edit: Based on theBjorn's suggestion, tried:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
#cur.execute(queryString)
cur.execute(actualSQL,
(configData["table"], sqlFieldMappingString, sqlFieldValuesString))
but it looks like I'm back to where I was when I was trying to escape using the single escape with .replace():
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''mytable' ('rating, Name, Image, mfg, price, URL') VALUES('\'5.0 (3)\', \'AR-1' at line 1

You should never construct sql that way. Use parameterized code instead:
cursor.execute(
"insert into mytable (rating, name) values (%s, %s);",
("5.0 (7)", "HOPPE'S No. 9")
)
your latest problem is due to the misconception that this is string interpolation, which it isn't (the use of %s is confusing), thus:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
will be wrong. It is possible to construct your sql string, but probably easier to do so in two steps so we don't trip over sql parameter markers looking like string interpolation markers. Assuming you have the values in a tuple named field_values:
params = ["%s"] * len(field_values) # create a list with the correct number of parameter markers
sql = "insert into %s (%s) values (%s)" % ( # here we're using string interpolation, but not with the values
configData["table"],
sqlFieldMappingString,
', '.join(params)
)
if you print sql it should look like my example above. Now you can execute it with:
cursor.execute(sql, field_values)

Right syntax to use near

I'm trying to run this script, but I have this error:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(than appear 3 users)' at
line 1
Code:
conn = mysql.connector.connect('localhost','root,'','dbtest')
cursor = conn.cursor()
add_data = ("""INSERT INTO data VALUES %s,%s,%s""" %(user, twet, time))
cursor.execute(add_data)
conn.commit()
cursor.close()
conn.close()
I belive that the error is in the %s,%s,%s, I tried a lot of diferent formats but it's always the same result.
Any ideas?
Thanks.

The correct syntax for your connection to your database should read in the lines of:
conn = mysql.connector.connect('localhost','root','','dbtest')
And maybe change your insert statement to:
add_data = ("INSERT INTO data VALUES (\"%s\",\"%s\",\"%s\")" % (user, twet, time))

You must never ever use string substitution in SQL queries. Use the db-api's parameter substitution. Apart from anything else, it will solve one of your syntax problems, which is that your strings are not quoted; it will do that automatically.
The other syntax issue you have, as Mario pointed out, is that arguments to VALUES need to be in parentheses. So, putting those together:
add_data = """INSERT INTO data VALUES (%s,%s,%s)"""
cursor.execute(add_data, (user, twet, time))

The right syntax should be with parenthesis:
INSERT INTO Customers (CustomerName, ContactName, Address, City, PostalCode, Country) VALUES ('Cardinal','Tom B. Erichsen','Skagen 21','Stavanger','4006','Norway');

python execute many with "on duplicate key update"?

I am trying to executemany in python with on duplicate key update, with the following script:
# data from a previous query (returns 4 integers in each row)
rows = first_cursor.fetchall()
query="""
INSERT INTO data (a, b, c)
VALUES (%s,%s,%s) ON DUPLICATE KEY UPDATE a=%s
"""
second_cursor.executemany(query,rows)
I'm getting this error:
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 212, in executemany
self.errorhandler(self, TypeError, msg)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
TypeError: not all arguments converted during string formatting
Is this even possible without creating my own loop?

This is a bug in MySQLdb due to the regex that MySQLdb uses to parse INSERT statements:
In /usr/lib/pymodules/python2.7/MySQLdb/cursors.py:
restr = (r"\svalues\s*"
r"(\(((?<!\\)'[^\)]*?\)[^\)]*(?<!\\)?'"
r"|[^\(\)]|"
r"(?:\([^\)]*\))"
r")+\))")
insert_values= re.compile(restr)
Although there have been numerous bug reports about this problem that have been closed as fixed, I was able to reproduce the error in MySQLdb version 1.2.3. (Note the latest version of MySQLdb at the moment is 1.2.4b4.)
Maybe this bug is fixable, I don't really know. But I think it is just the tip of the iceberg -- it points to much more trouble lurking just a little deeper. You could have for instance an INSERT ... SELECT statement with nested SELECT statements with WHERE conditions and parameters sprinkled all about... Making the regex more and more complicated to handle these cases seems to me like a losing battle.
You could use oursql; it does not use regex or string formating. It passes parametrized queries and arguments to the server separately.

When you write sql like following:
sql = insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=%s, count=count+%s'
You will get the following error: TypeError: not all arguments converted during string formatting.
So when you use "ON DUPLICATE KEY UPDATE" in python, you need to write sql like this:
sql = 'insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=values(last_date),count=count+values(count)'

found:
on duplicate key update col1=VALUES(col1), col2=VALUES(col2)
https://hardforum.com/threads/python-mysql-not-all-arguments-converted-during-string-formatting.1367039/

It is a bug of mysqldb as ubuntu said, sightly change the sql then it works:
insert into tb_name(col1, col2) select 1,2 on duplicate key update col1=1

Insert python list into Postgres database

I am having trouble in formatting the list for insertion using psycopg.
Here is a sample of code i am trying to do.
Basically I am just reading data from one table and trying to insert it into another table.
Code:
cur.execute("""select data from \"Table1\" where lat=-20.004189 and lon=-63.848004""")
rows = cur.fetchall()
print rows
cur.execute("""INSERT INTO \"%s\" (data) VALUES (ARRAY%s)""" % (args.tableName,rows)))
The result returned by first select query is like this:
[([6193, 3975, 4960, 5286, 3380, 970, 3328, 3173, 2897, 2457, 2443, 2674, 2172, 2740, 3738, 4907, 3691, 4234, 3651, 3215],)]
When I try to insert this into another table I get the following format error.
cur.execute(cur.mogrify("""INSERT INTO \"%s\" (data) VALUES (%s)""" % (args.tableName,rows)))
psycopg2.ProgrammingError: syntax error at or near "["
LINE 1: INSERT INTO "DUMMY1km" (data) VALUES ([([6193, 3975, 4960, 5...
I tried cur.mogrify, but it does not seem to help.
Please let me know if anyone has a work around for this issue.
Thanks
Adi

I don't think mogrify is needed here. Use executemany and pass rows as the second argument.
cur.executemany(
"""INSERT INTO "%s" (data) VALUES (%%s)""" % (args.tableName),rows)
Using parametrized arguments helps prevent SQL injection.
The table name can not be parametrized, so we do have to use string interpolation to place the table name in the SQL query. %%s gets escapes the percent sign and becomes %s after string interpolation.
By the way, (as a_horse_with_no_name has already pointed out) you can use the INSERT INTO ... SELECT form of INSERT to perform both SQL queries as one:
cur.execute(
"""INSERT INTO %s (data)
SELECT data FROM Table1
WHERE lat=-20.004189 AND lon=-63.848004""" % (args.tableName))
Per the question in the comments, if there are multiple fields, then the SQL becomes:
cur.executemany(
"""INSERT INTO {t} (lat,lon,data1,data2)
VALUES (%s,%s,%s,%s)""".format(t=args.tableName),rows)
(If you use the format method, then you don't have to escape all the other %ss.)