Develop Reference

Use of '.format()' vs. '%s' in cursor.execute() for mysql JSON field, with Python mysql.connector,

My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?

First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.

In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.

Syntax error when inserting strings into MySQL using PyMySQL

I frequently use pymysql to insert data into a MySQL server.
When inserting strings, I usually (but not every time) receive: pymysql.err.ProgrammingError: (1064, ...) when I insert a string using the code (where refers to a varchar):
cursor.execute("Insert into table (column) values (%s)", (stringVar))
Typically I have to do something like:
cursor.execute("Insert into table (column) values ('"+stringVar+"')"))
However, sometimes that throws the same error and I have to do something like:
stringVar="'"+stringVar
stringVar=stringVar+"'"
cursor.execute("Insert into table (column) values ("+stringVar+")")
This just isn't a feasible way to program this operation.
I assume I am messing up something simple but I cannot figure out what this is. I use pymysql a lot and this error is really starting to wear on me. Any help would be much appreciated!

cursor.execute('INSERT INTO table (column) VALUES (?)', (stringVar,))
Whenever you're trying to directly format a string into a query like that, it's basically always a sign you're doing something wrong. Every python database interface I'm aware of has a way to pass parameters to queries like above. Note that having the stringVar contained within an iterable is required.

Inserting python variable into SQlite Database

I have seen some posts that suggesting using a ? as a place holder when inserting python variables into a SQL Query but all of these examples show the question mark at the end of the query followed by the python variable. What if you want to insert a python variable in the middle of a query and want to avoid SQL injection? I am using Python 3.6 and SQLite.
Update* - This code is working:
id='13'
text='YES'
db=sqlite3.connect('NEW_Inventory.sqlite')
cursor=db.cursor()
query=('''
INSERT
OR REPLACE
INTO
text (id, text)
VALUES
(?,
(SELECT
CASE
WHEN exists(SELECT 1 FROM text WHERE id=?)
THEN 'good'
ELSE 'Hello'
END
)
)''')
cursor.execute(query, (id, id))
db.commit()

You need to pass the parameters to execute() as a tuple. In your case you need to call it like this:
cursor.execute(query, (id, id))
where query is your parameterised SQL query string.
I assume that your code defines id somewhere, otherwise, execute() will try to use the builtin function id() to construct the query, resulting in another error.
It also worth mentioning that if you have only one parameter it must also be passed as a tuple like this (id,). Avoid the common mistake of this: (id) which is not a tuple.

Inserting DB name into Sqlite3 command

I'm using python 2.7.3 and Sqlite3 to save some data in a small db. I have the following sql command:
thedb = "allpeople"
cursor.execute("INSERT INTO %s VALUES (?,?,?,?,?,?,?,?,?,?,?,?)" % (thedb, data))
conn.commit()
But its throwing the following error:
TypeError: not all arguments converted during string formatting

You are trying to insert the table name (not the database). You appear to be mixing that up with SQL parameters; string templating and providing SQL parameters for your query are two entirely separate operations.
You'd have to use string formatting separately to build the query string:
cursor.execute("INSERT INTO %s VALUES (?,?,?,?,?,?,?,?,?,?,?,?)" % thedb, data)
or perhaps a little clearer to illustrate what is going on:
query = "INSERT INTO %s VALUES (?,?,?,?,?,?,?,?,?,?,?,?)" % thedb
cursor.execute(query, data)
Note that this opens you up to a possible SQL injection vector. Perhaps you should look into a decent SQL library instead; SQLAlchemy lets you build SQL queries from Python calls (among other tasks).

python execute many with "on duplicate key update"?

I am trying to executemany in python with on duplicate key update, with the following script:
# data from a previous query (returns 4 integers in each row)
rows = first_cursor.fetchall()
query="""
INSERT INTO data (a, b, c)
VALUES (%s,%s,%s) ON DUPLICATE KEY UPDATE a=%s
"""
second_cursor.executemany(query,rows)
I'm getting this error:
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 212, in executemany
self.errorhandler(self, TypeError, msg)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
TypeError: not all arguments converted during string formatting
Is this even possible without creating my own loop?

This is a bug in MySQLdb due to the regex that MySQLdb uses to parse INSERT statements:
In /usr/lib/pymodules/python2.7/MySQLdb/cursors.py:
restr = (r"\svalues\s*"
r"(\(((?<!\\)'[^\)]*?\)[^\)]*(?<!\\)?'"
r"|[^\(\)]|"
r"(?:\([^\)]*\))"
r")+\))")
insert_values= re.compile(restr)
Although there have been numerous bug reports about this problem that have been closed as fixed, I was able to reproduce the error in MySQLdb version 1.2.3. (Note the latest version of MySQLdb at the moment is 1.2.4b4.)
Maybe this bug is fixable, I don't really know. But I think it is just the tip of the iceberg -- it points to much more trouble lurking just a little deeper. You could have for instance an INSERT ... SELECT statement with nested SELECT statements with WHERE conditions and parameters sprinkled all about... Making the regex more and more complicated to handle these cases seems to me like a losing battle.
You could use oursql; it does not use regex or string formating. It passes parametrized queries and arguments to the server separately.

When you write sql like following:
sql = insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=%s, count=count+%s'
You will get the following error: TypeError: not all arguments converted during string formatting.
So when you use "ON DUPLICATE KEY UPDATE" in python, you need to write sql like this:
sql = 'insert into A (id, last_date, count) values(%s, %s, %s) on duplicate key update last_date=values(last_date),count=count+values(count)'

found:
on duplicate key update col1=VALUES(col1), col2=VALUES(col2)
https://hardforum.com/threads/python-mysql-not-all-arguments-converted-during-string-formatting.1367039/

It is a bug of mysqldb as ubuntu said, sightly change the sql then it works:
insert into tb_name(col1, col2) select 1,2 on duplicate key update col1=1