Develop Reference

Use of '.format()' vs. '%s' in cursor.execute() for mysql JSON field, with Python mysql.connector,

My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?

First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.

In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.

Passing string variable to MySQL, fails as tuple

Working with a newly purchased RaspberryPi and I am very new to Python/MySQL so please excuse the naive question I am posting.
I have looked at many Q&A's about this but I cannot seem to get my head around 'why' this is failing. I get error: "must be string or read-only buffer, not tuple". My variable appears as a string if I test it with TYPE so now I am lost.
import MySQLdb
import time
db = MySQLdb.connect(host="localhost", user="user",passwd="easypwd", db="imagepi")
cursor = db.cursor()
current_time = time.strftime("%H:%M:%S")
current_date = time.strftime("%Y-%m-%d")
filename = (current_time+'.jpg')
sql = ("""INSERT INTO imagelocator(batch, date, time, filename) VALUES
('1001', current_date, current_time, %s)""", filename)
cursor.execute(sql)
db.commit()
db.close()
Thanks so much for offering me a little push in the right direction.

The sql variable is a tuple. One half of it is your SQL statement, and the other half is the token value for the %s parameter in your statement. However, simply passing a tuple to an argument does not break it apart and use each element in the tuple as a separate parameter. For that, you have to use an asterisk: function_to_call(*tuple_args) ... but I think you'll have a problem with that, as well, since the database cursor expects a string for the statement argument, and a sequence for the parameters argument. The parameters argument must be a sequence (tuple, list, set, etc.) even if there is only one value.
TL;DR - You need to do something more like this:
sql = "INSERT INTO table_name (a_column, b_column) VALUES ('asdf', %s)"
args = (filename,)
cursor.execute(sql, args)
... or, if you really wanted to be tricksy and use a tuple for everything:
sql = ("INSERT INTO table_name (a_column, b_column) VALUES ('asdf', %s)", (filename,))
cursor.execute(*sql)
Edit: I guess I didn't clarify... while enclosing a string with parentheses does not create a tuple, the addition of a comma does. So, (string_var) is not a tuple, while (string_var,) is. Hopefully, that removes any confusion with how the above code operates.
Also, here's some documentation on the asterisk stuff; both the boring official docs and an easier-to-understand blog post:
Boring Python docs
Blog post

Prepared statement on MySQL not working

I am using Python 3 with mysql connector
I am trying to run a Select statement on a db, but I am having problems with a prepared statement:
This is the piece of code that does the query
cursor = cnx.cursor()
name = 'Bob'
query = ('SELECT author FROM bib WHERE author=%s')
records = cursor.execute(query, name)
I tried different syntaxes, but all with the same result. If I try to insert Bob direct on the query string it works, but with the prepared statement
You have an error in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near '%s' at line 1
Thanks

As Wrikken pointed out in a comment, the params parameter to execute have to be a tuple or a dictionary:
iterator = cursor.execute(operation, params=None, multi=True)
This method executes the given database operation (query or command). The parameters found in the tuple or dictionary params are bound to the variables in the operation. Specify variables using %s or %(name)s parameter style (that is, using format or pyformat style). execute() returns an iterator if multi is True.
In fact, this is true of any DB-API 2.0 module:
Parameters may be provided as sequence or mapping and will be bound to variables in the operation.
So:
records = cursor.execute(query, (name,))

Python+MySQLConnector: Substitution in query results in an error

I used MySQL Connector/Python API, NOT MySQLdb.
I need to dynamically insert values into a sparse table so I wrote the Python code like this:
cur.executemany("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
where
data=[('Depth', '17.5cm', Decimal('3003')), ('Input_Voltage', '110 V AC', Decimal('3004'))]
But it resulted an error:
TypeError: not enough arguments for format string
Is there any solution for this problem? Is it possible to use executemany when there is a
substitution of a field in query?
Thanks.

Let's start with the original method:
As the error message suggests you have a problem with your SQL syntax (not Python). If you insert your values you are effectively trying to execute
UPDATE myTABLE SET 'Depth'='17.5cm' WHERE id='3003'
You should notice that you are trying to assign a value to a string 'Depth', not a database field. The reason for this is that the %s substitution of the mysql module is only possible for values, not for tables/fields or other object identifiers.
In the second try you are not using the substitution anymore. Instead you use generic python string interpolation, which however looks similar. This does not work for you because you have a , and a pair of brackets too much in your code. It should read:
cur.execute("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
I also replaced executemany with execute because this method will work only for a single row. However your example only has one row, so there is no need to use executemany anyway.
The second method has some drawbacks however. The substitution is not guaranteed to be quoted or formatted in a correct manner for the SQL query, which might cause unexpected behaviour for certain inputs and may be a security concern.
I would rather ask, why it is necessary to provide the field name dynamically in the first place. This should not be necessary and might cause some trouble.

How to display an SQL string that has been constructed by MySQLDB cursor?

I would like to verify for myself how this MySQL string is translated by the cursor:
cursor.execute("SELECT * from elements where id = %s", (element_id))
Is there a way to get at the computed SQL string and print it out, preferably before execution? (the latter is not absolutely necessary - I'm just doing this because I'm learning Python and want to make sure my SQL strings are being sanitized)

Yes. As Ferdinand has pointed out, there is a MySQLdb/cursors.py containing an execute(), which in turn calls _query().
This puts the executed query into self._executed.
So you can get it from cursor._executed.

MySQL-Python does nothing special, it just encodes each argument to prevent SQL injections and uses the standard Python % operator to replace the %s placeholders with the encoded arguments.
If you really want to see the result, run the same code as cursor.execute() does:
from MySQLdb.converters import get_codec
def prepare_sql(cursor, query, args=None):
if args is not None:
query = query % tuple(( get_codec(a, cursor.encoders)(db, a) for a in args ))
return query
See the definition of execute() starting at line 168 in MySQLdb/cursors.py.