Develop Reference

sql data formatting and sql injections

I have a database with 2 tables: students, employees and I want to update one of those tables:
import sqlite3
db_file = "school.db"
def update_address(identifier, user_address, user_id):
with sqlite3.connect(db_file) as conn:
c = conn.cursor()
c.execute(f"""
UPDATE {identifier}
SET address = ?
WHERE id = ?;
""",
(user_address, user_id))
update_address("students", "204 Sycamore Street", 2)
The above code works, the problem is I know that using python string formatting in an sql operation can lead to vulnerabilities per sqlite3 docs:
Usually your SQL operations will need to use values from Python variables. You shouldn’t assemble your query using Python’s string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack (see https://xkcd.com/327/ for humorous example of what can go wrong).
Instead, use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.
The placeholder '?' works when it comes to inserting values but not for sql identifiers. Output:
sqlite3.OperationalError: near "?": syntax error
So the question here is: can an sql injection occur if I use python string formatting on an sql identifier or does it only occur on values ?
If it also occurs on identifiers is there a way to format the string in a safe manner?

Yes, if you interpolate any content into an SQL query unsafely, it is an SQL injection vulnerability. It doesn't matter if the content is supposed to be used as a value in the SQL expression, or an identifier, SQL keyword, or anything else.
It's pretty common to format queries from fragments of SQL expressions, if you want to write a query with a variable set of conditions. These are also possible SQL injection risks.
The way to mitigate the SQL injection risk is: don't interpolate untrusted input into your SQL query.
For identifiers, you should make sure the content matches a legitimate name of a table (or column, or other element, if that's what you're trying to make dynamic). I.e. create an "allowlist" of tables known to exist in your database that are permitted to update using your function. If the input doesn't match one of these, then don't run the query.
It's also a good idea to use back-ticks to delimit identifiers, because if one of the table names happens to be a reserved keyword in SQLite, that will allow the table to be used in the SQL query.
if identifier not in ["table1", "table2", "table3"]:
raise Exception("Unknown table name: '{identifier}'")
c.execute(f"""
UPDATE `{identifier}`
SET address = ?
WHERE id = ?;
""",
(user_address, user_id))

SQL sanitize python [duplicate]

This question already has answers here:
How can prepared statements protect from SQL injection attacks?
(10 answers)
Closed 11 months ago.
What is the best way to sanitize a SQL to prevent injection when using python? I'm using mysql-connector. I have read that I should use a structure similar to:
import mysql.connector
connection = mysql.connector.connect(host="", port="", user="", password="", database="")
cursor = connection.cursor( buffered = True )
sql = "INSERT INTO mytable (column1, column2) VALUES (%s, %s)"
val = (myvalue1, myvalue2)
cursor.execute(sql, val)
connection.commit()
However, I don't understand why this can prevent an injection. Is this sufficient? A user could introduce me anything on myvalue1 or myvalue2, even if it is not suposed to. Is there any useful library?

SQL injection works when untrusted input is interpolated into an SQL query and the input contains characters that change the syntax of the query.
Query parameters are kept separate from the SQL query, never interpolated into it. The values of the parameters are combined with the SQL query after it is parsed, so there is no longer any opportunity to change the syntax. The parameter is guaranteed to be treated as a single scalar value (i.e. as if it's just a string literal in an SQL expression).
This is the way the Python connector works if you use the MySQLCursorPrepared cursor subclass. See https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html
Otherwise, the Python connector "simulates" prepared queries. It actually does interpolate parameters into the SQL query before it is parsed, but it does so safely, by escaping special characters that would cause SQL injection. It is well-tested so it's reliable.
Both cursor types are used the same way, passing an SQL query string with %s placeholders, and another argument with a tuple of parameter values. You are using it correctly.
Re comment from #Learningfrommasters:
Yes, a string stored in your database can be used unsafely in another SQL query, and cause SQL injection. Some people think that only user input must be treated safely, but this is not true. Any variable should be treated as a query parameter, whether the value for that variable comes from user input, or read from a file, or even pulled out of your own database.
Example: Suppose my name is Bill O'Karwin. It has an apostrophe in it, which you know is a special character to SQL because it terminates a string literal.
If my name were stored in the database and then fetched into an application into a variable userlastname, then I could search for other people with the same last name:
sql = f"SELECT * FROM Users WHERE lastname = '{userlastname}'"
That is unsafe because the apostrophe would cause SQL injection. Even though the value didn't come directly from user input, it came from my own database.
So use parameters for all variables. Then you don't have to think about whether the source is safe or not.
sql = "SELECT * FROM Users WHERE lastname = %s"
cur.execute(sql, (userlastname,))

Passing strings to pymysql cursor using Python function

I'm working on a script that will pull data from a database using pymysql and a simple SELECT statement. I'm going to run this statement many times, so I was hoping to simplify things by putting it in a function and passing the column name, table name, where clause and value as arguments to the function.
def retrieve_value_from_db(connection,column_name,table_name,where_clause,where_value):
with connection:
with connection.cursor() as cursor:
sql = "SELECT %s FROM %s WHERE %s=%s"
logging.debug(cursor.mogrify(sql,(column_name,table_name,where_clause,where_value)))
cursor.execute(sql,(column_name,table_name,where_clause,where_value))
result = cursor.fetchone()
connection.commit()
return result
However calling the function below returns the following error
retrieve_value_from_db(connection,"last_name","mother_table","id","S50000")
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''mother_table' WHERE 'id'='S50000'' at line 1")
Cursor.execute seems to be reading the quotation mark portion of the string, which is causing the programming error. So how do I pass a string argument to the function that cursor.execute can read? Is what I want to do even possible? Thanks in advance.

Perhaps surprisingly, you should not let the database substitution handle table names and column names. It tries to quote them as if they were fields, which is wrong.
sql = "SELECT %s FROM %s WHERE %s=%%s" % (column_name,table_name,where_clause)
...
cursor.execute(sql, (where_value,))

Use of '.format()' vs. '%s' in cursor.execute() for mysql JSON field, with Python mysql.connector,

My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?

First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.

In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.

psycopg2 TypeError: not all arguments converted during string formatting

I'm trying execute a simple query, but getting this error no matter how I pass the parameters.
Here is the query (I'm using Trac db object to connect to a DB):
cursor.execute("""SELECT name FROM "%s".customer WHERE firm_id='%s'""" % (schema, each['id']))
schema and each['id'] both are simple strings
print("""SELECT name FROM "%s".customer WHERE firm_id='%s'""" % (schema, each['id']))
Result:
SELECT name FROM "Planing".customer WHERE firm_id='135'
There is on error is a remove quote after firm_id=, but that way parameter is treated a an integer and ::text leads to the very same error.

In my case I didn't realize that you had to pass a tuple to cursor.execute. I had this:
cursor.execute(query, (id))
But I needed to pass a tuple instead
cursor.execute(query, (id,))

I got this same error and couldn't for the life of me work out how to fix, in the end it was my mistake because I didn't have enough parameters matching the number of elements in the tuple:
con.execute("INSERT INTO table VALUES (%s,%s,%s,%s,%s)",(1,2,3,4,5,6))
Note that I have 5 elements in the values to be inserted into the table, but 6 in the tuple.

It is recommended to not use string interpolation for passing variables in database queries, but using string interpolation to set the table name is fine as long as it's not an external input or you restrict the allowed value. Try:
cursor.execute("""
SELECT name FROM %s.customer WHERE firm_id=%%s
""" % schema, (each['id'],))
Rules for DB API usage provides guidance for programming against the database.

Use AsIs
from psycopg2.extensions import AsIs
cursor.execute("""
select name
from %s.customer
where firm_id = %s
""",
(AsIs(schema), each['id'])
)

You could try this:
cursor.execute("INSERT INTO table_name (key) VALUES(%s)",(value1,))
You will get an error if you are missing a (,) after value1.

The correct way to pass variables in a SQL command is using the second argument of the execute() method. And i think you should remove single quotes from second parameter, read about it here - http://initd.org/psycopg/docs/usage.html#the-problem-with-the-query-parameters.
Note that you cant pass table name as parameter to execute and it considered as bad practice but there is some workarounds:
Passing table name as a parameter in psycopg2
psycopg2 cursor.execute() with SQL query parameter causes syntax error
To pass table name try this:
cursor.execute("""SELECT name FROM "%s".customer WHERE firm_id=%s""" % (schema, '%s'), (each['id'],))

Every time I have this kind of error, I am passing the wrong amount of values. Try check it