I have an extra filter in Django with a where clause, but the table name is dynamic.
filtered_queryset = queryset.extra(
where=[
f'({table_name}.modified_on, {table_name}.id) > (%s, %s)',
],
params=(after_ts, after_id),
)
How can I best avoid the f-string to make really sure it's not open to SQL injection?
I would like the SQL to have the tuple comparison, rather than multiple > and >= with AND. From previous testing, it seemed more likely to use multi-column indexes.
(This is part of some custom pagination code, where the cursor is essentially a tuple of a datetime and id)
Why don't you write it as a filter?
filtered_queryset = queryset.filter(
Q(modified_on__gt=after_ts) |
Q(Q(modified_on__gte=after_ts) & Q(id__gt=after_id))
)
PS: It's a bit unclear what your query is trying to do, I think it's this, but maybe you want to filter on something else.
First, validate your table_name variable against a list of known table names.
Also, use double-quotes as identifier delimiters, which allows table_name to be an SQL reserved keyword, or contain spaces or punctuation. These are all legal in SQL, as long as you use identifier delimiters.
f'("{table_name}".modified_on, "{table_name}".id) > (%s, %s)',
See https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS
However, if your table name contains a literal " character, this could cause trouble, because the " symbol would be interpreted as the terminating identifer delimiter. So it's up to you to filter your table_name, replacing a literal " with "".
As in this answer, extra can be avoided by using annotate, and the table name is taken from the Queryset's model internally, which avoids having to worry about escaping any identifiers:
from django.db.models import F, Func, TextField
col_a_col_b = Func(F('col_a'), F('col_b'), function='ROW', output_type=TextField())
col_a_col_b_from = Func(col_a_value, col_b_value, function='ROW')
filtered_queryset = queryset
.annotate(col_a_col_b=col_a_col_b)
.filter(col_a_col_b__gt=col_a_col_b_from)
.order_by('col_a', 'col_b')
(Apparently in Django 3.2+ alias can be used instead of extra, and the output_field hack avoided)
Related
I am trying to execute mysql query from python. I want the output
query = "UPDATE 'college_general' SET 'fees' = '180000' WHERE ('college_id' = '2')"
Below is the snippet of the code
def update(table, column, value):
return f"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
Instead Python is storing it like
query = 'UPDATE \'college_general\' SET \'fees\' = \'180000\' WHERE (\'college_id\' = \'2\')'
which is causing the script to fail. How can I achieve the desired output?
Thanks in advance!
You can replace the identifiers single quotes with backticks. For more detailed answers visit this question.
There are two types of quotes in MySQL:
' for enclosing string literals
` for enclosing identifiers such as table and column names
There are multiple issues here:
First, I suspect that the string handling bit of your program is actually working, but you are being confused by the external representation of strings. For example, if you do
x = "O'Reilly"
Python will, in some circumstances, display the string as
'O\'Reilly'
Second, I think you are using the wrong kind of quotes. Single quotes in SQL are for strings; MySQL uses backticks for names when necessary, while other SQL implementations usually use double quotes for this.
Third, AND THIS IS IMPORTANT! Do not use string manipulation for building SQL queries. The database library almost certainly has a feature for parametrized queries and you should be using that. Your query should look something like this:
query = 'UPDATE college_general SET fees = ? WHERE college_ID = ?'
cursor.execute(query, [180000, '2'])
but the details will depend on the DB library you are using. For example, some use %s instead of ?. This saves you from all kinds of headaches with quoting strings.
raw string is the simplest solution to your problem.
I believe the code below will achieve what you wanted.
def update(table, column, value):
return fr"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
I am creating a Python Flask app that interfaces with an SQL database. One of the things it does is take user input and stores it in a database. My current way of doing it looks something like this
mycursor.execute(f"SELECT * FROM privileges_groups WHERE id = {PrivID}")
This is not a good or correct way of doing this. Not only can certain characters such as ' cause errors, it also leaves me susceptible to SQL injection. Could anyone inform me of a good way of doing this?
To protect against injection attacks you should use placeholders for values.
So change
mycursor.execute(f"SELECT * FROM privileges_groups WHERE id = {PrivID}")
to
mycursor.execute("SELECT * FROM privileges_groups WHERE id = ?", (PrivID,))
Placeholders can only store a value of the given type and not an arbitrary SQL fragment. This will help to guard against strange (and probably invalid) parameter values.
However, you can't use placeholders for table names and column names.
Note: trailing comma is required for one-element tuples only but not necessary for multiple-element tuples. The comma disambiguates a tuple from an expression surrounded by parentheses.
Related: How do parameterized queries help against SQL injection?
So, if you want to avoid a sql injection...you have to have a secure query i.e. you don't want your query to doing something it shouldn't be.
queryRun = "SELECT * FROM privileges_groups WHERE id = %s" % (PrivID)
When you use "%s" this variable as a placeholder, you avoid ambiguity as to what the injection can or cannot cause to the overall system.
then..run the .execute() call:
mycursor.execute(queryRun)
Note: this also can be done in one step having all the changes within the .execute() call but you maybe better off splitting into piece-wise approach.
This isn't 100 % but should help a lot.
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.