Develop Reference

%s variable in Query Execution Python 3.8 (pymssql)

I have a python script with a basic GUI that logs into a DB and executes a query.
The Python script also asks for 1 parameter called "collection Name" which is taken from the tkinter .get function and is added as a %s inside the Query text. The result is that each time I can execute a query with a different "Collection name". This works and it is fine
Now, I want to add a larger string of Collection Names into my .get function so I can do cursor.execute a query with multiple collection names to get more complex data. But I am having issues with inputing multiple "collection names" into my app.
Below is a piece of my Query1, which has the %s variable that it then gets from the input to tkinter.
From #Session1
Join vGSMRxLevRxQual On(#Session1.SessionId = vGSMRxLevRxQual.SessionId)
Where vGSMRxLevRxQual.RxLevSub<0 and vGSMRxLevRxQual.RxLevSub>-190
and #Session1.CollectionName in (%s)
Group by
#Session1.Operator
Order by #Session1.Operator ASC
IF OBJECT_ID('tempdb..#SelectedSession1') IS NOT NULL DROP TABLE #SelectedSession1
IF OBJECT_ID('tempdb..#Session1') IS NOT NULL DROP TABLE #Session1
Here, is where I try to execute the query
if Query == "GSMUERxLevelSub" :
result = cursor.execute(GSMUERxLevelSub, (CollectionName,))
output = cursor.fetchmany
df = DataFrame(cursor.fetchall())
filename = "2021_H1 WEEK CDF GRAPHS().xlsx"
df1 = DataFrame.transpose(df, copy=False)
Lastly, here is where I get the value for the Collection name:
CollectionName = f_CollectionName.get()
enter image description here
enter code here

Your issues are due to a list/collection being a invalid parameter.
You'll need to transform collectionName
collection_name: list[str] = ['collection1', 'collection2']
new_collection_name = ','.join(f'"{c}"' for c in collection_name)
cursor.execute(sql, (new_collection_name,))
Not sure if this approach will be susceptible to SQL injection if that's a concern.
Edit:
Forgot the DBAPI would put another set of quotes around the parameters. If you can do something like:
CollectionName = ["foo", "bar"]
sql = f"""
From #Session1
Join vGSMRxLevRxQual On(#Session1.SessionId = vGSMRxLevRxQual.SessionId)
Where vGSMRxLevRxQual.RxLevSub<0 and vGSMRxLevRxQual.RxLevSub>-190
and #Session1.CollectionName in ({",".join(["%s"] * len(CollectionName))})
"""
sql += """
Group by
#Session1.Operator
Order by #Session1.Operator ASC
"""
cursor.execute(sql, (CollectionName,))
EDIT: Update to F-string

How to pass table name as variable in to PostgreSQL query in python 3 code and prevent SQL injection? [duplicate]

I have the following code, using pscyopg2:
sql = 'select %s from %s where utctime > %s and utctime < %s order by utctime asc;'
data = (dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql, data)
This outputs:
select 'waterTemp, airTemp, utctime' from 'ss2012_t02' where utctime > '2012-05-03T17:01:35+00:00'::timestamptz and utctime < '2012-05-01T17:01:35+00:00'::timestamptz order by utctime asc;
When I execute this, it falls over - this is understandable, as the quotes around the table name are illegal.
Is there a way to legally pass the table name as a parameter, or do I need to do a (explicitly warned against) string concatenation, ie:
voyage = 'ss2012_t02'
sql = 'select %s from ' + voyage + ' where utctime > %s and utctime < %s order by utctime asc;'
Cheers for any insights.

According to the official documentation:
If you need to generate dynamically an SQL query (for instance
choosing dynamically a table name) you can use the facilities
provided by the psycopg2.sql module.
The sql module is new in psycopg2 version 2.7. It has the following syntax:
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {table} values (%s, %s)")
.format(table=sql.Identifier('my_table')),
[10, 20])
More on: https://www.psycopg.org/docs/sql.html#module-usage
[Update 2017-03-24: AsIs should NOT be used to represent table or fields names, the new sql module should be used instead: https://stackoverflow.com/a/42980069/5285608 ]
Also, according to psycopg2 documentation:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.

Per this answer you can do it as so:
import psycopg2
from psycopg2.extensions import AsIs
#Create your connection and cursor...
cursor.execute("SELECT * FROM %(table)s", {"table": AsIs("my_awesome_table")})

The table name cannot be passed as a parameter, but everything else can. Thus, the table name should be hard coded in your app (Don't take inputs or use anything outside of the program as a name). The code you have should work for this.
On the slight chance that you have a legitimate reason to take an outside table name, make sure that you don't allow the user to directly input it. Perhaps an index could be passed to select a table, or the table name could be looked up in some other way. You are right to be wary of doing this, however. This works, because there are relatively few table names around. Find a way to validate the table name, and you should be fine.
It would be possible to do something like this, to see if the table name exists. This is a parameterised version. Just make sure that you do this and verify the output prior to running the SQL code. Part of the idea for this comes from this answer.
SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' and table_name=%s LIMIT 1

This is a workaround I have used in the past
query = "INSERT INTO %s (col_1, col_2) VALUES (%%s, %%s)" % table_name
cur.execute(query, (col_1_var, col_2_var))
Hope it help :)

This is a small addition to #Antoine Dusséaux's answer. If you want to pass two (unquoted) parameters in a SQL query, you can do it as follows: -
query = sql.SQL("select {field} from {table} where {pkey} = %s").format(
field=sql.Identifier('my_name'),
table=sql.Identifier('some_table'),
pkey=sql.Identifier('id'))
As per the documentation,
Usually you should express the template of your query as an SQL
instance with {}-style placeholders and use format() to merge the
variable parts into them, all of which must be Composable subclasses.
You can still have %s-style placeholders in your query and pass values
to execute(): such value placeholders will be untouched by format()
Source: https://www.psycopg.org/docs/sql.html#module-usage
Also, please keep this in mind while writing queries.

I have created a little utility for preprocessing of SQL statements with variable table (...) names:
from string import letters
NAMECHARS = frozenset(set(letters).union('.'))
def replace_names(sql, **kwargs):
"""
Preprocess an SQL statement: securely replace table ... names
before handing the result over to the database adapter,
which will take care of the values.
There will be no quoting of names, because this would make them
case sensitive; instead it is ensured that no dangerous chars
are contained.
>>> replace_names('SELECT * FROM %(table)s WHERE val=%(val)s;',
... table='fozzie')
'SELECT * FROM fozzie WHERE val=%(val)s;'
"""
for v in kwargs.values():
check_name(v)
dic = SmartDict(kwargs)
return sql % dic
def check_name(tablename):
"""
Check the given name for being syntactically valid,
and usable without quoting
"""
if not isinstance(tablename, basestring):
raise TypeError('%r is not a string' % (tablename,))
invalid = set(tablename).difference(NAMECHARS)
if invalid:
raise ValueError('Invalid chars: %s' % (tuple(invalid),))
for s in tablename.split('.'):
if not s:
raise ValueError('Empty segment in %r' % tablename)
class SmartDict(dict):
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
check_name(key)
return key.join(('%(', ')s'))
The SmartDict object returns %(key)s for every unknown key, preserving them for the value handling. The function could check for the absence of any quote characters, since all quoting now should be taken care of ...

If you want to pass the table name as a parameter, you can use this wrapper:
class Literal(str):
def __conform__(self, quote):
return self
#classmethod
def mro(cls):
return (object, )
def getquoted(self):
return str(self)
Usage: cursor.execute("CREATE TABLE %s ...", (Literal(name), ))

You can just use the module format for the table name and then use the regular paramaterization for the execute:
xlist = (column, table)
sql = 'select {0} from {1} where utctime > %s and utctime < %s order by utctime asc;'.format(xlist)
Keep in mind if this is exposed to the end user, you will not be protected from SQL injection unless you write for it.

Surprised no one has mentioned doing this:
sql = 'select {} from {} where utctime > {} and utctime < {} order by utctime asc;'.format(dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql)
format puts in the string without quotations.

Using WHERE in query PyMySQL

I am trying to create a program where a user can enter an operator i.e. <> or = and then a number for a database in pymysql. I have tried a number of different ways of doing this but unfortunately unsuccessful. I have two documents with display being one and importing display into the other document.
Docuemnt 1
def get_pop(op, pop):
if (not conn):
connect();
query = "SELECT * FROM city WHERE Population %s %s"
with conn:
cursor = conn.cursor()
cursor.execute(query, (op, pop))
x = cursor.fetchall()
return x
Document two
def city():
op = input("Enter < > or =: ")
population = input("Enter population: ")
pop = display.get_pop(op, population)
for p in pop:
print(pop)
I am getting the following error.
pymysql.err.ProgrammingError: (1064,......
Please help thanks

You can't do this. Parameterization works for values only, not operators or table names, or column names. You'll need to format the operator into the string. Do not confuse the %s placeholder here with Python string formatting; MySQL is awkward in that it uses %s for binding parameters, which clashes with regular Python string formatting.
The MySQL %s in a query string escapes the user input to protect against SQL Injection. In this case, I set up a basic test to see if the operation part submitted by the user was in a list of accepted operations.
def get_pop(op, pop):
query = "SELECT * FROM city WHERE Population {} %s" # Add a placeholder for format
with conn: # Where does this come from?
cursor = conn.cursor()
if op in ['=', '!=']:
cursor.execute(query.format(op), (pop,))
x = cursor.fetchall()
return x
You'll want to come up with some reasonable return value in the case that if op in ['=', '!='] is not True but that depends entirely on how you want this to behave.

After checking that op indeed contains either "<>" or "=" and that pop indeed contains a number you could try:
query = "SELECT * FROM city WHERE Population " + op + " %s";
Beware of SQL injection.
Then
cursor.execute(query, (pop))

In python, How to do the postgresql 'LIKE %name%' query which mitigate SQL injection?

I want to execute below query,
select * from table where name LIKE %sachin%;
I created sql query in this way,
sql = "select * from table where %s like '\%%s\%'"
It gives me following error,
ValueError: unsupported format character ''' (0x27) at index 42
I want '%' symbol before and after the string.
How can I achive this? It should also mitigate SQL injection.

Your best option is to use placeholders and generate the right-hand value of the LIKE in SQL like as follows. The big difficulty is you are also expecting to pass in the identifier which means you will probably have to do something a little different:
sqltemplate = "select * from table where {} like '%%' || %s || '%%'"
Into this we fill in our identifier. Note it is important to whitelist the value.
allowed_columns = ['foo', 'bar', 'bar']
if colname in allowed_columns:
sql = sqltemplate.format(colname);
else:
raise ValueError('Bad column name!')
Then you can use a placeholder for %s and it will just work.
conn.execute(sql, (searchval,));
Note: In psycopg2, you use %s and %d for placeholders, and %% to represent a literal percent.

You can use % as the escape character.
sql = "select * from table where name like '%%sachin%%'"
For your case, The above query should work.

python avoiding %s in sql query

I am trying to query a mysql db from python but having troubles generating the query ebcasue of the wildcard % and python's %s. As a solution I find using ?, but when I run the following,
query = '''select * from db where name like'Al%' and date = '%s' ''', myDateString
I get an error
cursor.execute(s %'2015_05_21')
ValueError: unsupported format character ''' (0x27) at index 36 (the position of %)
How can i combine python 2.7 string bulding and sql wildcards? (The actual query is a lot longer and involves more variables)

First of all, you need to escape the percent sign near the Al:
'''select * from db where name like 'Al%%' and date = '%s''''
Also, follow the best practices and pass the query parameters in the second argument to execute(). This way your query parameters would be escaped and you would avoid sql injections:
query = """select * from db where name like 'Al%%' and date = %s"""
cursor.execute(query, ('2015_05_21', ))

Two things:
Don't use string formatting ('%s' % some_var) in SQL queries. Instead, pass the string as a sequence (like a list or a tuple) to the execute method.
You can escape your % so Python will not expect a format specifier:
q = 'SELECT foo FROM bar WHERE zoo LIKE 'abc%%' and id = %s'
cursor.execute(q, (some_var,))

Use the format syntax for Python string building, and %s for SQL interpolation. That way they don't conflict with each other.

You are not using the ? correctly.
Here's an example:
command = '''SELECT M.name, M.year
FROM Movie M, Person P, Director D
WHERE M.id = D.movie_id
AND P.id = D.director_id
AND P.name = ?
AND M.year BETWEEN ? AND ?;'''
*Execute the command, replacing the placeholders with the values of
the variables in the list [dirName, start, end]. *
cursor.execute(command, [dirName, start, end])
So, you want to try:
cursor.execute(query,'2015_05_21')