Avoid python interpreting % as a placeholder in like mysql clause - python

I'm trying to create a DataFrame through a sql query with pandas read_sql_query method. The query has a where clause that includes a like operation but it also includes a = operation that depends on a variable. The issue is that python is interpreting the % in the like operation as a place holder, just like in the = variable operation which is something I DO want.
Here's an example of it:
sql_string = """ SELECT a,b from table WHERE a = %(variable)s
AND b like '%fixed_chars%' """
params = {'variable':'AA'}
df = pandas.read_sql_query(sql_string, params=params, con=connection)
The error that I get is TypeError: not enough arguments for format string since it interprets the % you usually use as wildcard in mysql as the place holder in python.

In this case, you'll have to use two % for those not being formatting placeholders:
sql_string = "SELECT a,b from table WHERE a = %(variable)s AND \
b like '%%fixed_chars%%'"
Hope this helps!

Related

%s variable in Query Execution Python 3.8 (pymssql)

I have a python script with a basic GUI that logs into a DB and executes a query.
The Python script also asks for 1 parameter called "collection Name" which is taken from the tkinter .get function and is added as a %s inside the Query text. The result is that each time I can execute a query with a different "Collection name". This works and it is fine
Now, I want to add a larger string of Collection Names into my .get function so I can do cursor.execute a query with multiple collection names to get more complex data. But I am having issues with inputing multiple "collection names" into my app.
Below is a piece of my Query1, which has the %s variable that it then gets from the input to tkinter.
From #Session1
Join vGSMRxLevRxQual On(#Session1.SessionId = vGSMRxLevRxQual.SessionId)
Where vGSMRxLevRxQual.RxLevSub<0 and vGSMRxLevRxQual.RxLevSub>-190
and #Session1.CollectionName in (%s)
Group by
#Session1.Operator
Order by #Session1.Operator ASC
IF OBJECT_ID('tempdb..#SelectedSession1') IS NOT NULL DROP TABLE #SelectedSession1
IF OBJECT_ID('tempdb..#Session1') IS NOT NULL DROP TABLE #Session1
Here, is where I try to execute the query
if Query == "GSMUERxLevelSub" :
result = cursor.execute(GSMUERxLevelSub, (CollectionName,))
output = cursor.fetchmany
df = DataFrame(cursor.fetchall())
filename = "2021_H1 WEEK CDF GRAPHS().xlsx"
df1 = DataFrame.transpose(df, copy=False)
Lastly, here is where I get the value for the Collection name:
CollectionName = f_CollectionName.get()
enter image description here
enter code here
Your issues are due to a list/collection being a invalid parameter.
You'll need to transform collectionName
collection_name: list[str] = ['collection1', 'collection2']
new_collection_name = ','.join(f'"{c}"' for c in collection_name)
cursor.execute(sql, (new_collection_name,))
Not sure if this approach will be susceptible to SQL injection if that's a concern.
Edit:
Forgot the DBAPI would put another set of quotes around the parameters. If you can do something like:
CollectionName = ["foo", "bar"]
sql = f"""
From #Session1
Join vGSMRxLevRxQual On(#Session1.SessionId = vGSMRxLevRxQual.SessionId)
Where vGSMRxLevRxQual.RxLevSub<0 and vGSMRxLevRxQual.RxLevSub>-190
and #Session1.CollectionName in ({",".join(["%s"] * len(CollectionName))})
"""
sql += """
Group by
#Session1.Operator
Order by #Session1.Operator ASC
"""
cursor.execute(sql, (CollectionName,))
EDIT: Update to F-string

In python, How to do the postgresql 'LIKE %name%' query which mitigate SQL injection?

I want to execute below query,
select * from table where name LIKE %sachin%;
I created sql query in this way,
sql = "select * from table where %s like '\%%s\%'"
It gives me following error,
ValueError: unsupported format character ''' (0x27) at index 42
I want '%' symbol before and after the string.
How can I achive this? It should also mitigate SQL injection.
Your best option is to use placeholders and generate the right-hand value of the LIKE in SQL like as follows. The big difficulty is you are also expecting to pass in the identifier which means you will probably have to do something a little different:
sqltemplate = "select * from table where {} like '%%' || %s || '%%'"
Into this we fill in our identifier. Note it is important to whitelist the value.
allowed_columns = ['foo', 'bar', 'bar']
if colname in allowed_columns:
sql = sqltemplate.format(colname);
else:
raise ValueError('Bad column name!')
Then you can use a placeholder for %s and it will just work.
conn.execute(sql, (searchval,));
Note: In psycopg2, you use %s and %d for placeholders, and %% to represent a literal percent.
You can use % as the escape character.
sql = "select * from table where name like '%%sachin%%'"
For your case, The above query should work.

Using cursor.execute arguments in pymssql with IN sql statement

I have troubles using a simple sql statement with the operator IN through pymssql.
Here is a sample :
import pymssql
conn = pymssql.connect(server='myserver', database='mydb')
cursor = conn.cursor()
req = "SELECT * FROM t1 where id in (%s)"
cursor.execute(req, tuple(range(1,10)))
res = cursor.fetchall()
Surprisingly only the first id is returned and I can't figure out why.
Does anyone encounter the same behavior ?
You're trying to pass nine ID values to the query and you only have one placeholder. You can get nine placeholders by doing this:
ids = range(1,10)
placeholders = ','.join('%s' for i in ids)
req = "SELECT * FROM t1 where id in ({})".format(placeholders)
cursor.execute(req, ids)
res = cursor.fetchall()
As an aside, you don't necessarily need a tuple here. A list will work fine.
It looks like you are only passing SELECT * FROM t1 where id in (1). You call execute with the tuple but the string only has one formatter. To pass all values, call execute like this:
cursor.execute(req, (tuple(range(1,10)),))
This will pass the tuple as first argument to the string to format.
EDIT: Regarding the executeone/many() thing, if you call executemany and it returns the last instead of the first id, it seems that execute will run the query 10 times as it can format the string with 10 values. The last run will then return the last id.

python avoiding %s in sql query

I am trying to query a mysql db from python but having troubles generating the query ebcasue of the wildcard % and python's %s. As a solution I find using ?, but when I run the following,
query = '''select * from db where name like'Al%' and date = '%s' ''', myDateString
I get an error
cursor.execute(s %'2015_05_21')
ValueError: unsupported format character ''' (0x27) at index 36 (the position of %)
How can i combine python 2.7 string bulding and sql wildcards? (The actual query is a lot longer and involves more variables)
First of all, you need to escape the percent sign near the Al:
'''select * from db where name like 'Al%%' and date = '%s''''
Also, follow the best practices and pass the query parameters in the second argument to execute(). This way your query parameters would be escaped and you would avoid sql injections:
query = """select * from db where name like 'Al%%' and date = %s"""
cursor.execute(query, ('2015_05_21', ))
Two things:
Don't use string formatting ('%s' % some_var) in SQL queries. Instead, pass the string as a sequence (like a list or a tuple) to the execute method.
You can escape your % so Python will not expect a format specifier:
q = 'SELECT foo FROM bar WHERE zoo LIKE 'abc%%' and id = %s'
cursor.execute(q, (some_var,))
Use the format syntax for Python string building, and %s for SQL interpolation. That way they don't conflict with each other.
You are not using the ? correctly.
Here's an example:
command = '''SELECT M.name, M.year
FROM Movie M, Person P, Director D
WHERE M.id = D.movie_id
AND P.id = D.director_id
AND P.name = ?
AND M.year BETWEEN ? AND ?;'''
*Execute the command, replacing the placeholders with the values of
the variables in the list [dirName, start, end]. *
cursor.execute(command, [dirName, start, end])
So, you want to try:
cursor.execute(query,'2015_05_21')

Python sqlite parameter extension problem

I have a table with three columns, cell, trx and type.
This is the query I'm trying to run:
db.execute("SELECT cell,trx FROM tchdrop").fetchall()
It gives the correct output.
However when I try a = ("cell", "trx") and then
db.execute("SELECT ?,? FROM tchdrop", t).fetchall()
the output is [(u'cell', u'trx'), (u'cell', u'trx')] (which is wrong)
I'm doing this to figure out how to extract columns dynamically which is a part of a bigger problem.
The place holder (?) of python DB-API (like sqlite3) don't support columns names to be passed, so you have to use python string formatting like this:
a = ("cell", "trx")
query = "SELECT {0},{1} FROM tchdrop".format(*a)
db.execute(query)
EDIT:
if you don't know the length of the columns that you want to pass , you can do something like this:
a = ("cell", "trx", "foo", "bar")
a = ", ".join(a)
query = "SELECT {0} FROM tchdrop".format(a)
# OUTPUT : 'SELECT cell, trx, foo, bar FROM tchdrop'
db.execute(query)
The library replaces the specified values ("cell", "trx") with their quoted SQL equivalent, so what you get is SELECT "cell", "trx" FROM tchdrop. The result is correct.
What you are trying to achieve is not possible with the ? syntax. Instead, do string replacement yourself. You can check column names with regular expressions (like ^[a-zA-Z_]$) for more security.
For example:
columns = ",".join(("cell", "trx"))
db.execute("SELECT %s FROM tchdrop" % columns).fetchall()

Categories

Resources