Develop Reference

Passing a Date variable to a sql query with python

I´m trying to write a sql query in python, where I want the user to pass a date, saved in a variable. Then I want this to be processed in the query. While it´s pretty easy to do so in R, I´m curious if there´s an easy approach in python too.
The Date should be passed where 2022-12-12 is.
date = input("this format YYYY-MM-DD")
query_new_Limits = """
SELECT #######
, a.RULE_NAME
, a.RESULT_VALUE
, a.SUMMARY_DETAIL
, a.REF_DATE
FROM ######## a ################### b
ON b.T_PORTFOLIO_ID = a.PORTFOLIO_ID
WHERE a.REF_DATE = TO_DATE('2022-12-12','YYYY-MM-DD')
AND REGEXP_LIKE(a.PORTFOLIO_CODE,'^[[:digit:]]+$')
AND NOT b.POR_INVESTMENT_TYPE IN #######
AND b.REF_TO_DATE > TO_DATE('2022-12-12','YYYY-MM-DD')
AND a.RESULT_STATE_ID > 12
"""

Just a quick advice:
avoid "date" as variable name as it's a reserved word
maybe the underlying driver is not using parameters "bind by name" - try passing an array of parameters as ( '2022-12-31', '2022-12-31') or such, just to rule out the parameter name issue

From the docs
SQL and PL/SQL statements that pass data to and from Oracle Database should use placeholders in SQL and PL/SQL statements that mark where data is supplied or returned. These placeholders are referred to as bind variables or bind parameters. A bind variable is a colon-prefixed identifier or numeral. For example, there are two bind variables (dept_id and dept_name) in this SQL statement:
sql = """insert into departments (department_id, department_name)
values (:dept_id, :dept_name)"""
cursor.execute(sql, [280, "Facility"])
[or named binding]
cursor.execute("""
insert into departments (department_id, department_name)
values (:dept_id, :dept_name)""", dept_id=280,
dept_name="Facility")
Which applied to your example gives:
date = input("this format YYYY-MM-DD")
query_new_Limits = """
SELECT #######
, a.RULE_NAME
, a.RESULT_VALUE
, a.SUMMARY_DETAIL
, a.REF_DATE
FROM ######## a ################### b
ON b.T_PORTFOLIO_ID = a.PORTFOLIO_ID
WHERE a.REF_DATE = TO_DATE(:date,'YYYY-MM-DD')
AND REGEXP_LIKE(a.PORTFOLIO_CODE,'^[[:digit:]]+$')
AND NOT b.POR_INVESTMENT_TYPE IN #######
AND b.REF_TO_DATE > TO_DATE(:date,'YYYY-MM-DD')
AND a.RESULT_STATE_ID > 12
"""
cursor.execute(query_new_Limits, date=date) # using kwarg binding

How to pass table name as variable in to PostgreSQL query in python 3 code and prevent SQL injection? [duplicate]

I have the following code, using pscyopg2:
sql = 'select %s from %s where utctime > %s and utctime < %s order by utctime asc;'
data = (dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql, data)
This outputs:
select 'waterTemp, airTemp, utctime' from 'ss2012_t02' where utctime > '2012-05-03T17:01:35+00:00'::timestamptz and utctime < '2012-05-01T17:01:35+00:00'::timestamptz order by utctime asc;
When I execute this, it falls over - this is understandable, as the quotes around the table name are illegal.
Is there a way to legally pass the table name as a parameter, or do I need to do a (explicitly warned against) string concatenation, ie:
voyage = 'ss2012_t02'
sql = 'select %s from ' + voyage + ' where utctime > %s and utctime < %s order by utctime asc;'
Cheers for any insights.

According to the official documentation:
If you need to generate dynamically an SQL query (for instance
choosing dynamically a table name) you can use the facilities
provided by the psycopg2.sql module.
The sql module is new in psycopg2 version 2.7. It has the following syntax:
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {table} values (%s, %s)")
.format(table=sql.Identifier('my_table')),
[10, 20])
More on: https://www.psycopg.org/docs/sql.html#module-usage
[Update 2017-03-24: AsIs should NOT be used to represent table or fields names, the new sql module should be used instead: https://stackoverflow.com/a/42980069/5285608 ]
Also, according to psycopg2 documentation:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.

Per this answer you can do it as so:
import psycopg2
from psycopg2.extensions import AsIs
#Create your connection and cursor...
cursor.execute("SELECT * FROM %(table)s", {"table": AsIs("my_awesome_table")})

The table name cannot be passed as a parameter, but everything else can. Thus, the table name should be hard coded in your app (Don't take inputs or use anything outside of the program as a name). The code you have should work for this.
On the slight chance that you have a legitimate reason to take an outside table name, make sure that you don't allow the user to directly input it. Perhaps an index could be passed to select a table, or the table name could be looked up in some other way. You are right to be wary of doing this, however. This works, because there are relatively few table names around. Find a way to validate the table name, and you should be fine.
It would be possible to do something like this, to see if the table name exists. This is a parameterised version. Just make sure that you do this and verify the output prior to running the SQL code. Part of the idea for this comes from this answer.
SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' and table_name=%s LIMIT 1

This is a workaround I have used in the past
query = "INSERT INTO %s (col_1, col_2) VALUES (%%s, %%s)" % table_name
cur.execute(query, (col_1_var, col_2_var))
Hope it help :)

This is a small addition to #Antoine Dusséaux's answer. If you want to pass two (unquoted) parameters in a SQL query, you can do it as follows: -
query = sql.SQL("select {field} from {table} where {pkey} = %s").format(
field=sql.Identifier('my_name'),
table=sql.Identifier('some_table'),
pkey=sql.Identifier('id'))
As per the documentation,
Usually you should express the template of your query as an SQL
instance with {}-style placeholders and use format() to merge the
variable parts into them, all of which must be Composable subclasses.
You can still have %s-style placeholders in your query and pass values
to execute(): such value placeholders will be untouched by format()
Source: https://www.psycopg.org/docs/sql.html#module-usage
Also, please keep this in mind while writing queries.

I have created a little utility for preprocessing of SQL statements with variable table (...) names:
from string import letters
NAMECHARS = frozenset(set(letters).union('.'))
def replace_names(sql, **kwargs):
"""
Preprocess an SQL statement: securely replace table ... names
before handing the result over to the database adapter,
which will take care of the values.
There will be no quoting of names, because this would make them
case sensitive; instead it is ensured that no dangerous chars
are contained.
>>> replace_names('SELECT * FROM %(table)s WHERE val=%(val)s;',
... table='fozzie')
'SELECT * FROM fozzie WHERE val=%(val)s;'
"""
for v in kwargs.values():
check_name(v)
dic = SmartDict(kwargs)
return sql % dic
def check_name(tablename):
"""
Check the given name for being syntactically valid,
and usable without quoting
"""
if not isinstance(tablename, basestring):
raise TypeError('%r is not a string' % (tablename,))
invalid = set(tablename).difference(NAMECHARS)
if invalid:
raise ValueError('Invalid chars: %s' % (tuple(invalid),))
for s in tablename.split('.'):
if not s:
raise ValueError('Empty segment in %r' % tablename)
class SmartDict(dict):
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
check_name(key)
return key.join(('%(', ')s'))
The SmartDict object returns %(key)s for every unknown key, preserving them for the value handling. The function could check for the absence of any quote characters, since all quoting now should be taken care of ...

If you want to pass the table name as a parameter, you can use this wrapper:
class Literal(str):
def __conform__(self, quote):
return self
#classmethod
def mro(cls):
return (object, )
def getquoted(self):
return str(self)
Usage: cursor.execute("CREATE TABLE %s ...", (Literal(name), ))

You can just use the module format for the table name and then use the regular paramaterization for the execute:
xlist = (column, table)
sql = 'select {0} from {1} where utctime > %s and utctime < %s order by utctime asc;'.format(xlist)
Keep in mind if this is exposed to the end user, you will not be protected from SQL injection unless you write for it.

Surprised no one has mentioned doing this:
sql = 'select {} from {} where utctime > {} and utctime < {} order by utctime asc;'.format(dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql)
format puts in the string without quotations.

Avoid python interpreting % as a placeholder in like mysql clause

I'm trying to create a DataFrame through a sql query with pandas read_sql_query method. The query has a where clause that includes a like operation but it also includes a = operation that depends on a variable. The issue is that python is interpreting the % in the like operation as a place holder, just like in the = variable operation which is something I DO want.
Here's an example of it:
sql_string = """ SELECT a,b from table WHERE a = %(variable)s
AND b like '%fixed_chars%' """
params = {'variable':'AA'}
df = pandas.read_sql_query(sql_string, params=params, con=connection)
The error that I get is TypeError: not enough arguments for format string since it interprets the % you usually use as wildcard in mysql as the place holder in python.

In this case, you'll have to use two % for those not being formatting placeholders:
sql_string = "SELECT a,b from table WHERE a = %(variable)s AND \
b like '%%fixed_chars%%'"
Hope this helps!

Searching a field in SQL to see if it contains a python variable

I am using the Canari Framework to build a series of transforms within Malteog that scan a MySQL database and return the results as entities.
I've got the entire thing working bar the LIKE operator within the SQL statement. Here's the code:
#===========================================================================
# Request and set target from graph
target = (request.entity.value)
#===========================================================================
# Extract message column and check against target entity
statement = (
"SELECT * FROM tableT.TTO WHERE Text LIKE %(tq)s"
)
cursor.execute(statement, { 'tq': target })
results = cursor.fetchall()
#===========================================================================
# Create response entities based on SQL output
for r in results:
e = getTTO(target)
e.From = r[0]
e.Text = r[1]
e.date = r[2]
e.iconurl = 'http://local.paterva.com/phrase.png'
response += e
It works, but it's only returning matches that are exact. I need it to check the TEXT column to see if the entries contain any mention of the variable "target". I spent the best part of a day getting the python variable passed in, but I can't get it to return anything other than an exact match.

The SQL LIKE operator needs special "wildcard" markers ("%" in MySQL) before and/or after the value (depending on if you want a "starts with", "endswith" or "contains" search). You have to add those wildcards around your argument, ie:
cursor.execute(statement, {'tq': "%{}%".format(target)})
Also note that cursor.fetchall() will load your whole resultset in memory, which is asking for MemoryError. cursor objects are iterables and it's safer to use them that way:
cursor.execute(statement, {'tq': "%{}%".format(target)})
for row in cursor:
do_something_with(row)

Python sqlite parameter extension problem

I have a table with three columns, cell, trx and type.
This is the query I'm trying to run:
db.execute("SELECT cell,trx FROM tchdrop").fetchall()
It gives the correct output.
However when I try a = ("cell", "trx") and then
db.execute("SELECT ?,? FROM tchdrop", t).fetchall()
the output is [(u'cell', u'trx'), (u'cell', u'trx')] (which is wrong)
I'm doing this to figure out how to extract columns dynamically which is a part of a bigger problem.

The place holder (?) of python DB-API (like sqlite3) don't support columns names to be passed, so you have to use python string formatting like this:
a = ("cell", "trx")
query = "SELECT {0},{1} FROM tchdrop".format(*a)
db.execute(query)
EDIT:
if you don't know the length of the columns that you want to pass , you can do something like this:
a = ("cell", "trx", "foo", "bar")
a = ", ".join(a)
query = "SELECT {0} FROM tchdrop".format(a)
# OUTPUT : 'SELECT cell, trx, foo, bar FROM tchdrop'
db.execute(query)

The library replaces the specified values ("cell", "trx") with their quoted SQL equivalent, so what you get is SELECT "cell", "trx" FROM tchdrop. The result is correct.
What you are trying to achieve is not possible with the ? syntax. Instead, do string replacement yourself. You can check column names with regular expressions (like ^[a-zA-Z_]$) for more security.
For example:
columns = ",".join(("cell", "trx"))
db.execute("SELECT %s FROM tchdrop" % columns).fetchall()