Ultimately I'm trying to use pandas read_sql which takes in a raw SQL query.
I'm trying to convert something like;
sql = str(session.query(Post).filter(Post.user_id=1))
That generates something like
select * from Post where user_id = %(user_id_1)
Is there any way to generate that query with the parameter already interpolated?
As you have found, if we str() an ORM query we get the SQL command text with parameter placeholders using the paramstyle for our dialect:
qry = session.query(Parent).filter(Parent.id == 1)
sql = str(qry)
print(sql)
"""console output:
SELECT parent.id AS parent_id, parent.lastname AS parent_lastname, parent.firstname AS parent_firstname
FROM parent
WHERE parent.id = %(id_1)s
"""
If we want to have the parameter values embedded in the SQL statement then we need to .compile() it:
sql_literal = qry.statement.compile(
compile_kwargs={"literal_binds": True},
)
print(sql_literal)
"""console output:
SELECT parent.id, parent.lastname, parent.firstname
FROM parent
WHERE parent.id = 1
"""
(Standard disclaimers regarding SQL Injection apply.)
Related
I want to convert this sql query to SQLALCHEMY:
SELECT * FROM dbcloud.client_feedback as a
join (select distinct(max(submitted_on)) sub,pb_channel_id pb, mail_thread_id mail from client_feedback group by pb_channel_id, mail_thread_id) as b
where (a.submitted_on = b.sub and a.pb_channel_id = b.pb) or ( a.submitted_on = b.sub and a.mail_thread_id = b.mail )
I can't find as keyword in SQLALCHEMY
I think that what you may be looking for is .label(name).
Assuming you have a model
class MyModel(db.Model):
id = db.Column(primary_key=True)
name = db.Column()
here is an example of how .label(name) can be used
query = db.session.query(MyModel.name.label('a'))
will produce the SQL
SELECT my_model.name as a FROM my_model
It's not straight forward to find information on this so wondering if there are some docs I can look at but basically I want to achieve passing multiple conditions to either .where() or .order_by() that is safe from SQL injection.
Here's how I am currently doing this: Two tables: Archive and Backup, and I am trying to filter by archive.city, archive.zip, and backup.serial and then I am ordering by all of those fields. The values are coming from the user via URL parameters so I need to make sure these are safe from SQL injection and sanitized.
filters = []
sorts = []
if 'city' in query:
city = query['city']
filters.append(text(f'archive.city = {city}'))
sorts.append(text(f'archive.city = {city}'))
if 'zip' in query:
zip = query['zip']
filters.append(text(f'archive.zip > {zip}'))
sorts.append(text(f'archive.zip DESC'))
if 'serial' in query:
serial = query['serial']
filters.append(text(f'backup.serial IN {serial}'))
sorts.append(text(f'backup.serial ASC'))
with Session(engine) as session:
results = session.exec(select(Archive, Backup)
.join(Backup)
.where(and_(*filters))
.order_by(*sorts).all()
as I understand the text() is not safe from sql injection, so how do I transform this so that it does what I want and is safe from sql injection?
You can invoke .where() and .order_by() on a select() multiple times and SQLAlchemy will logically "and" them for you:
qry = select(Task)
qry = qry.where(Task.description == "foo")
qry = qry.where(Task.priority < 2)
qry = qry.order_by(Task.priority)
qry = qry.order_by(Task.description)
print(qry)
"""
SELECT task.id, task.description, task.priority
FROM task
WHERE task.description = :description_1 AND task.priority < :priority_1
ORDER BY task.priority, task.description
"""
I'm aware that the best way to prevent sql injection is to write Python queries of this form (or similar):
query = 'SELECT %s %s from TABLE'
fields = ['ID', 'NAME']
cur.execute(query, fields)
The above will work for a single query, but what if we want to do a UNION of 2 SQL commands? I've set this up via sqlite3 for sake of repeatability, though technically I'm using pymysql. Looks as follows:
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT, VALUE varchar(255))'
query2 = 'CREATE TABLE DUMMy2(ID int AUTO INCREMENT, VALUE varchar(255)'
try:
cur.execute(query)
cur.execute(query2)
except:
print('Already made table!')
tnames = ['DUMMY1', 'DUMMY2']
sqlcmds = []
for i in range(0,2):
query = 'SELECT %s FROM {}'.format(tnames[i])
sqlcmds.append(query)
fields = ['VALUE', 'VALUE']
sqlcmd = ' UNION '.join(sqlcmds)
cur.execute(sqlcmd, valid_fields)
When I run this, I get a sqlite Operational Error:
sqlite3.OperationalError: near "%": syntax error
I've validated the query prints as expected with this output:
INSERT INTO DUMMY VALUES(%s) UNION INSERT INTO DUMMY VALUES(%s)
All looks good there. What is the issue with the string substitutions here? I can confirm that running a query with direct string substitution works fine. I've tried it with both selects and inserts.
EDIT: I'm aware there are multiple ways to do this with executemany and a few other. I need to do this with UNION for the purposes I'm using this for because this is a very, very simplified example fo the operational code I'm using
The code below executes few INSERTS at once
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT NOT NULL, VALUE varchar(255))'
try:
cur.execute(query)
except:
print('Already made table!')
valid_fields = [('ya dummy',), ('stupid test example',)]
cur.executemany('INSERT INTO DUMMY (VALUE) VALUES (?)',valid_fields)
How to pass variable parameters dynamically
order = 10100
status = 'Shipped'
df1 = pd.read_sql_query("SELECT * from orders where orderNumber =""" +
str(10100) + """ and status = """ + 'status' +""" order by orderNumber """,cnx)
TypeError: must be str, not int
getting above error although i converted to strings any idea?
is there any alternative wy to pass the parameters?
Use parametrized sql by supplying the arguments via the params keyword argument. The proper quotation of arguments will be done for you by the database adapter and the code will be less vulnerable to SQL injection attacks. (See Little Bobby Tables for an example of the kind of trouble improperly quoted, non-parametrized sql can get you into.)
order = 10100
status = 'Shipped'
sql = """SELECT * from orders where orderNumber = ?
and status = ? order by orderNumber"""
df1 = pd.read_sql_query(sql, cnx, params=[order, status])
The ?s in sql are parameter markers. They get replaced with properly quoted values from params. Note that the proper parameter marker depends on the database adapter you are using. For example, MySQLdb and psycopg2 uses %s, while sqlite3, and oursql uses ?.
IN MYSQL ::
order = 10100
status = 'Shipped'
sql = """SELECT * from orders where orderNumber = %s
and status = %s order by orderNumber"""
df1 = pd.read_sql_query(sql, cnx, params=[order, status])
I am trying to perform a SELECT query with an IN() clause, and have sqlalchemy perform the
parameter escaping for me. I am using pyodbc as my database connector.
This is the code I have written so far:
tables = ['table1', 'table2', ... ]
sql = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME IN(:tables)"
result = session.execute(sql, {"tables": tables})
Unfortunatenly this fails with an error:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('Invalid parameter type. param-index=0 param-type=list', 'HY105')
Is there any way I can have sqlalchemy escape the whole list of parameters and join them with ,
without manually adding a :tableX placeholder for each item of the list?
Try something like this....
DECLARE #string Varchar(100) = 'Table1,table2,table3'
declare #xml xml
set #xml = N'<root><r>' + replace(#string,',','</r><r>') + '</r></root>'
SELECT * FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME IN( select
r.value('.','varchar(max)') as item
from #xml.nodes('//root/r') as records(r)
)
For good reasons it is not possible to expand a list of arguments as you wish.
If you really would like to create a raw SQL query, then you can just enumerate over your list and dynamically create the query:
vals = {"param{}".format(i): table for i, table in enumerate(tables)}
keys = ", ".join([":{}".format(k) for k in vals])
sql = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME IN ({keys})".format(keys=keys)
result = session.execute(sql, vals)
for tbl in result:
print(tbl)
But you could ask sqlalchemy to do this for you. Here we make a fake mapping of the INFORMATION_SCHEMA.tables view, and query it using sqlalchemy toolset:
# definition (done just once)
class ISTable(Base):
__tablename__ = 'tables'
__table_args__ = {'schema': 'INFORMATION_SCHEMA'}
_fake_id = Column(Integer, primary_key=True)
table_catalog = Column(String)
table_schema = Column(String)
table_name = Column(String)
table_type = Column(String)
# actual usage
result = session.query(
ISTable.table_catalog, ISTable.table_schema,
ISTable.table_name, ISTable.table_type,
).filter(
ISTable.table_name.in_(tables))
for tbl in result:
print(tbl)
One gotcha: you cannot query for the whole mapped class (like this query(ISTable)) because the primary_key does not exist and an exception will be raised. But querying only columns we can about (as shown above) is good enough for the purpose.