I want to pass in parameters to a sql query when using GeoPandas from_postgis functionality with SQLAlchemy.
classmethod GeoDataFrame.from_postgis(sql, con, geom_col='geom', crs=None, index_col=None, coerce_float=True, params=None)
I have looked at a previous similar question and also here which suggests to use SQLAlchemy's textual SQL option. However, this requires access to con.execute which isn't included in the GeoDataFrame from_postgis option.
Could you suggest the best way to pass the parameters to SQL? If not directly in from_postgis, then how best to construct the full SQL string separately and passing it in as the first sql argument to from_postgis.
For textual SQL, you can add parameters by using .bindparams:
query = text("select * from foo where bar = :a").bindparams(a=1)
For queries you construct in SQLAlchemy, bind parameters are automatically included:
foo = Table(...) # you need to define foo
query = select(["*"]).select_from(foo).where(foo.c.bar == 1)
You can also directly pass parameters via the params parameter of from_postgis, if that's more natural:
df.from_postgis(text("select * from foo where bar = :a"), params={"a": 1})
Do not use str.format as the other answer suggests because it's vulnerable to SQL injection.
Related
I have been trying to run a parameterized query using the client libraries in Python and I have tried both the named parameters and the positional parameters methods in the documentation, but neither of them worked.
Here's the positional parameters method:
And the output:
The code compiles with no error when I remove the last line of code:
I have a different function that does not use a parameterized query and works just fine:
How can I parameterize it with "id"?
I have tried the named parameters and the positional parameters methods described in the documentation but neither of them worked. I'm expecting to get data for the specific id parameter and put it in a data frame.
Try removing squar brackets from table name clause.
change
select * from [ Table_name ]
to this
select * from Table_name
I'm learning how to use SQL Alchemy, and I'm trying to re-implement a previously defined API but now using Python.
The REST API has the following query parameter:
myService/v1/data?range=time:2015-08-01:2015-08-02
So I want to map something like field:FROM:TO to filter a range of results, like a date range, for example.
This is what I'm using at this moment:
rangeStatement = range.split(':')
if(len(rangeStatement)==3):
query = query.filter(text('{} BETWEEN "{}" AND "{}"'.format(*rangeStatement)))
So, this will produce the following WHERE condition:
WHERE time BETWEEN "2015-08-01" AND "2015-08-02"
I know SQL Alchemy is a powerful tool that allows creating queries like Query.filter_by(MyClass.temp), but I need the API request to be as open as possible.
So, I'm worried that someone could pass something like DROP TABLE in the range parameter and exploit the text function
If queries are constructed using string formatting then sqlalchemy.text will not prevent SQL injection - the "injection" will already be present in the query text. However it's not difficult to build queries dynamically, in this case by using getattr to get a reference to the column. Assuming that you are using the ORM layer with model class Foo and table foos you can do
import sqlalchemy as sa
...
col, lower, upper = 'time:2015-08-01:2015-08-02'.split(':')
# Regardless of style, queries implement a fluent interface,
# so they can be built iteratively
# Classic/1.x style
q1 = session.query(Foo)
q1 = q1.filter(getattr(Foo, col).between(lower, upper))
print(q1)
or
# 2.0 style (available in v1.4+)
q2 = sa.select(Foo)
q2 = q2.where(getattr(Foo, col).between(lower, upper))
print(q2)
The respective outputs are (parameters will be bound at execution time):
SELECT foos.id AS foos_id, foos.time AS foos_time
FROM foos
WHERE foos.time BETWEEN ? AND ?
and
SELECT foos.id, foos.time
FROM foos
WHERE foos.time BETWEEN :time_1 AND :time_2
SQLAlchemy will delegate quoting of the values to the connector package being used by the engine, so you're protection against injection will be as good as that provided by the connector package*.
* In general I believe correct quoting should be a good defence against SQL injections, however I'm not sufficiently expert to confidently state that it's 100% effective. It will be more effective than building queries from strings though.
I'm trying to call the function json_extract_path which accepts variadic arguments. Does psycopg2 support queries of the form f(%s, %s, ...), which would accept a variable number of parameters?
The SQL format answer works, but I think it's far more complicated than you need. Just add the VARIADIC keyword and send your arguments as a list. Using the example from json_extract_path:
>>> cur.execute('''SELECT json_extract_path('{"f2":{"f3":1},"f4":{"f5":99,"f6":"foo"}}', VARIADIC %s)''', [['f4', 'f6']])
>>> cur.fetchall()
[('foo',)]
The answer is... mostly no, but kinda yes.
psycopg2 has no placeholder for variable arguments, AFAIK it doesn't parse the "format string" at all so it has no idea about context of use, it will just dumbly apply "data" escaping and formatting rules to individual parameters, so you need to create an SQL / format string with the correct number of placeholders, then flatten your sequence as "top-level" parameters.
However psycopg2 2.7 added a psycopg2.sql module which assists with "dynamic" SQL compositions. What you'd do here is use those facilities to generate your query e.g.
path = ['a', 'b', 'c']
cursor.execute(
sql.SQL('select json_extract_path(col, {}) from table').format(
sql.SQL(', ').join(sql.Placeholder() * len(path))
),
[*path]
)
(nb: code untested)
SQL.format validates that the formatted parameters are Composable, so they have been marked a safe explicitly
sql.Placeholder() is a Composable version of %s (it can take an optional name), as you can see it also supports "splatting" specifically for the case where you have a variable number of parameters e.g. VALUES enumeration or function calls
SQL.join is a Composable version of str.join
then you can just pass in your parameters normally, splatting your sequences to "flatten" them
I need to pass a partial raw sql query into sqlalchemy filter, like
s.query(account).filter("coordinate <#> point(%s,%s) < %s"%(lat,long,distance))
Yes, I'm trying to use earthdistance function in postgresql.
Of course, I could use PostGis and GeoAlchemy2, but I want to know the general solution to this kind of problems.
I know sqlalchemy can safely pass raw sql query .
result = db.engine.execute("select * coordinate <#> point(:lat,:long) < :distance",**params)
Is there any similar function that can be used to bind parameter of partial(?) sql query? I guess someone who implements custom sql function like func.ll_to_earth have used the function.
There is .params() on query. Try this:
query = s.query(account).filter(
"coordinate <#> point(:lat, :long_) < :dist").params(
lat=lat, long_=long_, dist=distance)
And there is the documentation on it.
Note: I renamed your long param, because there is alread a __builtin__ named long (long int) in python, it's good practice to not overwrite already used words for obvious reasons.
Can I implement the following in SQLAlchemy,
SELECT *
FROM TABLE
WHERE RIGHT(COLUMN_CODE, 2) = 'AX'
here RIGHT( ) returns the right part of a character string with the specified number of characters.
Is there a SQLAlchemy implementation of the RIGHT function?
You'd be better off using the .endswith() method instead:
select([tabledef]).where(tabledef.c.column_code.endswith('AX'))
or, when filtering with a mapper config and a session:
session.query(mappedobject).filter(mappedobject.column_code.endswith('AX'))
The column_code.endswith() method will be translated to whatever SQL is best for your particular engine, matching column values that end with AX.
You can always use the function generator to create arbitrary SQL functions if you have to use the RIGHT() sql function directly:
from sqlalchemy.sql.expression import func
select([tabledef]).where(func.right(tabledef.c.column_code, 2) == 'AX')
and the func.right() call will be translated to RIGHT(column_code, 2) by the SQL generation layer.
The documentation does not make it clear, but you can write any function using func.funcname sytle. funcname does not have to be defined natively by SQLAlchemy module. SQLAlchemy knows about common functions like min, max etc. and if there is dialect to dialect variation amongst those functions, SQLAlchemy takes care of that for you.
But the functions that SQLAlchemy does not know about are passed as is. So you can create your query that generates a SQL statement with the required RIGHT function like so
>>> from sqlalchemy import func
>>> select([table]).where(func.RIGHT(users.c.column_code, 2)='AX')