Does SQLAlchemy provide a function to safely quote/escape a literal when constructing raw queries? I've read through the docs and have seen nothing, not sure if it is a protected internal?
I am aware parameterized statements are the correct way to handle this, but due to a fringe case/bug with pyodbc/mssql-driver this is not an option. As a result I am forced to execute a compiled ORM statement with literal_binds=True. Worst case scenario I would just limit the input to A-Z|0-9 or get creative with chained subqueries/aliases but I would prefer to avoid this.
e.g.
variable = sqlalchemy.escape_text(input("User Input")) // something like this
q = session.query(MyObj).filter(MyObj.colA == variable))
connection.execute(q.selectable.compile(compile_kwargs={'literal_binds': True}), bind=session.bind)
Related
I created this procedure that I call with the cursor.execute method. The problem that I'm having is that PYODBC sees extra parameters than what I've given.
In this example query, the "-" and "-" are being read as extra parameters by PYODBC. Does anyone know why this is the case? This is happening any time I do any string concatenation in Access.
def GetAccessResults(self):
with pyodbc.connect(SQL.DBPath) as con:
cursor = con.cursor()
if self.parameters == None:
cursor.execute('{{Call {}}}'.format(self.storedProc))
else:
callString = self.__CreateStoredProcString()
cursor.execute(callString, self.parameters)
returnValue = cursor.fetchall()
return returnValue
def __CreateStoredProcString(self):
questionMarks = ('?,' * len(self.parameters))[:-1]
return '{{Call {} ({})}}'.format(self.storedProc, questionMarks)
As OP found out, MS Access being both a frontend GUI application and backend database operates differently in running SQL. Usually, the backend mode tends to be closer to standard SQL, namely:
Quotes: In backend, single quotes are reserved for literals and double quotes for identifiers as opposed to being interchangeable inside MSAccess.exe.
Wildcards: In backend, by default wildcards for LIKE uses % and GUI uses * unless running Access database in SQL Server compatible syntax (ANSI 92) that uses the standard %. For this reason, consider Access' ALIKE (ANSI-Like) with % to be compatible in both Access modes. Interestingly, this is not the case with stored queries as OP uses but will if writing queries in code.
Parameter: In backend, any unquoted object not recognized in query is considered a named parameter and errs out. Meanwhile the GUI launches a pop-up Enter Parameter box (which beginners do not know is actually a runtime error) allowing typed answers to then be evaluated on fly.
GUI Objects: While in GUI, Access queries can reference controls in open forms and reports, even user-defined functions defined in standalone VBA modules, these same references will error out in backend which essentially runs Access in headless mode and only recognizes other tables and queries.
Optimization: See Allen Browne's differences in optimizations that can occur when creating queries in GUI vs backend especially referencing Access object library functions.
By the way, using LIKE on subquery evaluates one scalar to another scalar. In fact, Access will err out if subquery returns more than one row which potentially can occur with current setup.
Error 3354: At most one record can be returned by this subquery
In other databases, the evaluation runs on first row of subquery (which without ORDER BY can be a random row) and not all records of subquery. Instead, consider re-factoring SQL to use EXISTS clause:
PARAMETERS prmServiceName Tex(255);
SELECT c.*
FROM Charts c
WHERE EXISTS
(SELECT 1
FROM Services s
WHERE s.ServiceName = prmService_Name
AND c.FileName ALIKE '%-' & s.Service_Abbreviation & '-%');
Try using ampersands:
Select "*-" & Service_Abbreviation & "-*"
Also, Like expects a string wrapped in quotes, and your subquery doesn't return that. So perhaps:
Select "'*-" & Service_Abbreviation & "-*'"
I need to pass a partial raw sql query into sqlalchemy filter, like
s.query(account).filter("coordinate <#> point(%s,%s) < %s"%(lat,long,distance))
Yes, I'm trying to use earthdistance function in postgresql.
Of course, I could use PostGis and GeoAlchemy2, but I want to know the general solution to this kind of problems.
I know sqlalchemy can safely pass raw sql query .
result = db.engine.execute("select * coordinate <#> point(:lat,:long) < :distance",**params)
Is there any similar function that can be used to bind parameter of partial(?) sql query? I guess someone who implements custom sql function like func.ll_to_earth have used the function.
There is .params() on query. Try this:
query = s.query(account).filter(
"coordinate <#> point(:lat, :long_) < :dist").params(
lat=lat, long_=long_, dist=distance)
And there is the documentation on it.
Note: I renamed your long param, because there is alread a __builtin__ named long (long int) in python, it's good practice to not overwrite already used words for obvious reasons.
I want to minimize the number of database queries my application makes, and I am familiarizing myself more with Django's ORM. I am wondering, what are the cases where a query is executed.
For instance, this format is along the lines of the answer I'm looking for (for example purposes, not accurate to my knowledge):
Model.objects.get()
Always launches a query
Model.objects.filter()
Launches a query if objects is empty only
(...)
I am assuming curried filter operations never make additional requests, but from the docs it looks like filter() does indeed make database requests if it's the first thing called.
If you're using test cases, you can use this custom assertion included in django's TestCase: assertNumQueries().
Example:
with self.assertNumQueries(2):
x = SomeModel.objects.get(pk=1)
y = x.some_foreign_key_in_object
If the expected number of queries was wrong, you'd see an assertion failed message of the form:
Num queries (expected - actual):
2 : 5
In this example, the foreign key would cause an additional query even though there's no explicit query (get, filter, exclude, etc.).
For this reason, I would use a practical approach: Test or logging, instead of trying to learn each of the cases in which django is supposed to query.
If you don't use unit tests, you may use this other method which prints the actual SQL statements sent by django, so you can have an idea of the complexity of the query, and not just the number of queries:
(DEBUG setting must be set to True)
from django.db import connection
x = SomeModel.objects.get(pk=1)
y = x.some_foreign_key_in_object
print connection.queries
The print would show a dictionary of queries:
[
{'sql': 'SELECT a, b, c, d ... FROM app_some_model', 'time': '0.002'},
{'sql': 'SELECT j, k, ... FROM app_referenced_model JOIN ... blabla ',
'time': '0.004'}
]
Docs on connection.queries.
Of course, you can also combine both methods and use the print connection.queries in your test cases.
See Django's documentation on when querysets are evaluated: https://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated
Evaluation in this case means that the query is executed. This mostly happens when you are trying to access the results, eg. when calling list() or len() on it or iterating over the results.
get()in your example doesn't return a queryset but a model objects, therefore it is evaluated immediately.
If I want to check for the existence and if possible retrieve an object, which of the following methods is faster? More idiomatic? And why? If not either of the two examples I list, how else would one go about doing this?
if Object.objects.get(**kwargs).exists():
my_object = Object.objects.get(**kwargs)
my_object = Object.objects.filter(**kwargs)
if my_object:
my_object = my_object[0]
If relevant, I care about mysql and postgres for this.
Why not do this in a try/except block to avoid the multiple queries / query then an if?
try:
obj = Object.objects.get(**kwargs)
except Object.DoesNotExist:
pass
Just add your else logic under the except.
django provides a pretty good overview of exists
Using your first example it will do the query two times, according to the documentation:
if some_queryset has not yet been evaluated, but you
know that it will be at some point, then using some_queryset.exists()
will do more overall work (one query for the existence check plus an
extra one to later retrieve the results) than simply using
bool(some_queryset), which retrieves the results and then checks if
any were returned.
So if you're going to be using the object, after checking for existance, the docs suggest just using it and forcing evaluation 1 time using
if my_object:
pass
In my AppEngine project I have a need to use a certain filter as a base then apply various different extra filters to the end, retrieving the different result sets separately. e.g.:
base_query = MyModel.all().filter('mainfilter', 123)
Then I need to use the results of various sub queries separately:
subquery1 = basequery.filter('subfilter1', 'xyz')
#Do something with subquery1 results here
subquery2 = basequery.filter('subfilter2', 'abc')
#Do something with subquery2 results here
Unfortunately 'filter()' affects the state of the basequery Query instance, rather than just returning a modified version. Is there any way to duplicate the Query object and use it as a base? Is there perhaps a standard Python way of duping an object that could be used?
The extra filters are actually applied by the results of different forms dynamically within a wizard, and they use the 'running total' of the query in their branch to assess whether to ask further questions.
Obviously I could pass around a rudimentary stack of filter criteria, but I'd rather use the Query itself if possible, as it adds simplicity and elegance to the solution.
There's no officially approved (Eg, not likely to break) way to do this. Simply creating the query afresh from the parameters when you need it is your best option.
As Nick has said, you better create the query again, but you can still avoid repeating yourself. A good way to do that would be like this:
#inside a request handler
def create_base_query():
return MyModel.all().filter('mainfilter', 123)
subquery1 = create_base_query().filter('subfilter1', 'xyz')
#Do something with subquery1 results here
subquery2 = create_base_query().filter('subfilter2', 'abc')
#Do something with subquery2 results here