I am trying to get a query into a variable called results, in which I query the database to find the books with a title like the input from the search bar received from a post method. The query I am running is as follows:
results = db.execute("SELECT * FROM books WHERE title LIKE (%:search%)", {"search": search}).fetchall();
With the above query, I get the following error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) syntax error at or near "%".
This works as expected if I remove the %, or if I manually give the LIKE a parameter (eg: LIKE ('%the%')), but this does not really give back any results unless the search is exactly as one of the book titles in the database, and it defeats the purpose of using variable substitution by hard coding the parameters. I am also wondering if it's possible to use ILIKE for case insensitive when querying with SQLAlchemy.
I am aware that I could use Object Relational Mapping, and use different functions such as the filter function and whatnot, but for this assignment we are meant to not use ORM and use simple queries. Any suggestions?
Pass the entire search string as the parameter to the LIKE operator:
results = db.execute(text("SELECT * FROM books WHERE title LIKE :search"),
{"search": f"%{search}%"}).fetchall();
or alternatively concatenate in the database:
results = db.execute(
text("SELECT * FROM books WHERE title LIKE ('%' || :search || '%')"),
{"search": search}).fetchall();
Related
Is there an SQL query builder (in Python) which allows me to "parse" and initial SQL query, add certain operators and then get the resulting SQL text?
My use case is the following:
Start with a query like: "SELECT * from my_table"
I want to be able to do something like query_object = Query.parse("SELECT * from my_table to get a query object I can manipulate and then write something like query_object.where('column < 10').limit(10) or similar (columns and operators could also be part of the library, may also have to consider existing WHERE clauses)
And finally getting the resulting query string str(query_object) with the final modified query.
Is this something that can be achieved with any of the ORMs? I don't need all the database connection to specific DB-engines or object mappings (although having it is not a limitation).
I've seen pypika, which allows to create an SQL query from code, but it doesn't allow one to parse an existing query and continue from there.
I've also seen sqlparse which allows me to parse and SQL query into tokens. But because it does not create a tree, it is non-trivial to add additional elements to am existing statement. (it is close to what I am looking for, if only it created an actual tree)
I am trying to get a query into a variable called results, in which I query the database to find the books with a title like the input from the search bar received from a post method. The query I am running is as follows:
results = db.execute("SELECT * FROM books WHERE title LIKE (%:search%)", {"search": search}).fetchall();
With the above query, I get the following error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) syntax error at or near "%".
This works as expected if I remove the %, or if I manually give the LIKE a parameter (eg: LIKE ('%the%')), but this does not really give back any results unless the search is exactly as one of the book titles in the database, and it defeats the purpose of using variable substitution by hard coding the parameters. I am also wondering if it's possible to use ILIKE for case insensitive when querying with SQLAlchemy.
I am aware that I could use Object Relational Mapping, and use different functions such as the filter function and whatnot, but for this assignment we are meant to not use ORM and use simple queries. Any suggestions?
Pass the entire search string as the parameter to the LIKE operator:
results = db.execute(text("SELECT * FROM books WHERE title LIKE :search"),
{"search": f"%{search}%"}).fetchall();
or alternatively concatenate in the database:
results = db.execute(
text("SELECT * FROM books WHERE title LIKE ('%' || :search || '%')"),
{"search": search}).fetchall();
Problem Summary:
I'm using Python to send a series of queries to a database (one by one) from a loop until a non-empty result set is found. The query has three conditions that must be met and they're placed in a where statement. Every iteration of the loop changes and manipulates the conditions from a specific condition to a more generic one.
Details:
Assuming the conditions are keywords based on a pre-made list ordered by accuracy such as:
Option KEYWORD1 KEYWORD2 KEYWORD3
1 exact exact exact # most accurate!
2 generic exact exact # accurate
3 generic generic exact # close enough
4 generic generic generic # close
5 generic+ generic generic # almost there
.... and so on.
On the database side, I have a description column that should contain all the three keywords either in their specific form or a generic form. When I run the loop in python this is what actually happens:
-- The first sql statement will be like
Select *
From MyTable
Where Description LIKE 'keyword1-exact$'
AND Description LIKE 'keyword2-exact%'
AND Description LIKE 'keyword3-exact%'
-- if no results, the second sql statement will be like
Select *
From MyTable
Where Description LIKE 'keyword1-generic%'
AND Description LIKE 'keyword2-exact%'
AND Description LIKE 'keyword3-exact%'
-- if no results, the third sql statement will be like
Select *
From MyTable
Where Description LIKE 'keyword1-generic%'
AND Description LIKE 'keyword2-generic%'
AND Description LIKE 'keyword3-exact%'
-- and so on until a non-empty result set is found or all keywords were used
I'm using the approach above to get the most accurate results with the minimum amount of irrelevant ones (the more generic the keywords, the more irrelevant results will show up and they will need additional processin)
Question:
My approach above is doing exactly what I want but I'm sure that it's not efficient.
What would be the proper way to do this operation in a query instead of Python loop (knowing that I only have a read access to the database so I can't store procedures)?
Here is an idea
select top 1
*
from
(
select
MyTable.*,
accuracy = case when description like keyword1 + '%'
and description like keyword2 + '%'
and description like keyword3 + '%'
then accuracy
end
-- an example of data from MyTable
from (select description = 'exact') MyTable
cross join
(values
-- generate full list like this in python
-- or read it from a table if it is in database
(1, ('exact'), ('exact'), ('exact')),
(2, ('generic'), ('exact'), ('exact')),
(3, ('generic'), ('generic'), ('exact'))
) t(accuracy, keyword1, keyword2, keyword3)
) t
where accuracy is not null
order by accuracy
I would not do a loop over database queries. Instead I would search for the least specific, i.e. most generic, keyword and return all these rows.
Select *
From MyTable
Where Description LIKE '%iPhone%'
This returns all the rows with iPhones. Now do the further processing, i.e. find the best match, in memory. This is much faster than multiple queries.
If you have several equally most generic keywords, then query them with OR
Select *
From MyTable
Where Description LIKE '%iPhone%' OR
Description LIKE '%i-Phone%'
But any case make only one query.
Please try using RegEx regular expression functionality of Sql server.
Or else you can try impoting re in python for regular expression.
First you can collect the data and then try re to achieve you r goal.
Hope this is helpful.
I'm trying to use both .contains and .ilike to filter a query from a string value.
For example the query should look something like this:
model.query.filter(model.column.contains.ilike("string").all()
Currently querying by .contains returns case sensitive results. And querying by ilike returns case unsensitive results, but the characters must match which is far too specific a query. I want potential results to appear to the user as they do with a .contains query.
foo = model.query.filter(model.column.ilike("%"+ string_variable +"%")).all()
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.