This question already has an answer here:
Dynamic SQL Queries with Python and mySQL
(1 answer)
Closed 3 months ago.
I am unable to execute the following statement I keep getting SQL syntax errors.
According to all the examples I can find this should work
Any help will be greatly appreciated.
d2 = df.iloc[-1,:]
q = symbol+'_ivol'
query = """SELECT close FROM %s WHERE date = %s"""
VALUES= (q, d2[1])
cursor.execute(query, VALUES)
ivol = cursor.fetchall()
conn.close()
Query parameters in SQL are not just string substitution. You can't use a query parameter for a table identifier. Parameters can only be used where you would normally use a quoted string literal or numeric literal.
Stated another way, all the identifiers must be fixed in the query string before you prepare it, because identifiers must be validated during the prepare phase, to make sure the table is a valid identifier, and that the table exists. You can't pass the name of a table identifier after the query has been prepared.
The Python driver unfortunately makes this more confusing because it uses %s instead of MySQL's own ? symbol for the parameter placeholder. This makes developers naturally think that %s is simply string substitution, like it is for Python string formatting.
So there's %s and there's %s, and they are handled differently. I know, it's confusing.
So you can do a plain string-formatting substitution to put your table into the query string:
query = """SELECT close FROM %s WHERE date = %%s""".format(q)
But it's more idiomatic for modern Python to use f-string formatting:
query = f"""SELECT close FROM `{q}` WHERE date = %s"""
I put back-ticks around the table name, just in case it's a SQL reserved keyword or something.
Then the other %s is an actual query parameter, because it works as a scalar value in the SQL expression. In this query, there is just one query parameter.
VALUES= [ d2[1] ]
cursor.execute(query, VALUES)
I have an query string in Python as follows:
query = "select name from company where id = 13 order by name;"
I want to be able to change the id dynamically. Thus I want to find id = 13 and replace it with a new id.
I can do it as follows:
query.replace("id = 13", "id = {}".format(some_new_id))
But if in the query is id= 13 or id=13 or id =13, ... it will not work.
How to avoid that?
Gluing variables directly into your query leaves you vulnerable to SQL injection.
If you are passing your query to a function to be executed in your database, that function should accept additional parameters.
For instance,
query = "select name from company where id = %s order by name"
cursor.execute(query, params=(some_other_id,))
It is better to use formatted sql.
Ex:
query = "select name from company where id = %s order by name;".
cursor.execute(query, (id,))
The usual solution when it comes to dynamically building strings is string formatting, ie
tpl = "Hello {name}, how are you"
for name in ("little", "bobby", "table"):
print(tpl.format(name))
BUT (and that's a BIG "but"): you do NOT want to do this for SQL queries (assuming you want to pass this query to your db using your db's python api).
There are two reasons to not use string formatting here: the first one is that correctly handling quoting and escaping is tricky at best, the second and much more important one is that it makes your code vulnerable to SQL injections attacks.
So in this case, the proper solution is to use prepared statements instead:
# assuming MySQL which uses "%" as placeholder,
# consult your db-api module's documentation for
# the proper placeholder
sql = "select name from company where id=%s order by name"
cursor = yourdbconnection.cursor()
cursor.execute(sql, [your_id_here])
I hope i can be clear about my problem. thank you :)
I'm using impala connection (library: from impala.dbapi import connect).
In order to run a query i'm using the execute command:
cursor.execute(query.value, (year_var, month_var,day_var))
Generally - it works just fine, also with variables. the problem begins when i use a SQL LIKE statement (e.g. like '%seo' - which contain %s in it).
The 1st argument (query.value) is a string:
create table bi_db.search_terms as
select search_query,search_contain,count(*) searches
from (
select search_query,
case when lower(search_query) like '%logo%' then 'logo'
when lower(search_query) like '%google%' then 'google'
when lower(search_query) like '%facebook%' then 'facebook'
when lower(search_query) like '%instagram%' then 'instagram'
when lower(search_query) like '%etsy%' then 'etsy'
when lower(search_query) like '%seo%' then 'seo'
when lower(search_query) like '%social media%' then 'social media'
else 'else' end as search_contain
from traffic_db.traffic_parq a
where year = %s AND month = %s AND day = %s AND controller = 'search' and action in ('gigs','users')
and search_query is not null and search_query<>'' ) t
group by search_query,search_contain
the second argument of the cursor.execute (e.g. (year_var, month_var,day_var)) refer to %s i'm putting on the query i run in order to use dynamic variables.
** The problem is that the python thinks it has 5 arguments instead of only 3. that caused because i have %seo and %social in the LIKE statments **
Anyone encountered this kind of problem? know how to solve it?
Many thanks!
You could escape literal percent signs in the query (e.g. %%seo%%), although it would be cleaner to pass the patterns as parameters to execute() as well:
sql = """
create table bi_db.search_terms as
select search_query,search_contain,count(*) searches
from (
select search_query,
case when lower(search_query) like %s then 'logo'
...
"""
cursor.execute(sql, ('%logo%', ...))
I've been using this syntax with great success in python mysql.
search = "O'%" # find names like O'Brien or O'Connell...
cursor.execute ("""
select userid
from usertab
where name like %s
""" , (search))
But sometimes I need to build my sql string before I execute it like the following, but the substitution technique is different than above and doesn't work in all cases.
search = "O'%" # find names like O'Brien or O'Connell...
sql = """
select userid
from usertab
where name like '%s'
""" % (search)
cursor.execute(sql)
How can I achieve the same kind of string substitution that works well in the first example, without executing the cursor?
MySQLdb uses the connection's literal() method to escape the arguments, so you could use:
sql = """
select userid
from usertab
where name like %s
""" % cursor.connection.literal(search)
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.