I'm trying to generate sql insert statements using sqlalchemy like this.
def get_sql(self):
"""Returns SQL as String"""
baz_ins = baz.insert().values(
Id=self._id,
Foo.Bar=self.foo_dot_bar,
)
return str(baz_ins.compile(dialect=mysql.dialect(),
compile_kwargs={"literal_binds": True}))
This returns keyword cannot be expression. Escaping the period like \. also doesn't work.
One solution I came up with is using FooDOTBar instead of Foo.Bar and then replacing all "DOT" with "." in the generated sql files, this corrupts some other data though and is not optimal. Any better suggestions to deal with this from the ground up?
In your query, values can be assigned with dictionary. So you can do something like:
baz_ins = baz.insert().values({"Id": self._id, "Foo.Bar": self.foo_dot_bar})
Check out the insert documentation for more options
There's an alternate form of .values where you pass in a dict instead:
baz.insert().values({
"Id": self._id,
"Foo.Bar": self.foo_dot_bar,
})
Related
I am trying to execute mysql query from python. I want the output
query = "UPDATE 'college_general' SET 'fees' = '180000' WHERE ('college_id' = '2')"
Below is the snippet of the code
def update(table, column, value):
return f"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
Instead Python is storing it like
query = 'UPDATE \'college_general\' SET \'fees\' = \'180000\' WHERE (\'college_id\' = \'2\')'
which is causing the script to fail. How can I achieve the desired output?
Thanks in advance!
You can replace the identifiers single quotes with backticks. For more detailed answers visit this question.
There are two types of quotes in MySQL:
' for enclosing string literals
` for enclosing identifiers such as table and column names
There are multiple issues here:
First, I suspect that the string handling bit of your program is actually working, but you are being confused by the external representation of strings. For example, if you do
x = "O'Reilly"
Python will, in some circumstances, display the string as
'O\'Reilly'
Second, I think you are using the wrong kind of quotes. Single quotes in SQL are for strings; MySQL uses backticks for names when necessary, while other SQL implementations usually use double quotes for this.
Third, AND THIS IS IMPORTANT! Do not use string manipulation for building SQL queries. The database library almost certainly has a feature for parametrized queries and you should be using that. Your query should look something like this:
query = 'UPDATE college_general SET fees = ? WHERE college_ID = ?'
cursor.execute(query, [180000, '2'])
but the details will depend on the DB library you are using. For example, some use %s instead of ?. This saves you from all kinds of headaches with quoting strings.
raw string is the simplest solution to your problem.
I believe the code below will achieve what you wanted.
def update(table, column, value):
return fr"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.
I'm having a problem with this sqlalchemy query:
def bvalue(value):
if isinstance(value, unicode):
value = re.sub('[^\w]', "", value).lower()
return value
basicValue = bvalue(someVariable)
q = self.session.query(sheet.id).\
filter(bvalue(sheet.column) == basicValue)
The bvalue function works. I'm trying to match values after stripping them from any special characters and capitalisation. The stripped variable does match with the stripped db value, but still the query is not retrieving any results.
What am I doing wrong? Can't you use custom methods in filters?
You are aware that SQLAlchemy translates your queries into plain SQL statements that are then emitted to your configured database?
So naturally you can't simply add arbitrary python functions, since they would have to be translated into SQL which can't be done in a generic way.
Aside from this general issue, bvalue(sheet.column) will simply return sheet.column (since it's not a unicode instance) and it is evaluated before creating the query. So your query is in fact equivalent to:
q = self.session.query(sheet.id).\
filter(sheet.column == basicValue)
How to get the regex into SQL depends on the database you're using. Check e.g.
REGEXP_LIKE in SQLAlchemy
for a some suggestions.
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.