I think my code is not very pythonic ，How to optimize？
code
lamp_keys=["ids"]
if len(lamp_keys) == 1:
rsql = f"""
SELECT * from brelation
where source_key = '{lamp_keys[0]}' and target_concept='aaa'
"""
else:
rsql = f"""
SELECT * from brelation
where source_key in {tuple(lamp_keys)} and target_concept='aa'
"""
robjs = RelationModel.objects.raw(rsql)
if lame_keys length is 1, if use in must be error, for example:
SELECT * from `brelation` WHERE source_key in ('xx',) and target_concept='aa'
You make the very dangerous mistake of formatting raw SQL yourself. Suppose in the part:
where source_key = '{lamp_keys[0]}'
Some user decides they want to be malicious and gives some value like '; DROP TABLE brelation; -- for lamp_keys[0] you can imagine what might happen yes? Your table will get dropped! This is called SQL Injection.
When you want to use some user defined value in a raw query always use parameterized queries. Reference: Passing parameters into raw() [Django docs]:
rsql = f"SELECT * from brelation where source_key in %s and target_concept='aaa'"
robjs = RelationModel.objects.raw(rsql, [lamp_keys])
Moving further it doesn't look like you even need to use raw queries, something like below would be equivalent to your raw query:
robjs = RelationModel.objects.filter(source_key__in=lamp_keys, target_concept='aaa')
Related
Im not sure exactly whats happening here but it might have something to do with format in python.
Running this causes an error.
x = '00000201000012EB'
sql = """ SELECT * FROM table WHERE id = {} """.format(x)
conn.execute(sql)
I get an error saying: syntax error near "EB"
however when i run the command like this:
sql = """ SELECT * FROM table WHERE id = '00000201000012EB' """
conn.execute(sql)
It works fine.
Is there something wrong with the way im formatting this sql statement?
Use the variable as an argument to execute():
cur.execute(""" SELECT * FROM my_table WHERE id = %s """, (x,))
If you are determined to use format(), you should add single quotes around the placeholder:
sql = """ SELECT * FROM my_table WHERE id = '{}' """.format(x)
Believe it or not it was fixed by adding more quotes to the string.
this finally worked.
x = '00000201000012EB'
sql = """ SELECT * FROM table WHERE id = {} """.format("'" + x + "'")
Since the sql statement required another set of quotes i just added them to ensure it was treated as its own string.
I'm trying to create a DataFrame through a sql query with pandas read_sql_query method. The query has a where clause that includes a like operation but it also includes a = operation that depends on a variable. The issue is that python is interpreting the % in the like operation as a place holder, just like in the = variable operation which is something I DO want.
Here's an example of it:
sql_string = """ SELECT a,b from table WHERE a = %(variable)s
AND b like '%fixed_chars%' """
params = {'variable':'AA'}
df = pandas.read_sql_query(sql_string, params=params, con=connection)
The error that I get is TypeError: not enough arguments for format string since it interprets the % you usually use as wildcard in mysql as the place holder in python.
In this case, you'll have to use two % for those not being formatting placeholders:
sql_string = "SELECT a,b from table WHERE a = %(variable)s AND \
b like '%%fixed_chars%%'"
Hope this helps!
I'm having trouble with SQLAlchemy when doing a raw SQL which checks against multiple values.
my_sess.execute(
"SELECT * FROM table WHERE `key`='rating' AND uid IN :uids",
params=dict(uids=some_list)
).fetchall()
There are 2 scenarios for this query, one that works and one that doesn't. If some_list = [1], it throws me an SQL error that I have a syntax error near ). But if some_list = [1, 2], the query executes successfully.
Any reason why this would happen?
No, SQL parameters only ever deal with scalar values. You'll have to generate the SQL here; if you need raw SQL, use:
statement = "SELECT * FROM table WHERE `key`='rating' AND uid IN ({})".format(
', '.join([':i{}'.format(i) for i in range(len(some_list))]))
my_sess.execute(
statement,
params={'i{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
e.g. generate enough parameters to hold all values in some_list with string formatting, then generate matching parameters to fill them.
Better still would be to use a literal_column() object to do all the generating for you:
from sqlalchemy.sql import literal_column
uid_in = literal_column('uid').in_(some_list)
statement = "SELECT * FROM able WHERE `key`='rating' AND {}".format(uid_in)
my_sess.execute(
statement,
params={'uid_{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
but then you perhaps could just generate the whole statement using the `sqlalchemy.sql.expression module, as this would make supporting multiple database dialects much easier.
Moreover, the uid_in object already holds references to the right values for the bind parameters; instead of turning it into a string as we do with the str.format() action above, SQLAlchemy would have the actual object plus the associated parameters and you would no longer have to generate the params dictionary either.
The following should work:
from sqlalchemy.sql import table, literal_column, select
tbl = table('table')
key_clause = literal_column('key') == 'rating'
uid_clause = literal_column('uid').in_(some_list)
my_sess.execute(select('*', key_clause & uid_clause, [tbl]))
where the sqlalchemy.sql.select() takes a column spec (here hard-coded to *), a where clause (generated from the two clauses with & to generate a SQL AND clause) and a list of selectables; here your one sqlalchemy.sql.table() value.
Quick demo:
>>> from sqlalchemy.sql import table, literal_column, select
>>> some_list = ['foo', 'bar']
>>> tbl = table('table')
>>> key_clause = literal_column('key') == 'rating'
>>> uid_clause = literal_column('uid').in_(some_list)
>>> print select('*', key_clause & uid_clause, [tbl])
SELECT *
FROM "table"
WHERE key = :key_1 AND uid IN (:uid_1, :uid_2)
but the actual object tree generated from all this contains the actual values for the bind parameters too, so my_sess.execute() can access these directly.
Please suggest is there way to write query multi-column in clause using SQLAlchemy?
Here is example of the actual query:
SELECT url FROM pages WHERE (url_crc, url) IN ((2752937066, 'http://members.aye.net/~gharris/blog/'), (3799762538, 'http://www.coxandforkum.com/'));
I have a table that has two columns primary key and I'm hoping to avoid adding one more key just to be used as an index.
PS I'm using mysql DB.
Update: This query will be used for batch processing - so I would need to put few hundreds pairs into the in clause. With IN clause approach I hope to know fixed limit of how many pairs I can stick into one query. Like Oracle has 1000 enum limit by default.
Using AND/OR combination might be limited by the length of the query in chars. Which would be variable and less predictable.
Assuming that you have your model defined in Page, here's an example using tuple_:
keys = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/')
]
select([
Page.url
]).select_from(
Page
).where(
tuple_(Page.url_crc, Page.url).in_(keys)
)
Or, using the query API:
session.query(Page.url).filter(tuple_(Page.url_crc, Page.url).in_(keys))
I do not think this is currently possible in sqlalchemy, and not all RDMBS support this.
You can always transform this to a OR(AND...) condition though:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter(or_(*(and_(Page.url_crc == crc, Page.url == url) for crc, url in filter_rows)))
print qry
should produce something like (for SQLite):
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE pages.url_crc = ? AND pages.url = ? OR pages.url_crc = ? AND pages.url = ?
-- (2752937066L, 'http://members.aye.net/~gharris/blog/', 3799762538L, 'http://www.coxandforkum.com/')
Alternatively, you can combine two columns into just one:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter((func.cast(Page.url_crc, String) + '|' + Page.url).in_(["{}|{}".format(*_frow) for _frow in filter_rows]))
print qry
which produces the below (for SQLite), so you can use IN:
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE (CAST(pages.url_crc AS VARCHAR) || ? || pages.url) IN (?, ?)
-- ('|', '2752937066|http://members.aye.net/~gharris/blog/', '3799762538|http://www.coxandforkum.com/')
I ended up using the test() based solution: generated "(a,b) in ((:a1, :b1), (:a2,:b2), ...)" with named bind vars and generating dictionary with bind vars' values.
params = {}
for counter, r in enumerate(records):
a_param = "a%s" % counter
params[a_param] = r['a']
b_param = "b%s" % counter
params[b_param] = r['b']
pair_text = "(:%s,:%s)" % (a_param, b_param)
enum_pairs.append(pair_text)
multicol_in_enumeration = ','.join(enum_pairs)
multicol_in_clause = text(
" (a,b) in (" + multicol_in_enumeration + ")")
q = session.query(Table.id, Table.a,
Table.b).filter(multicol_in_clause).params(params)
Another option I thought about using mysql upserts but this would make whole included even less portable for the other db engine then using multicolumn in clause.
Update SQLAlchemy has sqlalchemy.sql.expression.tuple_(*clauses, **kw) construct that can be used for the same purpose. (I haven't tried it yet)
I am new to python, I come here from the land of PHP. I constructed a SQL query like this in python based on my PHP knowledge and I get warnings and errors
cursor_.execute("update posts set comment_count = comment_count + "+str(cursor_.rowcount)+" where ID = " + str(postid))
# rowcount here is int
What is the right way to form queries?
Also, how do I escape strings to form SQL safe ones? like if I want to escape -, ', " etc, I used to use addslashes. How do we do it in python?
Thanks
First of all, it's high time to learn to pass variables to the queries safely, using the method Matus expressed. Clearer,
tuple = (foovar, barvar)
cursor.execute("QUERY WHERE foo = ? AND bar = ?", tuple)
If you only need to pass one variable, you must still make it a tuple: insert comma at the end to tell Python to treat it as a one-tuple: tuple = (onevar,)
Your example would be of form:
cursor_.execute("update posts set comment_count = comment_count + ? where id = ?",
(cursor_.rowcount, postid))
You can also use named parameters like this:
cursor_.execute("update posts set comment_count = comment_count + :count where id = :id",
{"count": cursor_.rowcount, "id": postid})
This time the parameters aren't a tuple, but a dictionary that is formed in pairs of "key": value.
from python manual:
t = (symbol,)
c.execute( 'select * from stocks where symbol=?', t )
this way you prevent SQL injection ( suppose this is the SQL safe you refer to ) and also have formatting solved