I am working in Django 1.8 and doing some raw SQL queries using connection.cursor.
My question is about how to safely supply multiple parameters to the cursor. Here is my code:
cursor = connection.cursor()
query = "SELECT cost, id, date, org_id FROM mytable "
query += " WHERE ("
for i, c in enumerate(codes):
query += "id=%s "
if (i != len(codes)-1):
query += ' OR '
query += " AND "
for i, c in enumerate(orgs):
query += "org_id=%s "
if (i != len(orgs)-1):
query += ' OR '
cursor.execute(query, tuple(codes), tuple(orgs))
But this gives me:
TypeError: execute() takes at most 3 arguments (4 given)
I'm trying to follow the PEP documentation on execute, it says that one can use executemany instead, but that doesn't seem to help either:
cursor.executemany(query, [tuple(codes), tuple(orgs)])
I just can't follow the PEP documentation without an example. Could anyone help?
Your problem is that you're passing more arguments to execute than it accepts. What you need is to combine the query's parameters into a single tuple. One way to do that is to use itertools.chain to chain both lists' elements into one iterable that can be used to create a single tuple:
import itertools
cursor.execute(query, tuple(itertools.chain(codes, orgs)))
Related
I have inherited some code similar to below. I understand the concept of passing values to make a query dynamic(in this case field_id) but I don't understand what the benefit of taking the passed-in field_id list and putting it into a dictionary parameters = {"logical_field_id": field_id} before accessing the newly created dictionary to build the SQL statement. Along the same line why return parameters=parameters rather than just listing parameters in the return? I assume this is all the make the request more secure but I would like to better understand of why/how as I need to take on a similar task on a slightly more complex query that is below
def get_related_art(self, field_id):
parameters = {"logical_field_id": field_id}
sql = (
"SELECT a.id AS id,"
" a.name AS name,"
" a.description AS description,"
" a.type AS type,"
" a.subtype AS subtype "
" FROM ArtclTbl AS a INNER JOIN ("
" SELECT article_id AS id FROM LogFldArtclTbl"
" WHERE logical_field_id = %(logical_field_id)s"
" ORDER BY a.name"
)
return self.query(sql, parameters=parameters)
My reason for asking this question is I was asked to parameterize this
def get_group_fields(self, exbytes=None):
parameters = {}
where_clause = (
f"WHERE eig_eb.ebyte in ({', '.join(str(e) for e in ebytes)})" if ebytes else ""
)
sql = (
"SELECT l.id AS id, "
" eig_eb.ebyte AS ebyte, "
" eig.id AS instrument_group_id, "
" eig_lf.relationship_type AS relationship "
....
f" {where_clause}"
)
I started to modify code to iterate when setting the parameters and then accessing that value in the original location. This 'works' except now the query string returns ([ebyte1, ebyte2] instead of (ebyte1, ebyte2). I could modify the string to work around this but i really wanted to understand the why of this first.
parameters = {"exbytes": ', '.join(str(e) for e in exbytes)}
...
where_clause = (
f"WHERE eig_eb.exbyte in " + str(exbytes) if exbytes else ""
The benefit of using named parameter placeholders is so you can pass the parameter values as a dict, and you can add values to that dict in any order. There's no benefit in the first example you show, because you only have one entry in the dict.
There's no benefit in the second example either, because the parameters are part of an IN() list, and there are no other parameterized parts of the query. The order of values in an IN() list is irrelevant. So you could just use positional parameters instead of named parameters.
where_clause = (
f"WHERE eig_eb.ebyte in ({', '.join('%s' for e in ebytes)})" if ebytes else ""
)
Then you don't need a dict at all, you can just pass the ebytes list as the parameters.
Using the syntax parameters=parameters looks like a usage of keyword arguments to a Python function. I don't know the function self.query() in your example, but I suppose it accepts keyword arguments to implement optional arguments. The fact that your local variable is the same name as the keyword argument name is a coincidence.
While reading this question: SQL Multiple Updates vs single Update performance
I was wondering how could I dynamically implement an update for several variables at the same time using a connector like MariaDB's. Reading the official documentation I did not find anything similar.
This question is similar, and it has helped me to understand how to use parametrized queries with custom connectors but it does not answer my question.
Let's suppose that, from one of the views of the project, we receive a dictionary.
This dictionary has the following structure (simplified example):
{'form-0-input_file_name': 'nofilename', 'form-0-id': 'K0944', 'form-0-gene': 'GJXX', 'form-0-mutation': 'NM_0040(p.Y136*)', 'form-0-trix': 'ZSSS4'}
Assuming that each key in the dictionary corresponds to a column in a table of the database, if I'm not mistaken we would have to iterate over the dictionary and build the query in each iteration.
Something like this (semi pseudo-code, probably it's not correct):
query = "UPDATE `db-dummy`.info "
for key in a_dict:
query += "SET key = a_dict[key]"
It is not clear to me how to construct said query within a loop.
What is the most pythonic way to achieve this?
Although this could work.
query = "UPDATE `db-dummy`.info "
for index, key in enumerate(a_dict):
query = query + ("," if index != 0 else "") +" SET {0} = '{1}'".format(key,a_dict[key])
You should consider parameterized queries for safety and security. Moreover, a dynamic dictionary may also raise other concerns, it may be best to verify or filter on a set of agreed keys before attempting such an operation.
query = "UPDATE `db-dummy`.info "
for index, key in enumerate(a_dict):
query = query + ("," if index != 0 else "") +" SET {0} = ? ".format(key)
# Then execute with your connection/cursor
cursor.execute(query, tuple(a_dict.values()) )
This is what I did (inspired by #ggordon's answer)
query = "UPDATE `db-dummy`.info "
for index, key in enumerate(a_dict):
if index == 0:
query = query + "SET {0} = ?".format(key)
else:
query = query + ", {0} = ?".format(key)
query += " WHERE record_id = " + record_id
And it works!
I am trying to create a program where a user can enter an operator i.e. <> or = and then a number for a database in pymysql. I have tried a number of different ways of doing this but unfortunately unsuccessful. I have two documents with display being one and importing display into the other document.
Docuemnt 1
def get_pop(op, pop):
if (not conn):
connect();
query = "SELECT * FROM city WHERE Population %s %s"
with conn:
cursor = conn.cursor()
cursor.execute(query, (op, pop))
x = cursor.fetchall()
return x
Document two
def city():
op = input("Enter < > or =: ")
population = input("Enter population: ")
pop = display.get_pop(op, population)
for p in pop:
print(pop)
I am getting the following error.
pymysql.err.ProgrammingError: (1064,......
Please help thanks
You can't do this. Parameterization works for values only, not operators or table names, or column names. You'll need to format the operator into the string. Do not confuse the %s placeholder here with Python string formatting; MySQL is awkward in that it uses %s for binding parameters, which clashes with regular Python string formatting.
The MySQL %s in a query string escapes the user input to protect against SQL Injection. In this case, I set up a basic test to see if the operation part submitted by the user was in a list of accepted operations.
def get_pop(op, pop):
query = "SELECT * FROM city WHERE Population {} %s" # Add a placeholder for format
with conn: # Where does this come from?
cursor = conn.cursor()
if op in ['=', '!=']:
cursor.execute(query.format(op), (pop,))
x = cursor.fetchall()
return x
You'll want to come up with some reasonable return value in the case that if op in ['=', '!='] is not True but that depends entirely on how you want this to behave.
After checking that op indeed contains either "<>" or "=" and that pop indeed contains a number you could try:
query = "SELECT * FROM city WHERE Population " + op + " %s";
Beware of SQL injection.
Then
cursor.execute(query, (pop))
Python 3.5.2, stdlib sqlite3.
I'm trying to issue a SQL query with a dynamic criterion in an IN operator in the WHERE clause:
bad code (doesn't work)
def top_item_counts_in_installations(cursor, installation_names):
return cursor.execute(
"SELECT itm.name, COUNT(*)"
" FROM Installations ins INNER JOIN Items itm ON itm.id=ins.itmid"
" WHERE ins.name IN (?)"
" GROUP BY itm.name"
" ORDER BY COUNT(*) DESC"
" LIMIT 3",
(installation_names,))
installation_names is a tuple, but the above does not work, obviously, with an error
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
So what I currently do is prepare a sequence of parameter holders based on the length of installation_names:
kludge (working but ugly)
def top_item_counts_in_installations(cursor, installation_names):
param_holders= ",".join("?" for i in installation_names)
return cursor.execute(
"SELECT itm.name, COUNT(*)"
" FROM Installations ins INNER JOIN Items itm ON itm.id=ins.itmid"
" WHERE ins.name IN (%s)"
" GROUP BY itm.name"
" ORDER BY COUNT(*) DESC"
" LIMIT 3" % param_holders,
installation_names)
Is there a proper way to parameterize the right term of IN instead of my kludge? I haven't managed to find something in the documentation.
I am trying to query a mysql db from python but having troubles generating the query ebcasue of the wildcard % and python's %s. As a solution I find using ?, but when I run the following,
query = '''select * from db where name like'Al%' and date = '%s' ''', myDateString
I get an error
cursor.execute(s %'2015_05_21')
ValueError: unsupported format character ''' (0x27) at index 36 (the position of %)
How can i combine python 2.7 string bulding and sql wildcards? (The actual query is a lot longer and involves more variables)
First of all, you need to escape the percent sign near the Al:
'''select * from db where name like 'Al%%' and date = '%s''''
Also, follow the best practices and pass the query parameters in the second argument to execute(). This way your query parameters would be escaped and you would avoid sql injections:
query = """select * from db where name like 'Al%%' and date = %s"""
cursor.execute(query, ('2015_05_21', ))
Two things:
Don't use string formatting ('%s' % some_var) in SQL queries. Instead, pass the string as a sequence (like a list or a tuple) to the execute method.
You can escape your % so Python will not expect a format specifier:
q = 'SELECT foo FROM bar WHERE zoo LIKE 'abc%%' and id = %s'
cursor.execute(q, (some_var,))
Use the format syntax for Python string building, and %s for SQL interpolation. That way they don't conflict with each other.
You are not using the ? correctly.
Here's an example:
command = '''SELECT M.name, M.year
FROM Movie M, Person P, Director D
WHERE M.id = D.movie_id
AND P.id = D.director_id
AND P.name = ?
AND M.year BETWEEN ? AND ?;'''
*Execute the command, replacing the placeholders with the values of
the variables in the list [dirName, start, end]. *
cursor.execute(command, [dirName, start, end])
So, you want to try:
cursor.execute(query,'2015_05_21')