Check out the following lines of Python 2.6 code I found:
key = 'hire_date'
update_dict['key'] = update_dict[key] #added e.g. {..., 'key': '12/31/1999'}
if key == 'hire_date':
query_string = "UPDATE employee SET " + key + "= CAST(%(key)s AS DATE) WHERE emp_id = '" + emp.employee + "'"
I've tested this code, and it works. It successfully updates the employee's hire_date field in the database to whatever date 'key''s value in the dictionary is.
I was in the middle of parameterizing it when I noticed the %(key)s somehow manages to get the value of the dictionary at 'key'. How does it do that? I always thought you had to add % dictionaryOrTupleOrWhatever after the string for this to work.
In the code that you pasted, no interpolation takes place. However, the variable could later be interpolated:
>>> x = "%(var)s" # no interpolation yet
>>> d = {'var': 88}
>>> x % d # interpolate into the stored string
'88'
Given that the code you posted looks like SQL, it could also be interpolated later via an SQL library call. Many SQL interface libraries provide a similar sort of string-substitution using % signs and encourage users to use these rather than the built-in string substitution, since the SQL library versions have various safeguards to prevent malicious injection attacks.
I bet you'll find later in the code that there is a DB API execute statement that takes update_dict as a parameter. The DB API then does the substitution instead of Python string formatting and thus properly handles binding.
Have a look at this: http://furius.ca/pubcode/pub/antiorm/lib/python/dbapiext.html#escaping-in-the-dbapi-2-0
Related
I wrote a python script to update currency exchange rates using API calls. I successfully parsed the json results and extracted individual exchange rates as floats. I am however struggling with formatting/implementing an SQL table update loop.
Here is a code snippet that is tripping me up, assume that val = an actual exchange rate variable that is supplied from the API fetching/parsing section of the code:
mycursor = mydb.cursor()
sql = "UPDATE currencies SET coefficient = %s"
val = 0.03137
mycursor.execute(sql, val)
mydb.commit()
Running this gives me the following error:
Could not process parameters: float(0.03137), it must be of type list, tuple or dict:
I do not even know what to search for in order to reach an explanation that I understand and that helps me implement what I want correctly.
From: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
Note
In Python, a tuple containing a single value must include a comma. For
example, ('abc') is evaluated as a scalar while ('abc',) is evaluated
as a tuple.
Thus in my example the expression that works is:
val = (0.03137, )
I am trying to execute mysql query from python. I want the output
query = "UPDATE 'college_general' SET 'fees' = '180000' WHERE ('college_id' = '2')"
Below is the snippet of the code
def update(table, column, value):
return f"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
Instead Python is storing it like
query = 'UPDATE \'college_general\' SET \'fees\' = \'180000\' WHERE (\'college_id\' = \'2\')'
which is causing the script to fail. How can I achieve the desired output?
Thanks in advance!
You can replace the identifiers single quotes with backticks. For more detailed answers visit this question.
There are two types of quotes in MySQL:
' for enclosing string literals
` for enclosing identifiers such as table and column names
There are multiple issues here:
First, I suspect that the string handling bit of your program is actually working, but you are being confused by the external representation of strings. For example, if you do
x = "O'Reilly"
Python will, in some circumstances, display the string as
'O\'Reilly'
Second, I think you are using the wrong kind of quotes. Single quotes in SQL are for strings; MySQL uses backticks for names when necessary, while other SQL implementations usually use double quotes for this.
Third, AND THIS IS IMPORTANT! Do not use string manipulation for building SQL queries. The database library almost certainly has a feature for parametrized queries and you should be using that. Your query should look something like this:
query = 'UPDATE college_general SET fees = ? WHERE college_ID = ?'
cursor.execute(query, [180000, '2'])
but the details will depend on the DB library you are using. For example, some use %s instead of ?. This saves you from all kinds of headaches with quoting strings.
raw string is the simplest solution to your problem.
I believe the code below will achieve what you wanted.
def update(table, column, value):
return fr"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
I have wrote a query which has some string replacements. I am trying to update a url in a table but the url has % signs in which causes a tuple index out of range exception.
If I print the query and run in manually it works fine but through peewee causes an issue. How can I get round this? I'm guessing this is because the percentage signs?
query = """
update table
set url = '%s'
where id = 1
""" % 'www.example.com?colour=Black%26white'
db.execute_sql(query)
The code you are currently sharing is incredibly unsafe, probably for the same reason as is causing your bug. Please do not use it in production, or you will be hacked.
Generally: you practically never want to use normal string operations like %, +, or .format() to construct a SQL query. Rather, you should to use your SQL API/ORM's specific built-in methods for providing dynamic values for a query. In your case of SQLite in peewee, that looks like this:
query = """
update table
set url = ?
where id = 1
"""
values = ('www.example.com?colour=Black%26white',)
db.execute_sql(query, values)
The database engine will automatically take care of any special characters in your data, so you don't need to worry about them. If you ever find yourself encountering issues with special characters in your data, it is a very strong warning sign that some kind of security issue exists.
This is mentioned in the Security and SQL Injection section of peewee's docs.
Wtf are you doing? Peewee supports updates.
Table.update(url=new_url).where(Table.id == some_id).execute()
I've seen a couple similar threads, but attempting to escape characters isn't working for me.
In short, I have a list of strings, which I am iterating through, such that I am aiming to build a query that incorporates however many strings are in the list, into a 'Select, Like' query.
Here is my code (Python)
def myfunc(self, cursor, var_list):
query = "Select var FROM tble_tble WHERE"
substring = []
length = len(var_list)
iter = length
for var in var_list:
if (iter != length):
substring.append(" OR tble_tble.var LIKE %'%s'%" % var)
else:
substring.append(" tble_tble.var LIKE %'%s'%" % var)
iter = iter - 1
for str in substring:
query = query + str
...
That should be enough. If it wasn't obvious from my previously stated claims, I am trying to build a query which runs the SQL 'LIKE' comparison across a list of relevant strings.
Thanks for your time, and feel free to ask any questions for clarification.
First, your problem has nothing to do with SQL. Throw away all the SQL-related code and do this:
var = 'foo'
" OR tble_tble.var LIKE %'%s'%" % var
You'll get the same error. It's because you're trying to do %-formatting with a string that has stray % signs in it. So, it's trying to figure out what to do with %', and failing.
You can escape these stray % signs like this:
" OR tble_tble.var LIKE %%'%s'%%" % var
However, that probably isn't what you want to do.
First, consider using {}-formatting instead of %-formatting, especially when you're trying to build formatted strings with % characters all over them. It avoids the need for escaping them. So:
" OR tble_tble.var LIKE %'{}'%".format(var)
But, more importantly, you shouldn't be doing this formatting at all. Don't format the values into a SQL string, just pass them as SQL parameters. If you're using sqlite3, use ? parameters markers; for MySQL, %s; for a different database, read its docs. So:
" OR tble_tble.var LIKE %'?'%"
There's nothing that can go wrong here, and nothing that needs to be escaped. When you call execute with the query string, pass [var] as the args.
This is a lot simpler, and often faster, and neatly avoids a lot of silly bugs dealing with edge cases, and, most important of all, it protects against SQL injection attacks.
The sqlite3 docs explain this in more detail:
Usually your SQL operations will need to use values from Python variables. You shouldn’t assemble your query using Python’s string operations… Instead, use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method. (Other database modules may use a different placeholder, such as %s or :1.) …
Finally, as others have pointed out in comments, with LIKE conditions, you have to put the percent signs inside the quotes, not outside. So, no matter which way you solve this, you're going to have another problem to solve. But that one should be a lot easier. (And if not, you can always come back and ask another question.)
You need to escape % like this you need to change the quotes to include the both % generate proper SQL
" OR tble_tble.var LIKE '%%%s%%'"
For example:
var = "abc"
print " OR tble_tble.var LIKE '%%%s%%'" % var
It will be translated to:
OR tble_tble.var LIKE '%abc%'
This is an old question so here is what I had to do to make this work with recent releases of all software mentioned above:
citp = "SomeText" + "%%" # if your LIKE wants database rows that match start text, else ...
citp = "%%" + "SomeQueryText" + "%%"
chek_copies = 'SELECT id, code, null as num from indicator WHERE code LIKE "%s" AND owner = 1 ;'
check_copies = (chek_copies % (citp))
copies_checked = pd.read_sql(check_copies, con=engine)
Works like a charm - but what a load of trial and error
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.