I'm trying to select a record from my MySQL db, the problem is that it returns with this error which i don't understand, i searched on the web for a solution, and there are many similar cases, but none apply to this specific one.
The expression i'm trying to execute is the following:
result = cursor.execute("""select * from urls where domain = '%s';"""%(found_url,))
it is in a try clause and it always goes right to the except giving me this error:
OperationalError(1054, "Unknown column 'http://..................' in 'field list'")
(i omitted the url, the dots act as placeholders)
after some cycles, being in a loop, it stops giving me that error and changes it to this:
ProgrammingError(2014, "Commands out of sync; you can't run this command now")
any idea? i'm going crazy on this one.
Use query parameters instead of how you are doing it with string interpolation. The protects you against SQL Injection attacks.
cursor.execute("""select * from urls where domain = %s""", (found_url,))
Notice the differences here:
The %s is NOT quoted, as the MySQLdb library will handle this automatically
You are not using string interpolation (no % after the string to replace your placeholders). Instead, you are passing a tuple as the second argument to the execute function.
Since you are only using one place holder and you still need to pass a tuple, there is a comma to indicate it is a tuple (found_url,)
Related
I'm to link my code to a MySQL database using pymysql. In general everything has gone smoothly but I'm having difficulty with the following function to find the minimum of a variable column.
def findmin(column):
cur = db.cursor()
sql = "SELECT MIN(%s) FROM table"
cur.execute(sql,column)
mintup = cur.fetchone()
if everything went smoothly this would return me a tuple with the minimum, e.g. (1,).
However, if I run the function:
findmin(column_name)
I have to put column name in "" (i.e. "column_name"), else Python sees it as an unknown variable. But if I put the quotation marks around column_name then SQL sees
SELECT MIN("column_name") FROM table
which just returns the column header, not the value.
How can I get around this?
The issue is likely the use of %s for the column name. That means the SQL Driver will try to escape that variable when interpolating it, including quoting, which is not what you want for things like column names, table names, etc.
When using a value in SELECT, WHERE, etc. then you do want to use %s to prevent SQL injections and enable quoting, among other things.
Here, you just want to interpolate using pure Python (assuming a trusted value; please see below for more information). That also means no bindings tuple passed to the execute method.
def findmin(column):
cur = db.cursor()
sql = "SELECT MIN({0}) FROM table".format(column)
cur.execute(sql)
mintup = cur.fetchone()
SQL fiddle showing the SQL working:
http://sqlfiddle.com/#!2/e70a41/1
In response to the Jul 15, 2014 comment from Colin Phipps (September 2022):
The relatively recent edit on this post by another community member brought it to my attention, and I wanted to respond to Colin's comment from many years ago.
I totally agree re: being careful about one's input if one interpolates like this. Certainly one needs to know exactly what is being interpolated. In this case, I would say a defined value within a trusted internal script or one supplied by a trusted internal source would be fine. But if, as Colin mentioned, there is any external input, then that is much different and additional precautions should be taken.
During an insert into MySQL 5.x via python 3.x I do receive an error which I can not find the reason for. It is a simple insert:
self.curr.execute("""
INSERT IGNORE INTO test (
`name`
)
VALUES (%s)
""", (
item['test']
)
)
Leading to this error:
Failed to save datasets. INSERT IGNORE INTO test (
`name`
)
VALUES (%s) Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s)' at line 4
Everything looks good to me, what is the problem with this insert?
Python's DB-API, the standard to which your database module was developed, specifies that there must be a "sequence or mapping" supplied as the parameters argument to .execute().
Your argument was a string, so what was placed into the query was not understood by the database.
What is needed is to make the second (2nd) argument to .execute() into a sequence, and the usual sequence to use is a tuple. It's sometimes a surprise to people that a tuple is not formed by enclosing a value in parentheses/brackets, and that the trailing comma is necessary. Note that if you were inserting multiple values you wouldn't have run into this issue at all.
So what is needed is:
self.curr.execute("""
INSERT IGNORE INTO test (
`name`
)
VALUES (%s)
""", (
item['test'],
# ^ trailing comma!
)
)
There is at least one database module which allows just one value to be used (i.e. not a sequence) -- in a departure from the DB-API specifications; which has thrown me off in the past.
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
I'm trying to write an SQL query in PyQt5 that updates some data in a table, but cannot get the query to work. I've read countless forums but as far as I can tell my code is correct. I also have read the documentation back to front so maybe I'm missing something?
I am using PyQt5, python3.5 and SQLITE. The following code (lastError/lastQuery not shown):
self.sqlWrite('ct','MarkerSize',123)
def sqlWrite(self,tbl,var,val):
query = QtSql.QSqlQuery(self.db) # First create query instance.
# Prepare query with placeholders, then bind values.
query.prepare('UPDATE :tbl SET value=:val WHERE property=:var')
query.bindValue(0,tbl)
query.bindValue(1,val)
query.bindValue(2,var)
# Finally execute query.
query.exec_()
...produces the error:
near "?": syntax error Unable to execute statement
near "?": syntax error Unable to execute statement
UPDATE :tbl SET value=:val WHERE property=:var
Parameter count mismatch
Have I lost the plot? What am I missing?
Thanks in advance.
A table name is not a parameter, so you cannot bind a value to it. Placeholders are intended for use with literal values, not arbitrary strings. For the latter, you should just use normal string interpolation:
query.prepare('UPDATE "%s" SET value=:val WHERE property=:var' % tbl)
query.bindValue(':val', val)
query.bindValue(':var', var)
For a more generic way to escape identifiers, use the query's driver:
tbl = query.driver().escapeIdentifier(tbl, QSqlDriver.TableName)
query.prepare('UPDATE %s SET value=:val WHERE property=:var' % tbl)
I used MySQL Connector/Python API, NOT MySQLdb.
I need to dynamically insert values into a sparse table so I wrote the Python code like this:
cur.executemany("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
where
data=[('Depth', '17.5cm', Decimal('3003')), ('Input_Voltage', '110 V AC', Decimal('3004'))]
But it resulted an error:
TypeError: not enough arguments for format string
Is there any solution for this problem? Is it possible to use executemany when there is a
substitution of a field in query?
Thanks.
Let's start with the original method:
As the error message suggests you have a problem with your SQL syntax (not Python). If you insert your values you are effectively trying to execute
UPDATE myTABLE SET 'Depth'='17.5cm' WHERE id='3003'
You should notice that you are trying to assign a value to a string 'Depth', not a database field. The reason for this is that the %s substitution of the mysql module is only possible for values, not for tables/fields or other object identifiers.
In the second try you are not using the substitution anymore. Instead you use generic python string interpolation, which however looks similar. This does not work for you because you have a , and a pair of brackets too much in your code. It should read:
cur.execute("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
I also replaced executemany with execute because this method will work only for a single row. However your example only has one row, so there is no need to use executemany anyway.
The second method has some drawbacks however. The substitution is not guaranteed to be quoted or formatted in a correct manner for the SQL query, which might cause unexpected behaviour for certain inputs and may be a security concern.
I would rather ask, why it is necessary to provide the field name dynamically in the first place. This should not be necessary and might cause some trouble.