I'm to link my code to a MySQL database using pymysql. In general everything has gone smoothly but I'm having difficulty with the following function to find the minimum of a variable column.
def findmin(column):
cur = db.cursor()
sql = "SELECT MIN(%s) FROM table"
cur.execute(sql,column)
mintup = cur.fetchone()
if everything went smoothly this would return me a tuple with the minimum, e.g. (1,).
However, if I run the function:
findmin(column_name)
I have to put column name in "" (i.e. "column_name"), else Python sees it as an unknown variable. But if I put the quotation marks around column_name then SQL sees
SELECT MIN("column_name") FROM table
which just returns the column header, not the value.
How can I get around this?
The issue is likely the use of %s for the column name. That means the SQL Driver will try to escape that variable when interpolating it, including quoting, which is not what you want for things like column names, table names, etc.
When using a value in SELECT, WHERE, etc. then you do want to use %s to prevent SQL injections and enable quoting, among other things.
Here, you just want to interpolate using pure Python (assuming a trusted value; please see below for more information). That also means no bindings tuple passed to the execute method.
def findmin(column):
cur = db.cursor()
sql = "SELECT MIN({0}) FROM table".format(column)
cur.execute(sql)
mintup = cur.fetchone()
SQL fiddle showing the SQL working:
http://sqlfiddle.com/#!2/e70a41/1
In response to the Jul 15, 2014 comment from Colin Phipps (September 2022):
The relatively recent edit on this post by another community member brought it to my attention, and I wanted to respond to Colin's comment from many years ago.
I totally agree re: being careful about one's input if one interpolates like this. Certainly one needs to know exactly what is being interpolated. In this case, I would say a defined value within a trusted internal script or one supplied by a trusted internal source would be fine. But if, as Colin mentioned, there is any external input, then that is much different and additional precautions should be taken.
I am trying to use the sqlite3 module in python to do a database lookup in a table that only has one column. The column contains phone numbers in the format of:
9545551212
???5551212
Here's what I am running in python:
import sqlite3
cti = '/home/user/test/cti.db'
conn = sqlite3.connect(cti)
c = conn.cursor()
c.execute('select * from ani_table_1 where number = 9545551212')
<sqlite3.Cursor object at 0x7f6b435316c0>
When I run that exact same select statement in sqlite3 I get the expected result:
sqlite> select * from ani_table_1 where number = 9545551212;
9545551212
I'm using python 3.6.5 and sqlite 3.7.17
What have I got wrong in my code? Any help is much appreciated.
You didn't iterate over the result. The sqlite3 command line tool is not the same thing as Python code; the latter always prints the results, because it is a command-line tool and will make sure you don't get flooded with large result sets.
When accessing a database in code, however, the library can't assume you want to print out all the rows to the end user. You maybe wanted to do something different with the data instead.
So you need to loop over the cursor and print each row:
c.execute('select * from ani_table_1 where number = 9545551212')
for row in c:
print(*row, sep='\t')
You may want to familiarise yourself with how the Python database API standard works; search around for a good tutorial. At a glance, this specific tutorial looks like it covers the most important basics.
So I've been looking around but I can't seem to find answer to a seemingly simple and probably commonly asked question. In SQLite, I have a query that I want to pass via user defined search text.
search = xChatMessageSplit[2]
c.execute("SELECT * FROM captured WHERE Nick=? AND Name LIKE '%search%'",(xChatNick,search))
Obviously the syntax or whatever is incorrect since I'm getting errors, but I want to basically allow users to define a search term for string, "search." How would I go about doing this? I tried using REGEXP but I can't seem to figure out how to define the function so I figured I'd just go with LIKE since it's already implemented into SQLite3
You need to use ? to show where the parameter's value will be used.
c.execute("""SELECT * FROM captured
WHERE Nick=?
AND Name LIKE ('%' || ? || '%')""", (xChatNick,search))
I have a python3 program that I'm making which uses a sqlite database with several tables, I want to create a selector module to allow me to chose which table to pull data from.
I have found out that I can't use parameter substitution for a table name as shown bellow, so I'm looking for some alternative methods to accomplish this.
c.execute("SELECT * FROM ? ", DB)
Any ideas?
Right. You can not use parameter substitution to specify the table.
So instead you must do string manipulation:
c.execute("SELECT * FROM {t} ".format(t=tablename))
I don't know if this is a python3 thing but it seems easiest to just do this:
c.execute("SELECT * FROM %s "% tablename)
Blockquote *
Right. You can not use parameter
substitution to specify the table. So
instead you must do string
manipulation: c.execute("SELECT * FROM
{t} ".format(t=tablename))*
Blockquote
Thanks unutbu, this is just what I needed.
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.