Develop Reference

Compile query from raw string (without using .text(...)) using Sqlalchemy connection and Postgres

I am using Sqlalchemy 1.3 to connect to a PostgreSQL 9.6 database (through Psycopg).
I have a very, very raw Sql string formatted using Psycopg2 syntax which I can not modify because of some legacy issues:
statement_str = SELECT * FROM users WHERE user_id=%(user_id)s
Notice the %(user_id)s
I can happily execute that using a sqlalchemy connection just by doing:
connection = sqlalch_engine.connect()
rows = conn.execute(statement_str, user_id=self.user_id)
And it works fine. I get my user and all is nice and good.
Now, for debugging purposes I'd like to get the actual query with the %(user_id)s argument expanded to the actual value. For instance: If user_id = "foo", then get SELECT * FROM users WHERE user_id = 'foo'
I've seen tons of examples using sqlalchemy.text(...) to produce a statement and then get a compiled version. I have that thanks to other answers like this one or this one been able to produce a decent str when I have an SqlAlchemy query.
However, in this particular case, since I'm using a more cursor-specific syntax %(user_id) I can't do that. If I try:
text(statement_str).bindparams(user_id="foo")
I get:
This text() construct doesn't define a bound parameter named 'user_id'
So I guess what I'm looking for would be something like
conn.compile(statement_str, user_id=self.user_id)
But I haven't been able to get that.

Not sure if this what you want but here goes.
Assuming statement_str is actually a string:
import sqlalchemy as sa
statement_str = "SELECT * FROM users WHERE user_id=%(user_id)s"
params = {'user_id': 'foo'}
query_text = sa.text(statement_str % params)
# str(query_text) should print "select * from users where user_id=foo"

Ok I think I got it.
The combination of SqlAlchemy's raw_connection + Psycopg's mogrify seems to be the answer.
conn = sqlalch_engine.raw_connection()
try:
cursor = conn.cursor()
s_str = cursor.mogrify(statement_str, {'user_id': self.user_id})
s_str = s_str.decode("utf-8") # mogrify returns bytes
# Some cleanup for niceness:
s_str = s_str.replace('\n', ' ')
s_str = re.sub(r'\s{2,}', ' ', s_str)
finally:
conn.close()
I hope someone else finds this helpful

Python MySQL Statement output: 0 or 1

I am trying to get Information from my database, and print it, but unfortunately, Instead of Printing the Information from the Table, it just Prints 0 or 1.
Why does it do this?
Can someone please help me?
sql = ("SELECT code FROM testing WHERE email = ((%s))")
sql2 = a.execute(sql, (fullemail))
sqlusername = ("SELECT username FROM testing123 WHERE code = ((%s))")
username = a.execute(sqlusername, (sql2))
print("Test3")
print(username)
Thank you.

The execute() method just returns the number of impacted rows.
You must use .fetchall() or equivalent (e.g. .fetchone()...) DBAPI methods to get a resultset.
Also, using parentheses alone around a single value: (fullemail) will not be recognized as a tuple, you need to explicitly add a comma so Python will recognize this as a tuple: (fullemail, )
sql = ("SELECT code FROM testing WHERE email = %s")
a.execute(sql, (fullemail, ))
sql2 = a.fetchall()
print(sql2)
sqlusername = ("SELECT username FROM testing123 WHERE code = %s")
a.execute(sqlusername, (sql2[0][0], ))
username = a.fetchall()
print("Test3")
print(username)
Depending on which library you are using:
MySQLdb (python 2.7)
mysqlclient (MySQLdb for python3)
PyMySQL (pure Python)
You can also use a DictCursor to get your result set rows as dict instead of list. Usage is like:
from pymysql.cursors import DictCursor
import pymysql
db = pymysql.connect(host="", user="", passwd="", cursorclass=DictCursor)
with db.cursor() as cur:
cur.execute("SELECT ...")
results = cur.fetchall()
This will give you a list of dictionaries instead of a list of lists.

TypeError When trying to insert byte data to MySQL query [duplicate]

Upon running this script:
#! /usr/bin/env python
import MySQLdb as mdb
import sys
class Test:
def check(self, search):
try:
con = mdb.connect('localhost', 'root', 'password', 'recordsdb');
cur = con.cursor()
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
ver = cur.fetchone()
print "Output : %s " % ver
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
finally:
if con:
con.close()
test = Test()
test.check("test")
I get an error of:
./lookup
Traceback (most recent call last):
File "./lookup", line 27, in <module>
test.check("test")
File "./lookup", line 11, in creep
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting
I have zero idea why. I'm trying to do parameterized querys, but it's been nothing but a pain. I'm somewhat new to Python, so it's probably an obvious problem.

Instead of this:
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
Try this:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", [search] )
See the MySQLdb documentation. The reasoning is that execute's second parameter represents a list of the objects to be converted, because you could have an arbitrary number of objects in a parameterized query. In this case, you have only one, but it still needs to be an iterable (a tuple instead of a list would also be fine).

You can try this code:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
You can see the documentation

'%' keyword is so dangerous because it major cause of 'SQL INJECTION ATTACK'.
So you just using this code.
cursor.execute("select * from table where example=%s", (example,))
or
t = (example,)
cursor.execute("select * from table where example=%s", t)
if you want to try insert into table, try this.
name = 'ksg'
age = 19
sex = 'male'
t = (name, age, sex)
cursor.execute("insert into table values(%s,%d,%s)", t)

cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
I do not why, but this works for me . rather than use '%s'.

The accepted answer by #kevinsa5 is correct, but you might be thinking "I swear this code used to work and now it doesn't," and you would be right.
There was an API change in the MySQLdb library between 1.2.3 and 1.2.5. The 1.2.3 versions supported
cursor.execute("SELECT * FROM foo WHERE bar = %s", 'baz')
but the 1.2.5 versions require
cursor.execute("SELECT * FROM foo WHERE bar = %s", ['baz'])
as the other answers state. I can't find the change in the changelogs, and it's possible the earlier behavior was considered a bug.
The Ubuntu 14.04 repository has python-mysqldb 1.2.3, but Ubuntu 16.04 and later have python-mysqldb 1.3.7+.
If you're dealing with a legacy codebase that requires the old behavior but your platform is a newish Ubuntu, install MySQLdb from PyPI instead:
$ pip install MySQL-python==1.2.3

I don't understand the first two answers. I think they must be version-dependent. I cannot reproduce them on MySQLdb 1.2.3, which comes with Ubuntu 14.04LTS. Let's try them. First, we verify that MySQL doesn't accept double-apostrophes:
mysql> select * from methods limit 1;
+----------+--------------------+------------+
| MethodID | MethodDescription | MethodLink |
+----------+--------------------+------------+
| 32 | Autonomous Sensing | NULL |
+----------+--------------------+------------+
1 row in set (0.01 sec)
mysql> select * from methods where MethodID = ''32'';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '9999'' ' at line 1
Nope. Let's try the example that Mandatory posted using the query constructor inside /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py where I opened "con" as a connection to my database.
>>> search = "test"
>>> "SELECT * FROM records WHERE email LIKE '%s'" % con.literal(search)
"SELECT * FROM records WHERE email LIKE ''test''"
>>>
Nope, the double apostrophes cause it to fail. Let's try Mike Graham's first comment, where he suggests leaving off the apostrophes quoting the %s:
>>> "SELECT * FROM records WHERE email LIKE %s" % con.literal(search)
"SELECT * FROM records WHERE email LIKE 'test'"
>>>
Yep, that will work, but Mike's second comment and the documentation says that the argument to execute (processed by con.literal) must be a tuple (search,) or a list [search]. You can try them, but you'll find no difference from the output above.
The best answer is ksg97031's.

According PEP8，I prefer to execute SQL in this way:
cur = con.cursor()
# There is no need to add single-quota to the surrounding of `%s`,
# because the MySQLdb precompile the sql according to the scheme type
# of each argument in the arguments list.
sql = "SELECT * FROM records WHERE email LIKE %s;"
args = [search, ]
cur.execute(sql, args)
In this way, you will recognize that the second argument args of execute method must be a list of arguments.
May this helps you.

I encountered this error while executing
SELECT * FROM table;
I traced the error to cursor.py line 195.
if args is not None:
if isinstance(args, dict):
nargs = {}
for key, item in args.items():
if isinstance(key, unicode):
key = key.encode(db.encoding)
nargs[key] = db.literal(item)
args = nargs
else:
args = tuple(map(db.literal, args))
try:
query = query % args
except TypeError as m:
raise ProgrammingError(str(m))
Given that I am entering any extra parameters, I got rid of all of "if args ..." branch. Now it works.

how to use variable as placeholder in python

i need help with variables in python, where they will be taken during run time, if i give the actual table name in the select query it works but when i am trying variable it fails.
import cx_Oracle
import csv
import sys
tablename = sys.argv[1] # pasing it as input at runtime
host = 'server.com'
port = 7111
SERVICE_NAME = 'gtpts'
login = 'USER'
passwrd = 'password'
SID = 'server.com'
dsn = cx_Oracle.makedsn(host, port, SID).replace('SID','SERVICE_NAME')
con = cx_Oracle.connect(login, passwrd, dsn)
cur = con.cursor()
cur.execute('select * from table') # direct table name works but variable doesnot
row = cur.fetchall()
Getting error as below:
[20020663]:20020663> python oracleconnect.py employee
Traceback (most recent call last):
File "oracleconnect.py", line 16, in <module>
cur.execute('select * from tablename')
cx_Oracle.DatabaseError: ORA-00911: invalid character
I could do the same by using $variable in perl do we have anything in python for same.

This uses the Oracle Bind Variable syntax, e.g.:
my_tablename = "SomeTableName"
cur = connection.cursor()
sql = "SELECT * FROM :tablename"
param = {"tablename": my_tablename}
cur.execute(sql, param)
row = cur.fetchall()
Note that the parameters here are passed as a dict and Oracle DB API will handle the escaping and quoting of variables, such as your table name. This method, as you can read about in the above link, is also much faster than doing the "hard parse" of the string.
Be careful not to use the string formatting operator %, like so:
cur.execute('SELECT * FROM %s' % my_tablename)
because it does not do any escaping or quoting, and it's prone to things like SQL injection, (not to mention it's less performant).

Thanks for this post. this has helped me to develop parameter driven connection strings.
following parameters are passed when the python script is run, after that python logs me into the database and I can perform my inserts.
import cx_Oracle
import csv
import sys
host_args = sys.argv[1]
port_args = sys.argv[2]
SID_args = sys.argv[3]
login_args = sys.argv[4]
passwrd_args = sys.argv[5]
data_dir = sys.argv[6]
dsn = cx_Oracle.makedsn(host_args, port_args, SID_args).replace('SID','SERVICE_NAME')
con = cx_Oracle.connect(login_args, passwrd_args, dsn)
The following command, I type on the command line to run this python script.
python /dera/ordsDataBranch/nsar/Load_NSARs_to_Oracle.py "myservername" "1521" "devrf1" "hr" "password2016" "/dera/ordsDataBranch/nsar/data"

Mysqldb Update error with set %s

I have created a database with MySQLdb.
In database I have a table with name student with columns:
id(is int),
id_user(is int),
f_name(is str),
l_name(is str)
I want to update a row.
My code is below:
db=mdb.connect(host="localhost", use_unicode="True", charset="utf8",
user="", passwd="", db="test")
# prepare a cursor object using cursor() method
cursor = db.cursor()
sql="""SELECT id_user FROM student"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
rows = cursor.fetchall()
the=int(7)
se=str('ok')
for row in rows:
r=int(row[0])
if r==the:
sql2 = """UPDATE student
SET f_name=%s
WHERE id_user = %s"""% (se,the)
# Execute the SQL command
cursor.execute(sql2)
# Commit your changes in the database
db.commit()
db.rollback()
# disconnect from server
db.close()
When I run it I take the error there is column with name ok why?
Can anyone help me find what I am doing wrong please?

str doesn't wrap its argument in quotation marks, so your statement is this:
UPDATE student SET f_name=ok WHERE id_user = 7
when it needs to be this:
UPDATE student SET f_name='ok' WHERE id_user = 7
So, either change this line:
SET f_name=%s
to this:
SET f_name='%s'
or else change this line:
se=str('ok')
to this:
se="'" + str('ok') + "'"
Though I recommend reading about SQL injection, which will become a concern as soon as you start using user-supplied data instead of hard-coded values.

You should run the query like this:
sql2 = """UPDATE student
SET f_name = %s
WHERE id_user = %s"""
cursor.execute(sql2, (se, the))
Don't use string interpolation, let the database driver handle passing the parameters for you. Otherwise you have to deal with syntax errors like this, or worse, SQL injection.
More details here.

You should always enclose your data with quotes.
Instead of %s solely use '%s' the only types you dont need it are numeric ones, but even there i would enclose %d with '%d' cos it is more save.
And you should use at least db.escape_string(your_data) before inserting or updating same values into your database.
Or have a look at the pdo-using style of mysqldb:
http://mysql-python.sourceforge.net/MySQLdb.html#some-examples
c=db.cursor()
max_price=5
c.execute("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < %s""", (max_price,))