Develop Reference

Mocking database call in a way that it checks the SQL query

I have a code that executes queries to redshift like this:
def send_sql_query(source, sql_query, lst=None):
connection = psycopg2.connect(
host=os.environ["REDSHIFT_HOST"],
port="5439",
dbname="dbname",
user=os.environ["REDSHIFT_USERNAME"],
password=os.environ["REDSHIFT_PASSWORD"],
cursor = connection.cursor()
cursor.execute(sql_query, lst)
sql_results = cursor.fetchall()
return sql_results
finally:
if connection:
connection.close()
I would like to mock the method in a way that it will retrieve and sql_query, and the method will hold a fake db data (preferable in json), but will execute the SQL on the fake data with the sql_query and return the result.
Using mock.return_value and mock.side_effect will not help, because I want to verify that the SQL query is correct. Writing a code to return results doesn't really check the SQL query
Is there a framework in python for it?

Testing the SQL requires a SQL engine. As different databases use different dialects and as you use PostgreSQL as you main database, you should install a PostgreSQL instance on you dev environment with fake data and redirect your queries there while testing.
As you use the environment to store the reference of the database, you have just to setup a test environment pointing to the test database.

stored proc not committing with sqlalchemy/pyodbc

I am using sqlalchemy/pyodbc to connect to a MS SQL 2012 server. I chose sqlalchemy because of the direct integration with pandas dataframes using .read_sql and .to_sql.
At a high level, my code is:
df = dataframe.read_sql("EXEC sp_getsomedata")
<do some stuff here>
finaldf.to_sql("loader_table", engine,...)
This part works great, very easy to read, etc. The problem is that I have to run a final stored proc to insert the data from the loader table into the live table. Normally, sqlalchemy knows to commit after INSERT/UPDATE/DELETE, but doesn't want to do the commit for me when I run this final stored proc.
After having tried multiple approaches, I see the transaction in the db sitting uncommitted. I know sqlalchemy is very flexible and I am using about 3% of its functionality, what is the simplest way to get this working? I think I need to be using sqlalchemy core and not ORM. I saw examples using sessionmaker, but I think that monopolizes the engine object and doesn't let pandas access it.
connection = engine.connect()
transaction = connection.begin()
connection.execute("EXEC sp_doLoaderStuff")
transaction.commit()
connection.close()
I have tried calling .execute from the connection level, from a cursor level, and even using the .raw_connection() method without success.
connection = engine.raw_connection()
connection.autocommit = True
cursor = connection.cursor()
cursor.execute("EXEC sp_doLoaderStuff")
connection.commit()
connection.close()
Any ideas what I am missing here?

Completely self-inflicted. The correct working code using the raw_connection() method that is working fine is:
connection = engine.raw_connection()
cursor = connection.cursor()
cursor.execute("EXEC sp_doLoaderStuff")
connection.commit()
connection.close()

Accessing test PostgreSQL database from python

I can't seem to correctly connect and pull from a test postgreSQL database in python. I installed PostgreSQL using Homebrew. Here's how I have been accessing the database table and value from the terminal:
xxx-macbook:~ xxx$ psql
psql (9.4.0)
Type "help" for help.
xxx=# \dn
List of schemas
Name | Owner
--------+---------
public | xxx
(1 row)
xxx=# \connect postgres
You are now connected to database "postgres" as user "xxx".
postgres=# SELECT * from test.test;
coltest
-----------
It works!
(1 row)
But when trying to access it from python, using the code below, it doesn't work. Any suggestions?
########################################################################################
# Importing variables from PostgreSQL database via SQL commands
db_conn = psycopg2.connect(database='postgres',
user='xxx')
cursor = db_conn.cursor()
#querying the database
result = cursor.execute("""
Select * From test.test
""")
print "Result: ", result
>>> Result: None
It should say: Result: It works!

You need to fetch the results.
From the docs:
The [execute()-]method returns None. If a query was executed, the returned values can be retrieved using fetch*() methods.
Example:
result = cursor.fetchall()
For reference:
http://initd.org/psycopg/docs/cursor.html#execute
http://initd.org/psycopg/docs/cursor.html#fetch
Note that (unlike psql) psycopg2 wraps anything in transactions. So if you intend to issue persistent changes to the database (INSERT, UPDATE, DELETE, ...) you need to commit them explicitly. Otherwise changes will be rolled back automatically when the connection object is destroyed. Read more on that topic here:
http://initd.org/psycopg/docs/usage.html
http://initd.org/psycopg/docs/usage.html#transactions-control

Why isn't the 'insert' function adding rows using MySQLdb?

I'm trying to figure out how to use the MySQLdb library in Python (I am novice at best for both of them).
I'm following the code here, specifically:
cursor = conn.cursor ()
cursor.execute ("DROP TABLE IF EXISTS animal")
cursor.execute ("""
CREATE TABLE animal
(
name CHAR(40),
category CHAR(40)
)
""")
cursor.execute ("""
INSERT INTO animal (name, category)
VALUES
('snake', 'reptile'),
('frog', 'amphibian'),
('tuna', 'fish'),
('racoon', 'mammal')
""")
print "Number of rows inserted: %d" % cursor.rowcount
cursor.close ()
conn.close ()
I can change this code to create or drop tables, but I can't get it to actually commit the INSERT. It returns the row.count value as expected (even when I change the value in the table, it changes to what I expect it to be).
Every time I look into the database with PHPMyAdmin there are no inserts made. How do I commit the INSERT to the database?

You forget commit data changes, autocommit is disabled by default:
cursor.close ()
conn.commit ()
conn.close ()
Quoting Writing MySQL Scripts with Python DB-API documentation:
"The connection object commit() method commits any outstanding changes
in the current transaction to make them permanent in the database. In
DB-API, connections begin with autocommit mode disabled, so you must
call commit() before disconnecting or changes may be lost."

How do you change the SQL isolation level from Python using MySQLdb?

The documentation I've run across researching this indicates that the way to do it for other databases is to use multiple statements in your query, a la:
>>> cursor = connection.cursor()
>>> cursor.execute("set session transaction isolation level read uncommitted;
select stuff from table;
set session transaction isolation level repeatable read;")
Unfortunately, doing that yields no results, as apparently the Python DB API (or maybe just this implementation of it?) doesn't support multiple recordsets within a single query.
Has anyone else had success with this in the past?

I don't think this works for the MySQLdb driver; you'll have to issue separate queries:
cur = conn.cursor()
cur.execute("SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED")
cur.execute("SELECT ##session.tx_isolation")
print cur.fetchall()[0]
cur.execute("SELECT * FROM bar")
print cur.fetchall()
cur.execute("SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ")
cur.execute("SELECT ##session.tx_isolation")
print cur.fetchall()[0]
# output
('READ-UNCOMMITTED',)
(('foo',), ('bar',))
('REPEATABLE-READ',)
The MySQLdb cursor's execute() method only sees the first query up to the semicolon:
cur.execute("SELECT * FROM bar WHERE thing = 'bar'; SELECT * FROM bar")
print cur.fetchall()
# output
(('bar',),)

cur.executemany("SELECT * FROM bar WHERE thing = 'bar'; SELECT * FROM bar")
print cur.fetchall()
use cur.executemany to run multiple sql statements with ; separated.