Develop Reference

Python SQLalchemy - can I pass the connection object between functions?

I have a python application that is reading from mysql/mariadb, uses that to fetch data from an api and then inserts results into another table.
I had setup a module with a function to connect to the database and return the connection object that is passed to other functions/modules. However, I believe this might not be a correct approach. The idea was to have a small module that I could just call whenever I needed to connect to the db.
Also note, that I am using the same connection object during loops (and within the loop passing to the db_update module) and call close() when all is done.
I am also getting some warnings from the db sometimes, those mostly happen at the point where I call db_conn.close(), so I guess I am not handling the connection or session/engine correctly. Also, the connection id's in the log warning keep increasing, so that is another hint, that I am doing it wrong.
[Warning] Aborted connection 351 to db: 'some_db' user: 'some_user' host: '172.28.0.3' (Got an error reading communication packets)
Here is some pseudo code that represents the structure I currently have:
################
## db_connect.py
################
# imports ...
from sqlalchemy import create_engine
def db_connect():
# get env ...
db_string = f"mysql+pymysql://{db_user}:{db_pass}#{db_host}:{db_port}/{db_name}"
try:
engine = create_engine(db_string)
except Exception as e:
return None
db_conn = engine.connect()
return db_conn
################
## db_update.py
################
# imports ...
def db_insert(db_conn, api_result):
# ...
ins_qry = "INSERT INTO target_table (attr_a, attr_b) VALUES (:a, :b);"
ins_qry = text(ins_qry)
ins_qry = ins_qry.bindparams(a = value_a, b = value_b)
try:
db_conn.execute(ins_qry)
except Exception as e:
print(e)
return None
return True
################
## main.py
################
from sqlalchemy import text
from db_connect import db_connect
from db_update import db_insert
def run():
try:
db_conn = db_connect()
if not db_conn:
return False
except Exception as e:
print(e)
qry = "SELECT *
FROM some_table
WHERE some_attr IN (:some_value);"
qry = text(qry)
search_run_qry = qry.bindparams(
some_value = 'abc'
)
result_list = db_conn.execute(qry).fetchall()
for result_item in result_list:
## do stuff like fetching data from api for every record in the query result
api_result = get_api_data(...)
## insert into db:
db_ins_status = db_insert(db_conn, api_result)
## ...
db_conn.close
run()
EDIT: Another question:
a) Is it ok in a loop, that does an update on every iteration to use the same connection, or would it be wiser to instead pass the engine to the run() function and call db_conn = engine.connect() and db_conn.close() just before and after each update?
b) I am thinking about using ThreadPoolExecutor instead of the loop for the API calls. Would this have implications on how to use the connection, i.e. can I use the same connection for multiple threads that are doing updates to the same table?
Note: I am not using the ORM feature mostly because I have a strong DWH/SQL background (though not so much as DBA) and I am used to writing even complex sql queries. I am thinking about switching to just using PyMySQL connector for that reason.
Thanks in advance!

Yes you can return/pass connection object as parameter but what is the aim of db_connect method, except testing connection ? As I see there is no aim of this db_connect method therefore I would recommend you to do this as I done it before.
I would like to share a code snippet from one of my project.
def create_record(sql_query: str, data: tuple):
try:
connection = mysql_obj.connect()
db_cursor = connection.cursor()
db_cursor.execute(sql_query, data)
connection.commit()
return db_cursor, connection
except Exception as error:
print(f'Connection failed error message: {error}')
and then using this one as for another my need
db_cursor, connection, query_data = fetch_data(sql_query, query_data)
and after all my needs close the connection with this method and method call.
def close_connection(connection, db_cursor):
"""
This method used to close SQL server connection
"""
db_cursor.close()
connection.close()
and calling method
close_connection(connection, db_cursor)
I am not sure can I share my github my check this link please. Under model.py you can see database methods and to see how calling them check it main.py
Best,
Hasan.

Python Unit Test : How to unit test the module which contains database operations?

I am using pymysql client library to connect to the real database. I have a function in module, where I connect to the database using pymysql and do only database insert operations.How to unit test this function in python without hitting the real database?
import pymysql
def connectDB(self):
# Connect to the database
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db')
try:
with connection.cursor() as cursor:
# Create a new record
sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
cursor.execute(sql, ('newuser#some.com', 'newpassword'))
connection.commit()
My python version is 2.7.

You can use patch, like this:
from unittest.mock import patch, MagicMock
#patch('mypackage.mymodule.pymysql')
def test(self, mock_sql):
self.assertIs(mypackage.mymodule.pymysql, mock_sql)
conn = Mock()
mock_sql.connect.return_value = conn
cursor = MagicMock()
mock_result = MagicMock()
cursor.__enter__.return_value = mock_result
cursor.__exit___ = MagicMock()
conn.cursor.return_value = cursor
connectDB()
mock_sql.connect.assert_called_with(host='localhost',
user='user',
password='passwd',
db='db')
mock_result.execute.assert_called_with("sql request", ("user", "pass"))

You need a series of fake databases, called stubs, which return hardcoded values. During the test these stubs are used instead of the real database. I am not familiar with Python, but one way to do this in C++ is to make your object to receive the database as a constructor parameter. In production code you use a real database parameter, in the test the stub. This can be done because the constructor expects a pointer to a common base class. Even it is not written for Python I suggest to read the first chapters from Roy Osherove: The art of unit testing. The book clearly explains why these fake databases are stubs and not mocks.

You've just rediscovered one of the most compelling reasons why testing is important: it tells you when your design is bad.
To put it slightly differently, testability is a good first-order proxy for quality. Consider the following:
class DB(object):
def __init__(self, **credentials):
self._connect = partial(pymysql.connect, **credentials)
def query(self, q_str, params):
with self._connect as conn:
with conn.cursor() as cur:
cur.execute(q_str, params)
return cur.fetchall()
# now for usage
test_credentials = {
# use credentials to a fake database
}
test_db = DB(**test_credentials)
test_db.query(write_query, list_of_fake_params)
results = test_db.query(read_query)
assert results = what_the_results_should_be
If you work with multiple databases you could use polymorphism or depending on API similarity make the specific DB be a constructor parameter to your object.

Mock a MySQL database in Python

I use Python 3.4 from the Anaconda distribution. Within this distribution, I found the pymysql library to connect to an existing MySQL database, which is located on another computer.
import pymysql
config = {
'user': 'my_user',
'passwd': 'my_passwd',
'host': 'my_host',
'port': my_port
}
try:
cnx = pymysql.connect(**config)
except pymysql.err.OperationalError :
sys.exit("Invalid Input: Wrong username/database or password")
I now want to write test code for my application, in which I want to create a very small database at the setUp of every test case, preferably in memory. However, when I try this out of the blue with pymysql, it cannot make a connection.
def setUp(self):
config = {
'user': 'test_user',
'passwd': 'test_passwd',
'host': 'localhost'
}
cnx = pymysql.connect(**config)
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I have been googling around, and found some things about SQLite and MySQLdb. I have the following questions:
Is sqlite3 or MySQLdb suitable for creating quickly a database in memory?
How do I install MySQLdb within the Anaconda package?
Is there an example of a test database, created in the setUp? Is this even a good idea?
I do not have a MySQL server running locally on my computer.

You can mock a mysql db using testing.mysqld (pip install testing.mysqld)
Due to some noisy error logs that crop up, I like this setup when testing:
import testing.mysqld
from sqlalchemy import create_engine
# prevent generating brand new db every time. Speeds up tests.
MYSQLD_FACTORY = testing.mysqld.MysqldFactory(cache_initialized_db=True, port=7531)
def tearDownModule():
"""Tear down databases after test script has run.
https://docs.python.org/3/library/unittest.html#setupclass-and-teardownclass
"""
MYSQLD_FACTORY.clear_cache()
class TestWhatever(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.mysql = MYSQLD_FACTORY()
cls.db_conn = create_engine(cls.mysql.url()).connect()
def setUp(self):
self.mysql.start()
self.db_conn.execute("""CREATE TABLE `foo` (blah)""")
def tearDown(self):
self.db_conn.execute("DROP TABLE foo")
#classmethod
def tearDownClass(cls):
cls.mysql.stop() # from source code we can see this kills the pid
def test_something(self):
# something useful

Both pymysql, MySQLdb, and sqlite will want a real database to connect too.
If you want just to test your code, you should just mock the pymysql module on the module you want to test, and use it accordingly
(in your test code: you can setup the mock object to return hardcoded results to predefined SQL statements)
Check the documentation on native Python mocking library at:
https://docs.python.org/3/library/unittest.mock.html
Or, for Python 2:
https://pypi.python.org/pypi/mock

SqlAlchemy equivalent of pyodbc connect string using FreeTDS

The following works:
import pyodbc
pyodbc.connect('DRIVER={FreeTDS};Server=my.db.server;Database=mydb;UID=myuser;PWD=mypwd;TDS_Version=8.0;Port=1433;')
The following fails:
import sqlalchemy
sqlalchemy.create_engine("mssql://myuser:mypwd#my.db.server:1433/mydb?driver=FreeTDS& odbc_options='TDS_Version=8.0'").connect()
The error message for above is:
DBAPIError: (Error) ('08001', '[08001] [unixODBC][FreeTDS][SQL Server]Unable to connect to data source (0) (SQLDriverConnectW)') None None
Can someone please point me in the right direction? Is there a way I can simply tell sqlalchemy to pass a specific connect string through to pyodbc?
Please Note: I want to keep this DSN-less.

The example by #Singletoned would not work for me with SQLAlchemy 0.7.2. From the SQLAlchemy docs for connecting to SQL Server:
If you require a connection string that is outside the options presented above, use the odbc_connect keyword to pass in a urlencoded connection string. What gets passed in will be urldecoded and passed directly.
So to make it work I used:
import urllib
quoted = urllib.quote_plus('DRIVER={FreeTDS};Server=my.db.server;Database=mydb;UID=myuser;PWD=mypwd;TDS_Version=8.0;Port=1433;')
sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(quoted))
This should apply to Sybase as well.
NOTE: In python 3 the urllib module has been split into parts and renamed. Thus, this line in python 2.7:
quoted = urllib.quote_plus
has to be changed to this line in python3:
quoted = urllib.parse.quote_plus

I'm still interested in a way to do this in one line within the sqlalchemy create_engine statement, but I found the following workaround detailed here:
import pyodbc, sqlalchemy
def connect():
pyodbc.connect('DRIVER={FreeTDS};Server=my.db.server;Database=mydb;UID=myuser;PWD=mypwd;TDS_Version=8.0;Port=1433;')
sqlalchemy.create_engine('mssql://', creator=connect)
UPDATE: Addresses a concern I raised in my own comment about not being able to pass arguments to the connect string. The following is a general solution if you need to dynamically connect to different databases at runtime. I only pass the database name as a parameter, but additional parameters could easily be used as needed:
import pyodbc
import os
class Creator:
def __init__(self, db_name='MyDB'):
"""Initialization procedure to receive the database name"""
self.db_name = db_name
def __call__(self):
"""Defines a custom creator to be passed to sqlalchemy.create_engine
http://stackoverflow.com/questions/111234/what-is-a-callable-in-python#111255"""
if os.name == 'posix':
return pyodbc.connect('DRIVER={FreeTDS};'
'Server=my.db.server;'
'Database=%s;'
'UID=myuser;'
'PWD=mypassword;'
'TDS_Version=8.0;'
'Port=1433;' % self.db_name)
elif os.name == 'nt':
# use development environment
return pyodbc.connect('DRIVER={SQL Server};'
'Server=127.0.0.1;'
'Database=%s_Dev;'
'UID=user;'
'PWD=;'
'Trusted_Connection=Yes;'
'Port=1433;' % self.db_name)
def en(db_name):
"""Returns a sql_alchemy engine"""
return sqlalchemy.create_engine('mssql://', creator=Creator(db_name))

This works:
import sqlalchemy
sqlalchemy.create_engine("DRIVER={FreeTDS};Server=my.db.server;Database=mydb;UID=myuser;PWD=mypwd;TDS_Version=8.0;Port=1433;").connect()
In that format, SQLAlchemy just ignores the connection string and passes it straight on to pyodbc.
Update:
Sorry, I forgot that the uri has to be url-encoded, therefore, the following works:
import sqlalchemy
sqlalchemy.create_engine("DRIVER%3D%7BFreeTDS%7D%3BServer%3Dmy.db.server%3BDatabase%3Dmydb%3BUID%3Dmyuser%3BPWD%3Dmypwd%3BTDS_Version%3D8.0%3BPort%3D1433%3B").connect()

Internally "my.db.server:1433" is passed as part of a connection string like SERVER=my.db.server:1433;.
Unfortunately unixODBC/FreeTDS won't accept a port in the SERVER bit. Instead it wants SERVER=my.db.server;PORT=1433;
To use the sqlalchemy syntax for a connection string, you must specify the port as a parameter.
sqlalchemy.create_engine("mssql://myuser:mypwd#my.db.server:1433/mydb?driver=FreeTDS& odbc_options='TDS_Version=8.0'").connect()
becomes:
sqlalchemy.create_engine("mssql://myuser:mypwd#my.db.server/mydb?driver=FreeTDS&port=1433& odbc_options='TDS_Version=8.0'").connect()

To pass various parameters to your connect function, it sounds like format string might do what you want:
def connect(server, dbname, user, pass):
pyodbc.connect('DRIVER={FreeTDS};Server=%s;Database=%s;UID=%s;PWD=%s;TDS_Version=8.0;Port=1433;' % (server, dbname, user, pass))
And you would then call it with something like:
connect('myserver', 'mydatabase', 'myuser', 'mypass')
More info on format strings is here: http://docs.python.org/library/string.html#formatstrings

Pylons: Sharing SQLAlchemy MySQL connection with external library

I am running Pylons using SQLAlchemy to connect to MySQL, so when I want to use a database connection in a controller, I can do this:
from myapp.model.meta import Session
class SomeController(BaseController):
def index(self):
conn = Session.connection()
rows = conn.execute('SELECT whatever')
...
Say my controller needs to call up an external library, that also needs a database connection, and I want to provide the connection for it from the SQLAlchemy MySQL connection that is already established:
from myapp.model.meta import Session
import mymodule
class SomeController(BaseController):
def index(self):
conn = Session.connection()
myobject = mymodule.someobject(DATABASE_OBJECT)
...
conn.close()
What should DATABSE_OBJECT be? Possibilities:
Pass Session -- and then open and close Session.connection() in the module code
Pass conn, and then call conn.close() in the controller
Just pass the connection parameters, and have the module code set up its own connection
There is another wrinkle, which is that I need to instantiate some objects in app_globals.py, and these objects need a database connection as well. It seems that app_globals.py cannot use Session's SQLAlchemy connection yet -- it's not bound yet.
Is my architecture fundamentally unsounds? Should I not be trying to share connections between Pylons and external libraries this way? Thanks!

You should not manage connections yourself - it's all done by SQLAlchemy. Just use scoped session object everywhere, and you will be fine.
def init_model(engine):
sm = orm.sessionmaker(autoflush=False, autocommit=False, expire_on_commit=False, bind=engine)
meta.engine = engine
meta.Session = orm.scoped_session(sm)
def index(self):
rows = Session.execute('SELECT ...')
You can pass Session object to your external library and do queries there as you wish. There is no need to call .close() on it.
Regarding app_globals, I solved that by adding other method in globals class which is called after db initialization from environment.py
class Globals(...):
def init_model(self, config):
self.some_persistent_db_object = Session.execute('...')
def load_environment(...):
...
config['pylons.app_globals'].init_model(config)
return config

What should DATABSE_OBJECT be? Possibilities:
4. pass a "proxy" or "helper" object with higher level of abstraction interface
Unless the external library really needs direct access to SQLAlchemy session, you could provide it with object that has methods like "get_account(account_no)" instead of "execute(sql)". Doing so would keep SQLAlchemy-specific code more isolated, and the code would be also easier to test.
Sorry that this is not so much an answer to your original question, more a design suggestion.