Develop Reference

Flask-SQLAlchemy how to manage one session/transaction in multiple requests

From what I have already learned, the Database sessions isolated in the request threads - I mean, when user send request to the Flask server, then the route function create session, manipulate the database and then commit/rollback database (which ends session). I would like to know how to manipulate one session with multiple requests. Something like:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
#app.route("/create_session")
def create_session():
session = db.session # create session
...
return response
#app.route("/query")
def query():
... # perform some operations with the session
return response
#app.route("/end_session")
def end_session():
session.commit() # commit session
...
return response
where:
First I am calling the request to the '/create_session' route to create session
Then I am calling the request to the '/query' route to do some operations with session
On the end I am calling the request to the '/end_session' route to end session
All routes manage only one, shared transaction.
Thanks for any response!

Flask-sqlalchemy: When to close sessions?

I have created a python webapp with Flask and it seems like I am having connection issues with the database. I think this is because I don't close my sessions somewhere in my code.
I have
db = SQLAlchemy(app)
for the database and use
#views.route('/test/', methods=['GET', 'POST'])
def test():
db.session.add(something)
db.session.commit()
#views.route('/another_page/', methods=['GET', 'POST'])
def page():
some_records = User.query.get(some_ids)
for adding records to the database.
When do I have to close my session in this case? Is there a way to close the connection after the user leaves? Should I close every time a page is done with the database? Do I need to close my connection after a query?

The documentation says next:
As in the declarative approach, you need to close the session after each request or application context shutdown. Put this into your application module:
from yourapplication.database import db_session
#app.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
UPD: In case of Flask-SQLAlchemy this staff is hardcoded, thus you don't need to care about it when develop.

Run flask tests within a transaction

I have a Flask application setup with Flask-SQLAlchemy, and I'm running the tests with factory-boy. I'm trying to wrap the tests within transactions, so as not to fill up the db with test data (and avoid having to drop/recreate the db between tests, since it's pretty expensive).
However, since the db session is removed after each request, the objects created for the test are lost unless I commit the session before making the request.
Is there a way of sharing the session between the test context and the request context?
Here's a test example:
class TestUserViews(unittest.TestCase):
def setUp(self):
self.client = app.test_client()
def test_user_detail(self):
with app.test_request_context():
# Make sure requesting an unknown user returns a 404
response = self.client.get('/users/123/')
assert response.status_code == 404
# Create a user
user = UserFactory()
db.session.commit()
response = self.client.get('/users/{}/'.format(user.id))
assert response.status_code == 200 # This works since the write was committed
def test_user_uncommitted(self):
with app.test_request_context():
# Create a user
uncommitted_user = UserFactory()
assert uncommitted_user in db.session
response = self.client.get('/users/{}/'.format(uncommitted_user.id))
assert response.status_code == 200 # This doesn't work, the session wasn't reused
I built a dummy project on github to show a more complete example if necessary. Any idea what I'm missing here? Thanks!

Flask-SQLAlchemy: Can't reconnect until invalid transaction is rolled back

So I am using Amazon Web Services RDS to run a MySQL server and using Python's Flask framework to run the application server and Flask-SQLAlchemy to interface with the RDS.
My app config.py
SQLALCHEMY_DATABASE_URI = '<RDS Host>'
SQLALCHEMY_POOL_RECYCLE = 60
My __ init __.py
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
application = Flask(__name__)
application.config.from_object('config')
db = SQLAlchemy(application)
I have my main application.py
from flask import Flask
from application import db
import flask.ext.restless
from application.models import Person
application = Flask(__name__)
application.debug=True
db.init_app(application)
#application.route('/')
def index():
return "Hello, World!"
manager = flask.ext.restless.APIManager(application, flask_sqlalchemy_db=db)
manager.create_api(Person, methods=['GET','POST', 'DELETE'])
if __name__ == '__main__':
application.run(host='0.0.0.0')
The models.py
class Person(db.Model):
__bind_key__= 'people'
id = db.Column(db.Integer, primary_key=True)
firstName = db.Column(db.String(80))
lastName = db.Column(db.String(80))
email = db.Column(db.String(80))
def __init__(self, firstName=None, lastName=None, email=None):
self.firstName = firstName
self.lastName = lastName
self.email = email
I then have a script to populate the database for testing purposes after db creation and app start:
from application import db
from application.models import Person
person = Person('Bob', 'Jones', 'bob#website.net')
db.session.add(person)
db.session.commit()
Once I've reset the database with db.drop_all() and db.create_all() I start the application.py and then the script to populate the database.
The server will respond with correct JSON but if I come back and check it hours later, I get the error that I need to rollback or sometimes the 2006 error that the MySQL server has gone away.
People suggested that I change timeout settings on the MySQL server but that hasn't fixed anything. Here are my settings:
innodb_lock_wait_timeout = 3000
max_allowed_packet = 65536
net_write_timeout = 300
wait_timeout = 300
Then when I look at the RDS monitor, it shows the MySQL server kept the connection open for quite a while until the timeout. Now correct me if I'm wrong but isn't the connection supposed to be closed after it's finished? It seems that the application server keeps making sure that the database connection exists and then when the MySQL server times out, Flask/Flask-SQLAlchemy throws an error and brings down the app server with it.
Any suggestions are appreciated, thanks!

I think what did it was adding
db.init_app(application)
in application.py, haven't had the error since.

Everytime checking rollback or not is troublesome..
I made insert, update functions which need commit.
#app.teardown_request
def session_clear(exception=None):
Session.remove()
if exception and Session.is_active:
Session.rollback()

It seems not to be a problem with the transactions at the first place, but this is probably caused by an MySQL Error like Connection reset by peer beforehand. That means your connection is lost, probably because your application context was not setup correctly.
In general it is preferrable to use the factory pattern to create your app. This has a lot of advantages, your code is
easier to read and setup
easier to test
avoid circular imports
To prevent the invalid transaction error (that is probably caused by an OperationalError: Connection reset by peer) you should ensure that you are handling the database connection right.
The following example is based on this article which gives a nice explanation of the flask application context and how to use it with database connections or any other extensions.
application.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
def create_app():
"""Construct the core application."""
application = Flask(__name__)
application.config.from_object('config') # Set globals
db = SQLAlchemy()
with application.app_context():
# Initialize globals/extensions in app context
db.init_app(app)
# import routes here
from . import routes
return application
if __name__ == "__main__":
app = create_app()
app.run(host="0.0.0.0")
routes.py
from flask import current_app as application
#application.route('/', methods=['GET'])
def index():
return "Hello, World!"
If you still run into disconnect-problems you should also check the SQLAlchemy documentation on dealing with disconnects and have a look at this question.

Here you missing pool recycle as MySql closes session after some time so you need to add pool recycle so that connections in pool get reconnect after pool recycle time.
app.config['SQLALCHEMY_POOL_RECYCLE'] = 3600

This error usually appears when you create sqlalchemy the engine as a singleton. In that case after the connection is invalidated (in my case it was 3600sec) you get the InvalidTransaction error.
Best advice would be to initialise the db session at the time of application initialisation
db.init_app(app)
and import this db session when ever you have to do some CRUD operation.
Never faced this issue post this change on my application.

Alternatively, use this at the end of the script that populates your database:
db.session.close()
That should prevent those annoying "MySQL server has gone away" errors.

Testing Flask login and authentication?

I'm developing a Flask application and using Flask-security for user authentication (which in turn uses Flask-login underneath).
I have a route which requires authentication, /user. I'm trying to write a unit test which tests that, for an authenticated user, this returns the appropriate response.
In my unittest I'm creating a user and logging as that user like so:
from unittest import TestCase
from app import app, db
from models import User
from flask_security.utils import login_user
class UserTest(TestCase):
def setUp(self):
self.app = app
self.client = self.app.test_client()
self._ctx = self.app.test_request_context()
self._ctx.push()
db.create_all()
def tearDown(self):
if self._ctx is not None:
self._ctx.pop()
db.session.remove()
db.drop_all()
def test_user_authentication():
# (the test case is within a test request context)
user = User(active=True)
db.session.add(user)
db.session.commit()
login_user(user)
# current_user here is the user
print(current_user)
# current_user within this request is an anonymous user
r = test_client.get('/user')
Within the test current_user returns the correct user. However, the requested view always returns an AnonymousUser as the current_user.
The /user route is defined as:
class CurrentUser(Resource):
def get(self):
return current_user # returns an AnonymousUser
I'm fairly certain I'm just not fully understanding how testing Flask request contexts work. I've read this Flask Request Context documentation a bunch but am still not understanding how to approach this particular unit test.

The problem is different request contexts.
In your normal Flask application, each request creates a new context which will be reused through the whole chain until creating the final response and sending it back to the browser.
When you create and run Flask tests and execute a request (e.g. self.client.post(...)) the context is discarded after receiving the response. Therefore, the current_user is always an AnonymousUser.
To fix this, we have to tell Flask to reuse the same context for the whole test. You can do that by simply wrapping your code with:
with self.client:
You can read more about this topic in the following wonderful article:
https://realpython.com/blog/python/python-web-applications-with-flask-part-iii/
Example
Before:
def test_that_something_works():
response = self.client.post('login', { username: 'James', password: '007' })
# this will fail, because current_user is an AnonymousUser
assertEquals(current_user.username, 'James')
After:
def test_that_something_works():
with self.client:
response = self.client.post('login', { username: 'James', password: '007' })
# success
assertEquals(current_user.username, 'James')

The problem is that the test_client.get() call causes a new request context to be pushed, so the one you pushed in your the setUp() method of your test case is not the one that the /user handler sees.
I think the approach shown in the Logging In and Out and Test Adding Messages sections of the documentation is the best approach for testing logins. The idea is to send the login request through the application, like a regular client would. This will take care of registering the logged in user in the user session of the test client.

I didn't much like the other solution shown, mainly because you have to keep your password in a unit test file (and I'm using Flask-LDAP-Login, so it's nonobvious to add a dummy user, etc.), so I hacked around it:
In the place where I set up my test app, I added:
#app.route('/auto_login')
def auto_login():
user = ( models.User
.query
.filter_by(username="Test User")
.first() )
login_user(user, remember=True)
return "ok"
However, I am making quite a lot of changes to the test instance of the flask app, like using a different DB, where I construct it, so adding a route doesn't make the code noticeably messier. Obv this route doesn't exist in the real app.
Then I do:
def login(self):
response = self.app.test_client.get("/auto_login")
Anything done after that with test_client should be logged in.

From the docs: https://flask-login.readthedocs.io/en/latest/
It can be convenient to globally turn off authentication when unit testing. To enable this, if the application configuration variable LOGIN_DISABLED is set to True, this decorator will be ignored.