I have a Flask REST API, running with a gunicorn/nginx stack. There is global SQLAlchemy session set up once for each thread that the API runs on. I set up an endpoint /test/ for running the unit tests for the API. One test makes a POST request to add something to the database, then has a finally: clause to clean up:
def test_something():
try:
url = "http://myposturl"
data = {"content" : "test post"}
headers = {'content-type': 'application/json'}
result = requests.post(url, json=data, headers=headers).json()
validate(result, myschema)
finally:
db.sqlsession.query(MyTable).filter(MyTable.content == "test post").delete()
db.sqlsession.commit()
The problem is that the thread to which the POST request is made now has a "test post" object in its session, but the database has no such object because the thread on which the tests ran deleted that thing from the database. So when I make a GET request to the server, about 1 in 4 times (I have 4 gunicorn workers), I get the "test post" object, and 3 in 4 times I do not. This is because the threads each have their own session object, and they are getting out of sync, but I don't really know what to do about it....
Here is my setup for my SQLAlchemy session:
def connectSQLAlchemy():
import sqlalchemy
import sqlalchemy.orm
engine = sqlalchemy.create_engine(connection_string(DBConfig.USER, DBConfig.PASSWORD, DBConfig.HOST, DBConfig.DB))
session_factory = sqlalchemy.orm.sessionmaker(bind=engine)
Session = sqlalchemy.orm.scoped_session(session_factory)
return Session()
# Create a global session for everyone
sqlsession = connectSQLAlchemy()
Please use flask-sqlalchemy if you're using flask, it takes care of the lifecycle of the session for you.
If you insist on doing it yourself, the correct pattern is to create a session for each request instead of having a global session. You should be doing
Session = scoped_session(session_factory, scopefunc=flask._app_ctx_stack.__ident_func__)
return Session
instead of
Session = scoped_session(session_factory)
return Session()
And do
session = Session()
every time you need a session. By virtue of the scoped_session and the scopefunc, this will return you a different session in each request, but the same session in the same request.
Figured it out. What I did was to add a setup and teardown to the request in my app's __init__.py:
#app.before_request
def startup_session():
db.session = db.connectSQLAlchemy()
#app.teardown_request
def shutdown_session(exception=None):
db.session.close()
still using the global session object in my db module:
db.py:
....
session = None
....
The scoped_session handles the different threads, I think...
Please advise if this is a terrible way to do this for some reason. =c)
Related
Per the docs, we should use the following pattern with a sessionmaker object:
Session = sessionmaker(engine)
with Session.begin() as session:
session.add(some_object)
In a multithreaded environment, we are also supposed to use a single scoped_session and share it. So in my __init__.py I create create one and import it everywhere else in my program:
engine = create_engine(config.SQLALCHEMY_DATABASE_URI)
Session = scoped_session(sessionmaker(bind=engine))
The question is, how am I supposed to combine these two approaches? This seems to be the suggested way, but it errors out:
from myapp import Session
with Session.begin() as session:
query_result = session.query(MyModel).all()
----
Exception has occurred: AttributeError
'SessionTransaction' object has no attribute 'query'
I tried the following and it works, but it seems like it doesn't follow the docs, and I'm afraid it breaks something not obvious. Can anyone confirm if this is correct?
from myapp import Session
with Session() as session, session.begin():
query_result = session.query(MyModel).all()
I've been looking around at other replies and seeing very little that addresses the specific question.
From the Session.begin() docs:
The Session object features autobegin behavior, so that normally it is not necessary to call the Session.begin() method explicitly. However, it may be used in order to control the scope of when the transactional state is begun.
You can use Session.begin() (new in 1.4) to obtain a SessionTransaction instance usable as a context manager which will autocommit on successful exit.
Calling the scoped_session returns a SessionTransaction right away, as per your error, so you do not need to begin it again.
All in all, you can definitely do the stacked context manager, but its unnecessary, so you might as well stick to using the original flow:
Session = scoped_session(...)
with Session() as session: # NB. session is a thread local SessionTransaction
...
session.commit()
or the proxied Session
Session = scoped_session(...)
#on_request_end
def remove_session(req):
Session.remove()
#route("/xyz", ...)
def handle_xyz():
instance = Class(...)
Session.add(instance)
Session.commit()
I know it is possible to create session object using session_transaction() method. However, is there a way to access the current session object which gets created when for example "/" route gets hit? I did from flask import session to access the session but it's empty. Let me know if it is possible. Thanks.
This is what you're looking for. As it says however, you'd have to use the instantiation you create in your with statement.
with app.test_client() as c:
with c.session_transaction() as sess:
sess['a_key'] = 'a value'
# once this is reached the session was stored
result = app.test_client.get('/a_url')
# NOT part of the 2nd context
Note that this won't work if you run your test within the scope of the with c.session_transaction() as sess statement, it needs to be run after that block.
If you want to read the session data written in your view from the test, one way is to mock the session view as a dict and verify the session in your test. Here's an example using Python's unittest.mock:
app.py
from flask import Flask, session, request
app = Flask(__name__)
app.config["SECRET_KEY"] = "my secret key"
#app.route("/", methods=["POST"])
def index():
session["username"] = request.form["username"]
return "Username saved in session"
test_index.py
from unittest.mock import patch
from app import app
def test_index():
with patch("app.session", dict()) as session:
client = app.test_client()
response = client.post("/", data={
"username": "test"
})
assert session.get("username") == "test"
assert response.data == b"Username saved in session"
You can use any mocking solution you prefer, of course.
What is the easiest way to have a server-side session variable in Flask?
Variable value:
A simple string
Not visible to the client (browser)
Not persisted to a DB -- simply vanishes when the session is gone
There is a built-in Flask session, but it sends the session data to the client:
session["secret"] = "I can see you"
The data is Base64 encoded and sent in a cryptographically signed cookie, but it is still trivial to read on the client.
In many frameworks, creating a server-side session variable is a one-liner, such as:
session.secret = "You can't see this"
The Flask solutions I have found so far are pretty cumbersome and geared towards handling large chunks of data. Is there a simple lightweight solution?
I think the Flask-Session extension is what you are looking for.
Flask-Session is an extension for Flask that adds support for Server-side Session to your application.
From the linked website:
from flask import Flask, session
from flask_session import Session # new style
# from flask.ext.session import Session # old style
app = Flask(__name__)
# Check Configuration section for more details
SESSION_TYPE = 'redis'
app.config.from_object(__name__)
Session(app)
#app.route('/set/')
def set():
session['key'] = 'value'
return 'ok'
#app.route('/get/')
def get():
return session.get('key', 'not set')
This answer is from June 2020 for flask-session 0.3.2.
The documentation is here.
There are several available SESSION_TYPESs. filesystem is the most straightforward while you're testing. The expectation is you already have a Redis, database, etc. setup if you are going to use the other SESSION_TYPEs. Section on SESSION_TYPE and requirements
null: NullSessionInterface (default)
Redis: RedisSessionInterface
Memcached: MemcachedSessionInterface
filesystem: FileSystemSessionInterface
MongoDB: MongoDBSessionInterface
SQLAlchemy: SqlAlchemySessionInterface
Code example from the documentation. If you go to /set/ then the session['key'] is populated with the word 'value'. But if you go to /get/ first, then `session['key'] will not exist and it will return 'not set'.
from flask import Flask, session
from flask_session import Session
app = Flask(__name__)
app.config['SESSION_TYPE'] = 'filesystem'
#personal style preference compared to the first answer
Session(app)
#app.route('/set/')
def set():
session['key'] = 'value'
return 'ok'
#app.route('/get/')
def get():
return session.get('key', 'not set')
I've created a web application using Twisted and SQLAlchemy. Since SQLAlchemy doesn't work together very well with Twisted's callback-based design (Twisted + SQLAlchemy and the best way to do it), I use deferToThread() within the root resource in order to run every request within its own thread. While this does generally work, about 10% of the requests get "stuck". This means that when I click a link in the browser, the request is handled by Twisted and the code for the respective resource runs and generates HTML output. But for whatever reason, that output is never sent back to the browser. Instead, Twisted sends the HTTP headers (along with the correct Content-Length), but never sends the body. The connection just stays open indefinitely with the browser showing the spinner icon. No errors are generated by Twisted in the logfile.
Below is a minimal example. If you want to run it, save it with a .tac extension, then run twistd -noy example.tac. On my server, the issue seems to occur relatively infrequently in this particular piece of code. Use something like while true; do wget -O- 'http://server.example.com:8080' >/dev/null; done to test it.
from twisted.web.server import Site
from twisted.application import service, internet
from twisted.web.resource import Resource
from twisted.internet import threads
from twisted.web.server import NOT_DONE_YET
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine, Column, Integer, String
Base = declarative_base()
class User(Base):
'''A user account.'''
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
login = Column(String(64))
class WebInterface(Resource):
def __init__(self):
Resource.__init__(self)
db_url = "mysql://user:password#mysql-server.example.com/myapp?charset=utf8"
db_engine = create_engine(db_url, echo=False, pool_recycle=300) #discard connections after 300 seconds
self.DBSession = sessionmaker(bind=db_engine)
def on_request_done(self, _, request):
'''All actions that need to be done after a request has been successfully handled.'''
request.db_session.close()
print('Session closed') #does get printed, so session should get closed properly
def on_request_failed(self, err, call):
'''What happens if the request failed on a network level, for example because the user aborted the request'''
call.cancel()
def on_error(self, err, request):
'''What happens if an exception occurs during processing of the request'''
request.setResponseCode(500)
self.on_request_done(None, request)
request.finish()
return err
def getChild(self, name, request):
'''We dispatch all requests to ourselves in order to be able to do the processing in separate threads'''
return self
def render(self, request):
'''Dispatch the real work to a thread'''
d = threads.deferToThread(self.do_work, request)
d.addCallbacks(self.on_request_done, errback=self.on_error, callbackArgs=[request], errbackArgs=[request])
#If the client aborts the request, we need to cancel it to avoid error messages from twisted
request.notifyFinish().addErrback(self.on_request_failed, d)
return NOT_DONE_YET
def do_work(self, request):
'''This method runs in thread context.'''
db_session = self.DBSession()
request.db_session = db_session
user = db_session.query(User).first()
body = 'Hello, {} '.format(user.login) * 1024 #generate some output data
request.write(body)
request.finish()
application = service.Application("My Testapp")
s = internet.TCPServer(8080, Site(WebInterface()), interface='0.0.0.0')
s.setServiceParent(application)
Its possible you are not closing your database connection or some dead lock situation in the database using SQLAlchemy? I've had flask lock up on me before from not closing connections / not ending transactions.
I've solved the issue. #beiller, you were pretty close to it with your guess. As can be seen in the source code of my question, the DB session gets opened after request processing has started, but the two are closed in the same (instead of the reverse) order. Close the session before calling request.finish(), and everything's fine.
I'm using Flask-Testing which says:
Another gotcha is that Flask-SQLAlchemy also removes the session
instance at the end of every request (as should any threadsafe
application using SQLAlchemy with scoped_session). Therefore the
session is cleared along with any objects added to it every time you
call client.get() or another client method.
However, I'm not seeing that. This test fails:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.testing import TestCase
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
#app.route('/')
def index():
print 'before request:', `db.session`
u = db.session.query(User).first()
u.name = 'bob'
return ''
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
class SessionTest(TestCase):
def create_app(self):
return app
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
(Run with $ nosetests test_file.py to see it in action.)
stdout:
-------------------- >> begin captured stdout << ---------------------
before request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
after request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
bob
--------------------- >> end captured stdout << ----------------------
According to the docs, user u should not be in the session after a get request, but it is! Does anybody know why this is happening?
Furthermore, u.name is bob and not joe, even though the request never committed! (So I'm convinced it's the same session.)
For the record,
$ pip freeze | grep Flask
Flask==0.10.1
Flask-Bcrypt==0.5.2
Flask-DebugToolbar==0.8.0
Flask-Failsafe==0.1
Flask-SQLAlchemy==0.16
Flask-Script==0.6.2
Flask-Testing==0.4
Flask-Uploads==0.1.3
Flask-WTF==0.8
I'm pretty sure the confusion comes from the fact that sessions in SQLAlchemy are scoped, meaning that each request handler creates and destroys its own session.
This is necessary because web servers can be multi-threaded, so multiple requests might be served at the same time, each working with a different database session.
For this reason, the session that you used outside of the context of a request is likely not the same session that the view function that handles the '/' route gets and then destroys at the end.
UPDATE: I dug around a bit and figured this thing out.
Flask-SQLAlchemy installs a hook on app.teardown_appcontext, and here is where it calls db.session.remove().
The testing environment does not fully replicate the environment of a real request because it does not push/pop the application context. Because of that the session is never removed at the end of the request.
As a side note, keep in mind that functions registered with before_request and after_request are also not called when you call client.get().
You can force an application context push and pop with a small change to your test:
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
with app.app_context():
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
with this change the test passes.
The documentation for Flask-Testing seems to be wrong or more likely outdated. Maybe things worked like they describe at some point, but that isn't accurate for current Flask and Flask-SQLAlchemy versions.
I hope this helps!
FlaskClient works with request context while Flask-SQLAlchemy calls it's shutdown_session on app.teardown_appcontext since Flask version 0.9. Thats why nothing happens after test client call, bacause app context started by flask.ext.testing.TestCase even before test's setUp and will be closed after tearDown.
I got the same problem when tried to use Flask-Manage to run my tests. Running tests in a separate thread solved the problem.
import threading
# some code omited
runner = unittest.TextTestRunner(verbosity=2)
t = threading.Thread(target=runner.run, args=[test_suite])
t.start()
t.join()
# other code omited