Per the docs, we should use the following pattern with a sessionmaker object:
Session = sessionmaker(engine)
with Session.begin() as session:
session.add(some_object)
In a multithreaded environment, we are also supposed to use a single scoped_session and share it. So in my __init__.py I create create one and import it everywhere else in my program:
engine = create_engine(config.SQLALCHEMY_DATABASE_URI)
Session = scoped_session(sessionmaker(bind=engine))
The question is, how am I supposed to combine these two approaches? This seems to be the suggested way, but it errors out:
from myapp import Session
with Session.begin() as session:
query_result = session.query(MyModel).all()
----
Exception has occurred: AttributeError
'SessionTransaction' object has no attribute 'query'
I tried the following and it works, but it seems like it doesn't follow the docs, and I'm afraid it breaks something not obvious. Can anyone confirm if this is correct?
from myapp import Session
with Session() as session, session.begin():
query_result = session.query(MyModel).all()
I've been looking around at other replies and seeing very little that addresses the specific question.
From the Session.begin() docs:
The Session object features autobegin behavior, so that normally it is not necessary to call the Session.begin() method explicitly. However, it may be used in order to control the scope of when the transactional state is begun.
You can use Session.begin() (new in 1.4) to obtain a SessionTransaction instance usable as a context manager which will autocommit on successful exit.
Calling the scoped_session returns a SessionTransaction right away, as per your error, so you do not need to begin it again.
All in all, you can definitely do the stacked context manager, but its unnecessary, so you might as well stick to using the original flow:
Session = scoped_session(...)
with Session() as session: # NB. session is a thread local SessionTransaction
...
session.commit()
or the proxied Session
Session = scoped_session(...)
#on_request_end
def remove_session(req):
Session.remove()
#route("/xyz", ...)
def handle_xyz():
instance = Class(...)
Session.add(instance)
Session.commit()
Related
I have a Flask REST API, running with a gunicorn/nginx stack. There is global SQLAlchemy session set up once for each thread that the API runs on. I set up an endpoint /test/ for running the unit tests for the API. One test makes a POST request to add something to the database, then has a finally: clause to clean up:
def test_something():
try:
url = "http://myposturl"
data = {"content" : "test post"}
headers = {'content-type': 'application/json'}
result = requests.post(url, json=data, headers=headers).json()
validate(result, myschema)
finally:
db.sqlsession.query(MyTable).filter(MyTable.content == "test post").delete()
db.sqlsession.commit()
The problem is that the thread to which the POST request is made now has a "test post" object in its session, but the database has no such object because the thread on which the tests ran deleted that thing from the database. So when I make a GET request to the server, about 1 in 4 times (I have 4 gunicorn workers), I get the "test post" object, and 3 in 4 times I do not. This is because the threads each have their own session object, and they are getting out of sync, but I don't really know what to do about it....
Here is my setup for my SQLAlchemy session:
def connectSQLAlchemy():
import sqlalchemy
import sqlalchemy.orm
engine = sqlalchemy.create_engine(connection_string(DBConfig.USER, DBConfig.PASSWORD, DBConfig.HOST, DBConfig.DB))
session_factory = sqlalchemy.orm.sessionmaker(bind=engine)
Session = sqlalchemy.orm.scoped_session(session_factory)
return Session()
# Create a global session for everyone
sqlsession = connectSQLAlchemy()
Please use flask-sqlalchemy if you're using flask, it takes care of the lifecycle of the session for you.
If you insist on doing it yourself, the correct pattern is to create a session for each request instead of having a global session. You should be doing
Session = scoped_session(session_factory, scopefunc=flask._app_ctx_stack.__ident_func__)
return Session
instead of
Session = scoped_session(session_factory)
return Session()
And do
session = Session()
every time you need a session. By virtue of the scoped_session and the scopefunc, this will return you a different session in each request, but the same session in the same request.
Figured it out. What I did was to add a setup and teardown to the request in my app's __init__.py:
#app.before_request
def startup_session():
db.session = db.connectSQLAlchemy()
#app.teardown_request
def shutdown_session(exception=None):
db.session.close()
still using the global session object in my db module:
db.py:
....
session = None
....
The scoped_session handles the different threads, I think...
Please advise if this is a terrible way to do this for some reason. =c)
I am using SQLAlchemy's provided contextmanager to handle sessions for me. What I don't understand is how to get the automatically generated ID because (1) the ID is not created until after commit() is called yet (2) the newly created instance is only available in the context manager's scope:
def save_soft_file(name, is_geo=False):
with session_scope() as session:
soft_file = models.SoftFile(name=name, is_geo=is_geo)
session.add(soft_file)
# id is not available here, because the session has not been committed
# soft_file is not available here, because the session is out of context
return soft_file.id
What am I missing?
Use session.flush() to execute pending commands within the current transaction.
def save_soft_file(name, is_geo=False):
with session_scope() as session:
soft_file = models.SoftFile(name=name, is_geo=is_geo)
session.add(soft_file)
session.flush()
return soft_file.id
If an exception occurs after a flush but before the session goes out of scope, the changes will be rolled back to the beginning of the transaction. In that case your soft_file would not actually be written to the database, even though it had been given an ID.
I'm using Flask-Testing which says:
Another gotcha is that Flask-SQLAlchemy also removes the session
instance at the end of every request (as should any threadsafe
application using SQLAlchemy with scoped_session). Therefore the
session is cleared along with any objects added to it every time you
call client.get() or another client method.
However, I'm not seeing that. This test fails:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.testing import TestCase
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
#app.route('/')
def index():
print 'before request:', `db.session`
u = db.session.query(User).first()
u.name = 'bob'
return ''
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
class SessionTest(TestCase):
def create_app(self):
return app
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
(Run with $ nosetests test_file.py to see it in action.)
stdout:
-------------------- >> begin captured stdout << ---------------------
before request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
after request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
bob
--------------------- >> end captured stdout << ----------------------
According to the docs, user u should not be in the session after a get request, but it is! Does anybody know why this is happening?
Furthermore, u.name is bob and not joe, even though the request never committed! (So I'm convinced it's the same session.)
For the record,
$ pip freeze | grep Flask
Flask==0.10.1
Flask-Bcrypt==0.5.2
Flask-DebugToolbar==0.8.0
Flask-Failsafe==0.1
Flask-SQLAlchemy==0.16
Flask-Script==0.6.2
Flask-Testing==0.4
Flask-Uploads==0.1.3
Flask-WTF==0.8
I'm pretty sure the confusion comes from the fact that sessions in SQLAlchemy are scoped, meaning that each request handler creates and destroys its own session.
This is necessary because web servers can be multi-threaded, so multiple requests might be served at the same time, each working with a different database session.
For this reason, the session that you used outside of the context of a request is likely not the same session that the view function that handles the '/' route gets and then destroys at the end.
UPDATE: I dug around a bit and figured this thing out.
Flask-SQLAlchemy installs a hook on app.teardown_appcontext, and here is where it calls db.session.remove().
The testing environment does not fully replicate the environment of a real request because it does not push/pop the application context. Because of that the session is never removed at the end of the request.
As a side note, keep in mind that functions registered with before_request and after_request are also not called when you call client.get().
You can force an application context push and pop with a small change to your test:
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
with app.app_context():
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
with this change the test passes.
The documentation for Flask-Testing seems to be wrong or more likely outdated. Maybe things worked like they describe at some point, but that isn't accurate for current Flask and Flask-SQLAlchemy versions.
I hope this helps!
FlaskClient works with request context while Flask-SQLAlchemy calls it's shutdown_session on app.teardown_appcontext since Flask version 0.9. Thats why nothing happens after test client call, bacause app context started by flask.ext.testing.TestCase even before test's setUp and will be closed after tearDown.
I got the same problem when tried to use Flask-Manage to run my tests. Running tests in a separate thread solved the problem.
import threading
# some code omited
runner = unittest.TextTestRunner(verbosity=2)
t = threading.Thread(target=runner.run, args=[test_suite])
t.start()
t.join()
# other code omited
I'm trying to do a two-phase commit using SQLalchemy 0.6.8 with Postgresql 8.3.4, but I think I'm missing something...
The workflow goes like this:
session = sessionmaker(engine)(autocommit=True)
tx = session.connection().begin_twophase(xid) # Doesn't issue any SQL
session.begin()
session.add(obj1)
session.flush()
tx.prepare()
then from another session
session = sessionmaker(engine)(autocommit=True)
session.connection().commit_prepared(xid, recover=True) # recover=True because otherwise it complains that you can't issue a COMMIT PREPARED from inside a transaction
This doesn't raise any error, but doesn't write anything to the table either... O_o
What am I missing?
I tried even blocking the application after the prepare() and issuing a COMMIT PREPARED 'xid' from pgadmin, but still nothing gets written.
I managed to get it working, here's how:
session = sessionmaker(engine)(twophase=True)
session.add(obj1)
session.prepare()
# Find transaction id
for k, v in s.transaction._connections.iteritems():
if isinstance(k, Connection):
return v[1].xid
then from another session
session = sessionmaker(engine)(twophase=True)
session.connection().commit_prepared(xid, recover=True)
Using tornado, I want to create a bit of middleware magic that ensures that my SQLAlchemy sessions get properly closed/cleaned up so that objects aren't shared from one request to the next. The trick is that, since some of my tornado handlers are asynchronous, I can't just share one session for each request.
So I am left trying to create a ScopedSession that knows how to create a new session for each request. All I need to do is define a scopefunc for my code that can turn the currently executing request into a unique key of some sort, however I can't seem to figure out how to get the current request at any one point in time (outside of the scope of the current RequestHandler, which my function doesn't have access to either).
Is there something I can do to make this work?
You might want to associate the Session with the request itself (i.e. don't use scopedsession if it's not convenient). Then you can just say, request.session. Still needs to have hooks at the start/end for setup/teardown.
edit: custom scoping function
def get_current_tornado_request():
# TODO: ask on the Tornado mailing list how
# to acquire the request currently being invoked
Session = scoped_session(sessionmaker(), scopefunc=get_current_tornado_request)
(This is a 2017 answer to a 2011 question) As #Stefano Borini pointed out, easiest way in Tornado 4 is to just let the RequestHandler implicitly pass the session around. Tornado will track the handler instance state when using coroutine decorator patterns:
import logging
_logger = logging.getLogger(__name__)
from sqlalchemy import create_engine, exc as sqla_exc
from sqlalchemy.orm import sessionmaker, exc as orm_exc
from tornado import gen
from tornado.web import RequestHandler
from my_models import SQLA_Class
Session = sessionmaker(bind=create_engine(...))
class BaseHandler(RequestHandler):
#gen.coroutine
def prepare():
self.db_session = Session()
def on_finish():
self.db_session.close()
class MyHander(BaseHandler):
#gen.coroutine
def post():
SQLA_Object = self.db_session.query(SQLA_Class)...
SQLA_Object.attribute = ...
try:
db_session.commit()
except sqla_exc.SQLAlchemyError:
_logger.exception("Couldn't commit")
db_session.rollback()
If you really really need to asynchronously reference a SQL Alchemy session inside a declarative_base (which I would consider an anti-pattern since it over-couples the model to the application), Amit Matani has a non-working example here.