Develop Reference

What is the correct way of using Flask-Sqlalchemy with multiprocessing?

I have a route in my Flask app that spawns a process (using multiprocessing.Process) to do some background work. That process needs to be able to write to the database.
__init__.py:
from flask_sqlalchemy import SQLAlchemy
from project.config import Config
db = SQLAlchemy()
# app factory
def create_app(config_class=Config):
app = Flask(__name__)
app.config.from_object(Config)
db.init_app(app)
return app
And this is the relevant code that illustrates that way i'm spawning the process and using the db connection:
def worker(row_id):
db_session = db.create_scoped_session()
# Do stuff with db_session here
db_session.close()
#app.route('/worker/<row_id>/start')
def start(row_id):
p = Process(target=worker, args=(row_id,))
p.start()
return redirect('/')
The problem is that sometimes (not always) i have errors like this one:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) insufficient data in "D" message lost synchronization with server: got message type "a", length 1668573551
I assume that this is related to the fact that there is another process accessing the database (because if i don't use a separate process, everything works fine) but i honestly can't find a way of fixing it. As you can see on my code, i tried used create_scoped_session() method as an attempt to fix this but the problem is the same.
Any help?

Ok so, i followed #antont 's hint and created a new sqlalchemy session inside the worker function this way and it worked flawlessly:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
def worker(row_id):
db_url = os.environ['DATABASE_URL']
db_engine = create_engine(db_url)
Session = sessionmaker(bind=db_engine)
db_session = Session()
# Do stuff with db_session here
db_session.close()

SQLAlchemy database session not being removed after web request

I'm using SQLAlchemy in my Flask app, and after a few requests have been made the app fails, with this error emitted:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at ..., port 5432 failed: FATAL: too many connections for role "...". I then have to restart the app to get it working again.
It seems as though with each web request, a database connection is being created but not removed. I can't figure out why that is, as I've added scoped_session.remove() to an #app.teardown_request function.
Here is my code...
app.py
...
#app.before_request
def before_request():
global database
database = PostgreSQL()
#app.teardown_request
def teardown(response):
database.session.remove()
...
PostgreSQL.py
class PostgreSQL:
def __init__(self):
self.database_url = os.environ.get('DATABASE_URL')
if self.database_url.startswith('postgres://'):
self.database_url = self.database_url.replace('postgres://', 'postgresql://')
self.engine = create_engine(self.database_url, echo = False)
Base.metadata.create_all(self.engine, checkfirst = True)
self.session = scoped_session(sessionmaker(bind = self.engine))
...
I guess I'd never have run into this problem if I used Flask-SQLAlchemy (which appears to handle things like database sessions for you) from the outset, but I'm keen to follow this problem through without resorting to that additional layer of abstraction - so please refrain from mentioning that module, unless it really is the only viable option here.
EDIT: Figured out the problem - needed to call database.engine.dispose() in #app.teardown_request

QueuePool limit reached for Flask application

I am running an application with Flask and Flask-SQLAlchemy.
from config import FlaskDatabaseConfig
from flask import Flask
from flask import request
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
application = Flask(__name__)
application.config.from_object(FlaskDatabaseConfig())
db = SQLAlchemy(application)
#application.route("/queue/request", methods=["POST"])
def handle_queued_request():
stuff()
return ""
def stuff():
# Includes database queries and updates and a call to db.session.commit()
# db.session.begin() and db.session.close() are not called
pass
if __name__ == "__main__":
application.run(debug=False, port=5001)
Now, from my understanding, by using Flask-SQLAlchemy I do not need to manage sessions on my own. So why am I getting the following error if I run several requests turn by turn to my endpoint?
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
I've tried using db.session.close() but then, instead of this error, my database updates are not committed properly. What am I doing incorrectly? Do I need to manually close connections with the database once a request has been handled?

I have found a solution to this. The issue was that I had a lot of processes that were "idle in transaction" because I did not call db.session.commit() after making certain database SELECT statements using Query.first()
To investigate this, I queried my (development) PostgreSQL database directly using:
SELECT * FROM pg_stat_activity

Just remove connection every time you make a query session to db.
products = db.session.query(Product).limit(20).all()
db.session.remove()

Flask unit tests with SQLAlchemy and PostgreSQL exhausts db connections

I'm running a suite of fairly straightforward test cases using Flask, SQLAlchemy, and PostgreSQL. Using an application factory, I've defined a base unit test class like so:
class BaseTestCase(unittest.TestCase):
def setUp(self):
self.app = create_app()
self.app.config.from_object('app.config.test')
self.api_base = '/api/v1'
self.ctx = self.app.test_request_context()
self.ctx.push()
self.client = self.app.test_client()
db.create_all()
def tearDown(self):
db.session.remove()
db.drop_all(app=self.app)
print db.engine.pool.status()
if self.ctx is not None:
self.ctx.pop()
All goes well for a few unit tests, up until:
OperationalError: (OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections
It seems that there is only 1 connection in the pool (db.engine.pool.status() for each test shows: Pool size: 5 Connections in pool: 1 Current Overflow: -4 Current Checked out connections: 0), but somehow the app never disconnects. Clearly a new app instance is created for each test case, which seems fine according to the documentation. If I move the app creation to the module level it works fine.
Does anyone know why this is happening?
Thanks

Add db.get_engine(self.app).dispose() after db.drop_all()

Pylons: Sharing SQLAlchemy MySQL connection with external library

I am running Pylons using SQLAlchemy to connect to MySQL, so when I want to use a database connection in a controller, I can do this:
from myapp.model.meta import Session
class SomeController(BaseController):
def index(self):
conn = Session.connection()
rows = conn.execute('SELECT whatever')
...
Say my controller needs to call up an external library, that also needs a database connection, and I want to provide the connection for it from the SQLAlchemy MySQL connection that is already established:
from myapp.model.meta import Session
import mymodule
class SomeController(BaseController):
def index(self):
conn = Session.connection()
myobject = mymodule.someobject(DATABASE_OBJECT)
...
conn.close()
What should DATABSE_OBJECT be? Possibilities:
Pass Session -- and then open and close Session.connection() in the module code
Pass conn, and then call conn.close() in the controller
Just pass the connection parameters, and have the module code set up its own connection
There is another wrinkle, which is that I need to instantiate some objects in app_globals.py, and these objects need a database connection as well. It seems that app_globals.py cannot use Session's SQLAlchemy connection yet -- it's not bound yet.
Is my architecture fundamentally unsounds? Should I not be trying to share connections between Pylons and external libraries this way? Thanks!

You should not manage connections yourself - it's all done by SQLAlchemy. Just use scoped session object everywhere, and you will be fine.
def init_model(engine):
sm = orm.sessionmaker(autoflush=False, autocommit=False, expire_on_commit=False, bind=engine)
meta.engine = engine
meta.Session = orm.scoped_session(sm)
def index(self):
rows = Session.execute('SELECT ...')
You can pass Session object to your external library and do queries there as you wish. There is no need to call .close() on it.
Regarding app_globals, I solved that by adding other method in globals class which is called after db initialization from environment.py
class Globals(...):
def init_model(self, config):
self.some_persistent_db_object = Session.execute('...')
def load_environment(...):
...
config['pylons.app_globals'].init_model(config)
return config

What should DATABSE_OBJECT be? Possibilities:
4. pass a "proxy" or "helper" object with higher level of abstraction interface
Unless the external library really needs direct access to SQLAlchemy session, you could provide it with object that has methods like "get_account(account_no)" instead of "execute(sql)". Doing so would keep SQLAlchemy-specific code more isolated, and the code would be also easier to test.
Sorry that this is not so much an answer to your original question, more a design suggestion.