Develop Reference

SQLAlchemy with asyncpg crashing with error: asyncpg.InterfaceError - cannot perform operation: another operation is in progress

I am developing a fastapi server using sqlalchemy and asyncpg to work with a postgres database. For each request, a new session is created (via fastapi dependency injection, as in the documentation). I used sqlite+aiosqlite before postgres+asyncpg and everything worked perfectly. After I switched from sqlite to postgres, every fastapi request crashed with the error:
sqlalchemy.dialects.postgresql.asyncpg.InterfaceError - cannot perform operation: another operation is in progress
This is how I create the engine and sessions:
from typing import Generator
import os
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
user = os.getenv('PG_USER')
password = os.getenv('PG_PASSWORD')
domain = os.getenv('PG_DOMAIN')
db = os.getenv('PG_DATABASE')
# db_async_url = f'sqlite+aiosqlite:///database.sqlite3'
db_async_url = f'postgresql+asyncpg://{user}:{password}#{domain}/{db}'
async_engine = create_async_engine(
db_async_url, future=True, echo=True
)
create_async_session = sessionmaker(
async_engine, class_=AsyncSession, expire_on_commit=False
)
async def get_async_session() -> Generator[AsyncSession]:
async with create_async_session() as session:
yield session

The error disappeared after adding poolclass=NullPool to create_async_engine, so here's what engine creation looks like now:
from sqlalchemy.pool import NullPool
...
async_engine = create_async_engine(
db_async_url, future=True, echo=True, poolclass=NullPool
)
I spent more than a day to solve this problem. I hope my answer will save a lot of time for other developers. Perhaps there are other solutions, and if so, I will be glad to see them here.

Parallelize integration tests with pytest-xdist

I was wondering if it is possible to use an in memory SQLite database to perform integration tests in parallel using pytest and pytrest-xdist on a FastAPI application?
Update
I have a good number of tests that I would like to run during my CI (GitLab CI), however, due to the number of IOPS that need to be executed for each test when using a file for SQLite the job times out so I would like to use an in-memory database, as well as parallelize the tests using pytest-xdist.
Every endpoint uses FastAPI's dependency injection for the db context, and what I have tried is to create a fixture for the app as so:
#pytest.fixture(scope="function")
def app():
"""
Pytest fixture that creates an instance of the FastAPI application.
"""
app = create_app()
app.dependency_overrides[get_db] = override_get_db
return app
def override_get_db():
SQLALCHEMY_DATABASE_URL = f"sqlite:///:memory:"
engine = create_engine(
SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
TestLocalSession = sessionmaker(autocommit=False, autoflush=False, bind=engine)
init_db(session=TestLocalSession)
engine.execute("PRAGMA foreign_keys=ON;")
try:
db = TestLocalSession()
yield db
finally:
db.close()
Because the endpoints are all sync I also need to use httpx instead of the built in TestClient:
#pytest.fixture(scope='function')
async def client(app):
"""
Pytest fixture that creates an instance of the Flask test client.
"""
async with AsyncClient(
app=app, base_url=f"{settings.BASE_URL}{settings.API_PREFIX}"
) as client:
yield client
The issue I have when I run this test (without pytest-xdist) is that the database is being created in a seperate thread as that which is being injected into the endpoints so I always get a SQL error: sqlite3.OperationalError: no such table: certification
Any suggestion on how to solve this? Thanks.

FastAPI dependencies (yield): how to call them manually?

FastAPI uses Depends() to inject variables either returned or yielded. Eg, FastAPI/SQL:
# Dependency
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
...
def create_user(db: Session = Depends(get_db)):
...
If I wanted to use that get_db() somewhere else (outside a FastAPI route), how would I do that? I know it's Python core knowledge, but I can't seem to figure it out. My initial thought was db = yield from get_db(), but I can't call yield from in async functions (and don't know if it would work besides). Then I tried:
with get_db() as db:
pass
Which fails as the original get_db() isn't wrapped as a #contextmanager. (Note, I don't want to decorate this - I'm using get_db as an example, I need to work with more complicated dependencies). Finally, I tried db = next(get_db()) - which works, but I don't think that's the correct solution. When/how will finally be invoked - when my method returns? And in some other dependencies, there's post-yield code that needs to execute; would I need to call next() again to ensure that code executes? Seems like next() isn't the right way. Any ideas?

You can use contextmanager not as a decorator but as a function returning context manager:
from contextlib import contextmanager
# Dependency
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
# synchronously
with contextmanager(get_db)() as session: # execute until yield. Session is yielded value
pass
# execute finally on exit from with
But keep in mind that the code will execute synchronously. If you want to execute it in a thread, then you can use the FastAPI tools:
import asyncio
from contextlib import contextmanager
from fastapi.concurrency import contextmanager_in_threadpool
async def some_coro():
async with contextmanager_in_threadpool(contextmanager(get_db)()) as session:
pass

Invoke method on SQLite connection in Flask-SQLAlchemy

I'm developing a web app with Flask-SQLAlchemy backed by a SQLite database. I need to call a method (create_collation) right after connecting. Without the SQLAlchemy framework, I can do that like this:
conn = sqlite3.connect(path)
conn.create_collation('my_collate', my_collate)
# ... go on and do fancy "order_by" stuff.
How do I do that in Flask-SQLAlchemy? Based on the API I was thinking of the following, but I get AttributeError: 'Engine' object has no attribute 'create_collation'.
from flask_sqlalchemy import SQLAlchemy
class MySQLAlchemy(SQLAlchemy):
def create_engine(self, sa_url, engine_opts):
engine = super().create_engine(sa_url, engine_opts)
engine.create_collation('my_collate', self.my_collate)
return engine
#staticmethod
def my_collate(string1, string2):
return string1.locateCompare(string2)
Following the SQLAlchemy docs I think I need to get the connection rather than the engine. But I can't find out how.
Also, where should this go specifically in Flask-SQLAlchemy? What part ultimately "connect"s, and how do I tune into that?

SQLAlchemy has an Events API that allows you to create a function that will be called whenever the connection pool creates a new connection:
from sqlalchemy.event import listens_for
from sqlalchemy.pool import Pool
#listens_for(Pool, "connect")
def my_on_connect(dbapi_con, connection_record):
dbapi_con.create_collation('my_collate', my_collate)

WSGI app factory pattern and import time created module level objects

I've been thinking about the factory pattern for WSGI applications, as recommended by the Flask docs, for a while now. Specifically about those functions usually being shown to make use of objects that have been created at module import time, like db in the example, as opposed to having been created in the factory function.
Would the factory function ideally create _everything_ anew or wouldn't that make sense for objects like the db engine?
(I'm thinking cleaner separation and better testability here.)
Here is some code, where I'm trying to accomplish creating all needed objects for the wsgi app. in its factory function.
# factories.py
def create_app(config, engine=None):
"""Create WSGI application to be called by WSGI server. Full factory function
that takes care to deliver entirely new WSGI application instance with all
new member objects like database engine etc.
Args:
config (dict): Dict to update the wsgi app. configuration.
engine (SQLAlchemy engine): Database engine to use.
"""
# flask app
app = Flask(__name__) # should be package name instead of __name__ acc. to docs
app.config.update(config)
# create blueprint
blueprint = ViewRegistrationBlueprint('blueprint', __name__, )
# request teardown behaviour, always called, even on unhandled exceptions
# register views for blueprint
from myapp.views import hello_world
# dynamically scrapes module and registers methods as views
blueprint.register_routes(hello_world)
# create engine and request scoped session for current configuration and store
# on wsgi app
if (engine is not None):
# delivers transactional scope when called
RequestScopedSession = scoped_session(
sessionmaker(bind=engine),
scopefunc=flask_request_scope_func
)
def request_scoped_session_teardown(*args, **kwargs):
"""Function to register and call by the framework when a request is finished
and the session should be removed.
"""
# wrapped in try/finally to make sure no error collapses call stack here
try:
RequestScopedSession.remove() # rollback all pending changes, close and return conn. to pool
except Exception as exception_instance:
msg = "Error removing session in request teardown.\n{}"
msg = msg.format(exception_instance)
logger.error(msg)
finally:
pass
app.config["session"] = RequestScopedSession
blueprint.teardown_request(request_scoped_session_teardown)
# register blueprint
app.register_blueprint(blueprint)
return app
def create_engine(config):
"""Create database engine from configuration
Args:
config (dict): Dict used to assemble the connection string.
"""
# connection_string
connection_string = "{connector}://{user}:{password}#{host}/{schema}"
connection_string = connection_string.format(**config)
# database engine
return sqlalchemy_create_engine(
connection_string,
pool_size=10,
pool_recycle=7200,
max_overflow=0,
echo=True
)
# wsgi.py (served by WSGI server)
from myapp.factories import create_app
from myapp.factories import create_engine
from myapp.configuration.config import Config
config = Config()
engine = create_engine(config.database_config)
app = create_app(config.application_config, engine=engine)
# conftest.py
from myapp.factories import create_app
from myapp.factories import create_engine
from myapp.configuration.config import Config
#pytest.fixture
def app():
config = TestConfig()
engine = create_engine(config.database_config)
app = create_app(config.application_config, engine=engine)
with app.app_context():
yield app

As you also tagged this with sanic I'll answer with that background. Sanic is async and thus relies on an event loop. An event loop is a resource and thus must not be shared between tests but created anew for each one. Hence, the database connection etc also need to be created for each test and cannot be re-used as it is async and depends on the event loop. Even without the async nature it would be cleanest to create db connections per test because they have state (like temp tables).
So I ended up with a create_app() that creates everything which allows me to create an arbitrary number of independent apps in a test run. (To be honest there are some global resources like registered event listeners, but tearing those down is easy with py.test factories.) For testability I'd try to avoid global resources that are created on module import. Although I've seen differently in big and successful projects.
That's not really a definite answer, I know...