How to implement SQLAlchemy Postgres async driver in a Django project? - python

I have come across really weird behavior implementing SQLAlchemy in my Django project using the asynchronous Postgres driver.
When I create a SQLAlchemy Session using a sessionmaker and then execute a simple SELECT statement within a session context manager, it generates the correct result the first time, but the proceeding attempts result in a RuntimeError exception stating "Task ... attached to a different loop". I looked up the error and it usually indicates that the session is persisted across multiple Django requests; however, in my code, I don't see how that is possible since I am creating the database session within the request using the sessionmaker and using the session context manager to close it.
It gets weirder when reloading the page for a 3rd time using the session context manager approach because the RuntimeError changes to an InterfaceError stating "cannot perform operation: another operation is in progress". But when not using the context manager approach and creating the session manually and immediately after using it closing the session it only produces the RuntimeError exception even after the 2nd attempt.
And it continues to get weird, when I do the second approach and do not await the session close, then it works completely fine other than a log message stating that the method was never awaited.
I have created a GitHub repository with a docker-compose.yml that reproduces the issue if anyone wants to see it for themselves.
https://github.com/enots227/Django-SQLAlchemy-Async
Django Setup
settings.py
# Database
DB_ENGINE = create_async_engine(URL.create(
"postgresql+asyncpg",
username=os.getenv('DB_USERNAME'),
password=os.getenv('DB_PASSWORD'),
host=os.getenv('DB_HOST'),
port=os.getenv('DB_PORT'),
database=os.getenv('DB_NAME')), echo=True)
DB_SESSION_MAKER = sessionmaker(DB_ENGINE, AsyncSession, expire_on_commit=False)
Approach 1: Using Context Manager
Here is my code for the Django view for the context manager (http://127.0.0.1:8000/sql/1/)
async def view1(request):
html = '<html><body><ul>'
async with settings.DB_SESSION_MAKER() as db_session:
items = await list_accounts(db_session)
for item in items:
html += '<li>' + item.name + '</li>'
html += '</ul></body></html>'
return HttpResponse(html)
For the first execution, it returns back the correct response:
<html><body><ul><li>test1</li></ul></body></html>
But if I try to reload the page after the first execution, it then raises a RuntimeError exception stating "Task ... attached to a different loop".
Internal Server Error: /sql/1/
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 204, in __call__
return call_result.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 270, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/usr/src/app/src/sql/views.py", line 12, in view1
items = await list_accounts(db_session)
File "/usr/src/app/src/sql/models.py", line 20, in list_accounts
return await db_session.scalars(select(Account))
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 269, in scalars
result = await self.execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 212, in execute
result = await greenlet_spawn(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 134, in greenlet_spawn
result = context.throw(*sys.exc_info())
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
return current.driver.switch(awaitable)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
value = await result
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
await adapt_connection._start_transaction()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
self._handle_exception(error)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 714, in _start_transaction
await self._transaction.start()
File "/usr/local/lib/python3.10/site-packages/asyncpg/transaction.py", line 138, in start
await self._connection.execute(query)
File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 318, in execute
return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 338, in query
RuntimeError: Task <Task pending name='Task-5' coro=<AsyncToSync.main_wrap() running at /usr/local/lib/python3.10/site-packages/asgiref/sync.py:270> cb=[_run_until_complete_cb() at /usr/local/lib/python3.10/asyncio/base_events.py:184]> got Future <Future pending cb=[Protocol._on_waiter_completed()]> attached to a different loop
And when I reload for the 3rd time it raises an InterfaceError stating "cannot perform operation: another operation is in progress".
Internal Server Error: /sql/1/
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 714, in _start_transaction
await self._transaction.start()
File "/usr/local/lib/python3.10/site-packages/asyncpg/transaction.py", line 138, in start
await self._connection.execute(query)
File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 318, in execute
return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 323, in query
File "asyncpg/protocol/protocol.pyx", line 707, in asyncpg.protocol.protocol.BaseProtocol._check_state
asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
return current.driver.switch(awaitable)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
value = await result
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
await adapt_connection._start_transaction()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
self._handle_exception(error)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 682, in _handle_exception
raise translated_error from error
sqlalchemy.dialects.postgresql.asyncpg.AsyncAdapt_asyncpg_dbapi.InterfaceError: <class 'asyncpg.exceptions._base.InterfaceError'>: cannot perform operation: another operation is in progress
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 204, in __call__
return call_result.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 270, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/usr/src/app/src/sql/views.py", line 12, in view1
items = await list_accounts(db_session)
File "/usr/src/app/src/sql/models.py", line 20, in list_accounts
return await db_session.scalars(select(Account))
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 269, in scalars
result = await self.execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 212, in execute
result = await greenlet_spawn(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 134, in greenlet_spawn
result = context.throw(*sys.exc_info())
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
return current.driver.switch(awaitable)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
value = await result
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
await adapt_connection._start_transaction()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
self._handle_exception(error)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 682, in _handle_exception
raise translated_error from error
sqlalchemy.exc.InterfaceError: (sqlalchemy.dialects.postgresql.asyncpg.InterfaceError) <class 'asyncpg.exceptions._base.InterfaceError'>: cannot perform operation: another operation is in progress
[SQL: SELECT account.acct_id, account.name
FROM account]
(Background on this error at: https://sqlalche.me/e/14/rvf5)
Approach 2: Manually Creating and Closing Session
Here is my code for the Django view by manually managing the session (http://127.0.0.1:8000/sql/2/)
async def view2(request):
html = '<html><body><ul>'
db_session: AsyncSession = settings.DB_SESSION_MAKER()
items = await list_accounts(db_session)
await db_session.close()
for item in items:
html += '<li>' + item.name + '</li>'
html += '</ul></body></html>'
return HttpResponse(html)
For the first execution, it returns back the correct response as approach 1. After the first execution, I get the RuntimeError in approach 1 and never get the InterfaceError.
Approach 3: Manually Creating and Closing Session (Not Awaiting Close)
Here is my code for the Django view for manually managing the session and not awaiting the close method (http://127.0.0.1:8000/sql/2/)
async def view2(request):
html = '<html><body><ul>'
db_session: AsyncSession = settings.DB_SESSION_MAKER()
items = await list_accounts(db_session)
db_session.close() # logs async warning since missing await, but works
for item in items:
html += '<li>' + item.name + '</li>'
html += '</ul></body></html>'
return HttpResponse(html)
All executions always return back the correct response.

Related

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) with PostgreSQL

I searched for this error a lot, but I only find some with more information behind that like "FATAL: ...". Mine has none. It only says
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
I have a postgres database inside a docker container that has set his port to the standard 5432.
I've created the container with the following command:
docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
It is clean so no database created. The API should automatically create them.
I'm using Pycharm IDE, maybe it has something todo with that.
Traceback (most recent call last):
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 373, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\debug.py", line 96, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\debug.py", line 93, in __call__
await self.app(scope, receive, inner_send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\applications.py", line 208, in __call__
await super().__call__(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\exceptions.py", line 82, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 580, in __call__
await route.handle(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 241, in handle
await self.app(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 52, in app
response = await func(request)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\routing.py", line 226, in app
raw_response = await run_endpoint_function(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\routing.py", line 159, in run_endpoint_function
return await dependant.call(**values)
File "C:\Users\Veril\PycharmProjects\partyapp-python\app\routers\v1\users.py", line 31, in create_user
session.commit()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 1428, in commit
self._transaction.commit(_to_root=self.future)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 829, in commit
self._prepare_impl()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 808, in _prepare_impl
self.session.flush()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3339, in flush
self._flush(objects)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3479, in _flush
transaction.rollback(_capture_exception=True)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3439, in _flush
flush_context.execute()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 456, in execute
rec.execute(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 209, in save_obj
for (
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 370, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 1709, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 626, in connection
return self._connection_for_bind(bind, execution_options)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 735, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 747, in _connection_for_bind
conn = bind.connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\future\engine.py", line 419, in connect
return super(Engine, self).connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3194, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 96, in __init__
else engine.raw_connection()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3273, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3243, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2097, in _handle_dbapi_exception_noconnection
util.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3240, in _wrap_pool_connect
return fn()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 476, in checkout
rec = pool._do_get()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 146, in _do_get
self._dec_overflow()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 143, in _do_get
return self._create_connection()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 256, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 371, in __init__
self.__connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 666, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 661, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\create.py", line 590, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\default.py", line 584, in connect
return self.dbapi.connect(*cargs, **cparams)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\psycopg2\__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
(Background on this error at: https://sqlalche.me/e/14/e3q8)
This is my error that i get. My code looks like this:
main.py:
import os
import uvicorn
if __name__ == '__main__':
port = int(os.getenv("PORT"))
uvicorn.run("main:app", host='0.0.0.0', port=port, reload=True, debug=True, workers=3)
app/main.py:
import os
from fastapi import FastAPI
from .database import engine
from .routers import v1
engine.init_db()
port = int(os.getenv("PORT")) #Port is 8000
app = FastAPI()
app.include_router(v1.router, prefix="/v1")
app/database/engine.py: (referenced in the file above)
import os
from fastapi.security import HTTPBearer
from sqlmodel import create_engine, SQLModel, Session
DATABASE_URL = "postgresql+psycopg2://postgres:mysecretpassword#localhost:5432"
engine = create_engine(DATABASE_URL, echo=True)
token_auth_scheme = HTTPBearer()
async def init_db():
async with engine.begin() as conn:
# await conn.run_sync(SQLModel.metadata.drop_all)
await conn.run_sync(SQLModel.metadata.create_all)
async def get_session():
session = Session(engine)
try:
yield session
finally:
session.close()
The route "users" inside routers/v1/users.py:
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Path, Response, status
from pydantic import ValidationError
from sqlalchemy.exc import IntegrityError
from sqlalchemy.sql.functions import concat
from sqlalchemy import func
from sqlmodel import Session, select, col
from starlette import status
from app.database import models
from app.database.authentication import VerifyToken
from app.database.engine import get_session, token_auth_scheme
router = APIRouter()
#router.post("", status_code=status.HTTP_201_CREATED, response_model=models.UserRead,
response_model_exclude_none=True, name="Create User", tags=["users"])
async def create_user(user_data: models.UserCreate,
session: Session = Depends(get_session)):
try:
new_user = models.User(**dict(user_data))
session.add(new_user)
session.commit()
session.refresh(new_user)
return new_user
except IntegrityError:
session.rollback()
raise HTTPException(
status_code=status.HTTP_409_CONFLICT, detail="IntegrityError")
except ValidationError:
session.rollback()
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail="ValidationError")
The models.User:
class UserBase(SQLModel):
id: str
username: Optional[str]
country_code: Optional[str]
phone: Optional[str]
class Config:
allow_population_by_field_name = True
class User(UserBase, table=True):
__tablename__ = 'users'
id: str = Field(primary_key=True)
username: Optional[str] = Field(sa_column=Column('username', VARCHAR(length=50), unique=True, default=None))
phone: Optional[str] = Field(sa_column=Column('phone', VARCHAR(length=20), unique=True, default=None))
I hope that this is everything you guys need to find something. If you need something else contact me.
Best regards
Colin
EDIT:
After I changed the link from `postgresql+psycopg2` to `postgresql+asyncpg` I get a new error:
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 373, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\debug.py", line 96, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\uvicorn\middleware\debug.py", line 93, in __call__
await self.app(scope, receive, inner_send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\applications.py", line 208, in __call__
await super().__call__(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\exceptions.py", line 82, in __call__
raise exc from None
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 580, in __call__
await route.handle(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 241, in handle
await self.app(scope, receive, send)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\starlette\routing.py", line 52, in app
response = await func(request)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\routing.py", line 226, in app
raw_response = await run_endpoint_function(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\fastapi\routing.py", line 159, in run_endpoint_function
return await dependant.call(**values)
File "C:\Users\Veril\PycharmProjects\partyapp-python\app\routers\v1\users.py", line 26, in create_user
session.commit()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 1428, in commit
self._transaction.commit(_to_root=self.future)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 829, in commit
self._prepare_impl()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 808, in _prepare_impl
self.session.flush()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3339, in flush
self._flush(objects)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3479, in _flush
transaction.rollback(_capture_exception=True)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 3439, in _flush
flush_context.execute()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 456, in execute
rec.execute(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 209, in save_obj
for (
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 370, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\persistence.py", line 1709, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 626, in connection
return self._connection_for_bind(bind, execution_options)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 735, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\orm\session.py", line 747, in _connection_for_bind
conn = bind.connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\future\engine.py", line 419, in connect
return super(Engine, self).connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3194, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 96, in __init__
else engine.raw_connection()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3273, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3240, in _wrap_pool_connect
return fn()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 476, in checkout
rec = pool._do_get()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 146, in _do_get
self._dec_overflow()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 143, in _do_get
return self._create_connection()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 256, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 371, in __init__
self.__connect()
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 666, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
compat.raise_(
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\pool\base.py", line 661, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\create.py", line 590, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\engine\default.py", line 584, in connect
return self.dbapi.connect(*cargs, **cparams)
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\dialects\postgresql\asyncpg.py", line 748, in connect
await_only(self.asyncpg.connect(*arg, **kw)),
File "C:\Users\Veril\PycharmProjects\partyapp-python\venv\lib\site-packages\sqlalchemy\util\_concurrency_py3k.py", line 61, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)
EDIT 2:
If I use my local postgres database in docker i get this error. If I use my extern database from Heroku it works perfectly fine!
EDIT 3:
So apparently it won't work with my local docker postgres database. Now I am using my production database over heroku (so externally) and everything works with psycopg2. I didn't found the error why it won't let me use my local db but whatever.

Even though I can see that you are using default port, maybe this can help to somebody with the same problem.
For me the problem was in explicit port definition - as I'm running two different postgres DB and (both from containers), one of them I set to listen to 5433 port - and this gives me exact same problem, so instead
SQLALCHEMY_DATABASE_URL = "postgresql://postgres:password#localhost/fastapi"
I just put:
SQLALCHEMY_DATABASE_URL = "postgresql://postgres:password#localhost:5433/fastapi"
Problem solved right away.

Why do my Flask psycopg2 app returns sync error with server?

I'm facing issue with my flask app that uses psycopg2 for postgres. Sometimes my API returns 500 server error in which it said in the log file:
[SQL: SELECT gallery.id AS gallery_id, gallery.caption AS gallery_caption, gallery.image AS gallery_image, gallery.created_date AS gallery_created_date, gallery.updated_date AS gallery_updated_date
FROM gallery ORDER BY gallery.id]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
[UID:1549][94744] Child process with pid: 98159 was killed by signal: 15, core dump: 0
[ERROR] [UID:1549][97456] wsgiAppHandler pApp->start_response() return NULL.
/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:873: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
[2021-12-27 11:22:11,886] ERROR in app: Exception on /gallery/find-all [GET]
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: lost synchronization with server: got message type "G", length 1935229752
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/myapp/api/myapplication/auth.py", line 32, in decorated
return f(auth_data, *args, **kwargs)
File "/home/myapp/api/myapplication/gallery.py", line 71, in fetch_image
image_data = Gallery.query.filter().order_by(Gallery.id).all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2759, in all
return self._iter().all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) lost synchronization with server: got message type "G", length 1935229752
This API just fetch all the data from the table and returns them to the client. The table only have max 12 rows and contains base-64 string of image data. However, this also happens at other API that fetch a single data from other table and returns only text data-type.
These APIs works well in my local development env but went error in production (CPanel shared hosting)

"KeyError: 0" when connecting to MySQL (get_default_isolation_level)

Currently developing for python 3.9.6 with sqlalchemy version 1.4.23 and whenever I use sqlalchemy to connect to a database, I get an error with this stack trace:
Traceback (most recent call last):
File "/mnt/c/Importer/common/db.py", line 106, in db_cur
with engine.connect() as con:
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
return fn()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
return _ConnectionFairy._checkout(self)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
rec = pool._do_get()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
return _ConnectionRecord(self)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
self.__connect()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 622, in __connect
pool.dispatch.connect.for_modify(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 329, in _exec_w_sync_on_first_run
self(*args, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 343, in __call__
fn(*args, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1691, in go
return once_fn(*arg, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect
dialect.initialize(c)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2961, in initialize
default.DefaultDialect.initialize(self, connection)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 407, in initialize
self.default_isolation_level = self.get_default_isolation_level(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 473, in get_default_isolation_level
return self.get_isolation_level(dbapi_conn)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2720, in get_isolation_level
val = row[0]
KeyError: 0
I can fix the issue by manually changing the file from
val = row[0]
to
val = row['##transaction_isolation']
but I don't want to have to manually change dependency files after installing them. I have tried manually setting the transaction isolation when creating the engine, but that doesn't prevent get_isolation_level() from being called so the error persists. Is sqlalchemy not compatible with python 3.9.6? Does this have anything to do with the mysql server version?

Problem was with the DictCursor being used in the connection. Changing _fetch_type for SELECT ##transaction_isolation to be a tuple instead of a dictionary solved the issue.

500 error when deleting entities from Apache Superset

I am stuck with a problem where when I try to delete a chart or a user or dashboard from Apache Superset throw the web UI (installed using helm on Kubernetes), the browser gets a 500 error and the logs indicated the following:
superset-prod-6785cd75df-zqp5j superset] [SQL: SELECT report_schedule.created_on AS report_schedule_created_on, report_schedule.changed_on AS report_schedule_changed_on, report_schedule.id AS report_schedule_id, report_schedule.type AS report_schedule_type, report_schedule.name AS report_schedule_name, report_schedule.description AS report_schedule_description, report_schedule.context_markdown AS report_schedule_context_markdown, report_schedule.active AS report_schedule_active, report_schedule.crontab AS report_schedule_crontab, report_schedule.sql AS report_schedule_sql, report_schedule.chart_id AS report_schedule_chart_id, report_schedule.dashboard_id AS report_schedule_dashboard_id, report_schedule.database_id AS report_schedule_database_id, report_schedule.last_eval_dttm AS report_schedule_last_eval_dttm, report_schedule.last_state AS report_schedule_last_state, report_schedule.last_value AS report_schedule_last_value, report_schedule.last_value_row_json AS report_schedule_last_value_row_json, report_schedule.validator_type AS report_schedule_validator_type, report_schedule.validator_config_json AS report_schedule_validator_config_json, report_schedule.log_retention AS report_schedule_log_retention, report_schedule.grace_period AS report_schedule_grace_period, report_schedule.working_timeout AS report_schedule_working_timeout, report_schedule.created_by_fk AS report_schedule_created_by_fk, report_schedule.changed_by_fk AS report_schedule_changed_by_fk
FROM report_schedule
WHERE report_schedule.chart_id = %(chart_id_1)s]
[parameters: {'chart_id_1': '51'}]
(Background on this error at: http://sqlalche.me/e/13/f405)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column report_schedule.working_timeout does not exist
LINE 1: ...ule.grace_period AS report_schedule_grace_period, report_sch...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/api/__init__.py", line 84, in wraps
return f(self, *args, **kwargs)
File "/app/superset/views/base_api.py", line 80, in wraps
duration, response = time_function(f, self, *args, **kwargs)
File "/app/superset/utils/core.py", line 1484, in time_function
response = func(*args, **kwargs)
File "/app/superset/utils/log.py", line 125, in wrapper
value = f(*args, **kwargs)
File "/app/superset/charts/api.py", line 383, in delete
DeleteChartCommand(g.user, pk).run()
File "/app/superset/charts/commands/delete.py", line 49, in run
self.validate()
File "/app/superset/charts/commands/delete.py", line 64, in validate
reports = ReportScheduleDAO.find_by_chart_id(self._model_id)
File "/app/superset/reports/dao.py", line 45, in find_by_chart_id
.filter(ReportSchedule.chart_id == chart_id)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3373, in all
return list(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column report_schedule.working_timeout does not exist
LINE 1: ...ule.grace_period AS report_schedule_grace_period, report_sch...
^
It seems like there is a mismatch between the backing database for Superset and the python code.

This issue seems to have been resolved after upgrading the superset docker image version to apache/superset:latest around 17 Jan 2021. Open to more in-depth answers, but felt it important to state this issue is no longer an issue for me.

ArgumentError: An error occurred parsing (locally or remotely) the arguments to datastore_v3.Put()

My app started throwing this error last night. Users are complaining the site is slow as well. Still on Python 2.5. As far as I can tell the request itself is not unusual.
Any ideas?
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/controller/frequest.py", line 57, in post
request_types[request_type]()
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/controller/frequest.py", line 78, in order
event, report, messages = Game.update(game_number, user_id, order)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/model/game.py", line 355, in update
event, report, game, universe, messages, updated, ended = db.run_in_transaction(Game.transactional_update, key_id, user_id, order)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2433, in RunInTransaction
return RunInTransactionOptions(None, function, *args, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2571, in RunInTransactionOptions
ok, result = _DoOneTry(new_connection, function, args, kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2593, in _DoOneTry
result = function(*args, **kwargs)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/model/game.py", line 348, in transactional_update
game.put()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1074, in put
return datastore.Put(self._entity, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 579, in Put
return PutAsync(entities, **kwargs).get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1577, in __put_hook
self.check_rpc_success(rpc)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1212, in check_rpc_success
rpc.check_success()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 570, in check_success
self.__rpc.CheckSuccess()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
raise self.exception
ArgumentError: An error occurred parsing (locally or remotely) the arguments to datastore_v3.Put()

Categories

Resources