Develop Reference

Sqlalchemy AttributeError: 'NoneType' object has no attribute 'dispatch'

Hello!
When session commit, infrequently happens this error.
Interestingly, this error can occur in any part of the project and rarely comes out intermittently.
We use fastapi structure with sqlalchemy
Session create:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from app.core.config import settings
engine = create_engine(
settings.SQLALCHEMY_DATABASE_URI,
pool_size=10, # default in SQLAlchemy
max_overflow=5, # default in SQLAlchemy
connect_args={"application_name": settings.PROJECT_NAME},
echo=settings.POSTGRES_ECHO,
)
SessionLocal = scoped_session(sessionmaker(bind=engine))
from typing import Generator, List
from app.db.session import SessionLoca
def get_db() -> Generator:
db = SessionLocal()
try:
yield db
finally:
db.close()
#router.put("/")
def endpoint(
data: schemas.UpdateSchemas,
db: Session = Depends(deps.get_db),
) -> Any:
Error
Traceback (most recent call last):
File "/app/app/api/api_v1/endpoints/otp.py", line 24, in check_otp
customer = crud.customer.create(
File "/app/app/crud/base.py", line 56, in create
db.commit()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
self._transaction.commit(_to_root=self.future)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 843, in commit
self.close()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 928, in close
self.session.dispatch.after_transaction_end(self.session, self)
AttributeError: 'NoneType' object has no attribute 'dispatch'

SqlAlchemy SQL Server Ambiguous Column Name Error

I have an issue that I could not manage. I am developing an FastApi SqlAlchemy App connected to SQL Server. I have created my models and views on locally. But when I deployed an server The following error occurred. I have tried Ubuntu, Linux, Windows VMs but the situation is same.
from sqlalchemy import create_engine
from sqlalchemy import Integer, String, Text, Column, DATE
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy import select, MetaData, Table
from sqlalchemy.orm import scoped_session, sessionmaker, Query
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.dialects.mssql import pymssql
import urllib
import datetime
from sqlalchemy.sql.sqltypes import Date
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};"
"SERVER=MYSERVER"
"UID=MYUSERNAME"
"PWD=MYPASSWD"
"DATABASE=MYDB"
"Trusted_Connection=no;"
"Integrated Security=false;")
engine = create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
engine.connect()
Base = declarative_base()
metadata = Base.metadata
session = scoped_session(sessionmaker(bind=engine))
class Users(Base):
__table__ = Table('astencube_users', Base.metadata, autoload=True, autoload_with=engine)
print(session.query(*Users.__table__.columns).all())
This is a part of my app I am trying to print a table but when I try to make any query I got error below. On my local machine there is no error when I deploy any machine it appears. Also the error gives me a parameter which represents my table but when I check the other tables still problem occurs. On my local machine there is no problem. Why is it happening?
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
self.dialect.do_execute(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Ambiguous column name 'constraint_schema'. (209) (SQLExecDirectW)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "file.py", line 35, in <module>
class Users(Base):
File "file.py", line 36, in Users
__table__ = Table('astencube_users', Base.metadata, autoload=True, autoload_with=engine)
File "<string>", line 2, in __new__
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 298, in warned
return fn(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 607, in __new__
metadata._remove_table(name, schema)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 602, in __new__
table._init(name, metadata, *args, **kw)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 677, in _init
self._autoload(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 712, in _autoload
conn_insp.reflect_table(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 795, in reflect_table
self._reflect_fk(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 948, in _reflect_fk
fkeys = self.get_foreign_keys(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 564, in get_foreign_keys
return self.dialect.get_foreign_keys(
File "<string>", line 2, in get_foreign_keys
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
ret = fn(self, con, *args, **kw)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/dialects/mssql/base.py", line 2498, in wrap
return _switch_db(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/dialects/mssql/base.py", line 2522, in _switch_db
return fn(*arg, **kw)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/dialects/mssql/base.py", line 3400, in get_foreign_keys
for r in connection.execute(s).fetchall():
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1286, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
ret = self._execute_context(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
self._handle_dbapi_exception(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2023, in _handle_dbapi_exception
util.raise_(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
self.dialect.do_execute(
File "/home/ubuntu/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Ambiguous column name 'constraint_schema'. (209) (SQLExecDirectW)")
[SQL: WITH fk_info AS (
SELECT
ischema_ref_con.constraint_schema,
ischema_ref_con.constraint_name,
ischema_key_col.ordinal_position,
ischema_key_col.table_schema,
ischema_key_col.table_name,
ischema_ref_con.unique_constraint_schema,
ischema_ref_con.unique_constraint_name,
ischema_ref_con.match_option,
ischema_ref_con.update_rule,
ischema_ref_con.delete_rule,
ischema_key_col.column_name AS constrained_column
FROM
INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS ischema_ref_con
INNER JOIN
INFORMATION_SCHEMA.KEY_COLUMN_USAGE ischema_key_col ON
ischema_key_col.table_schema = ischema_ref_con.constraint_schema
AND ischema_key_col.constraint_name =
ischema_ref_con.constraint_name
WHERE ischema_key_col.table_name = CAST(? AS NVARCHAR(max))
AND ischema_key_col.table_schema = CAST(? AS NVARCHAR(max))
),
constraint_info AS (
SELECT
ischema_key_col.constraint_schema,
ischema_key_col.constraint_name,
ischema_key_col.ordinal_position,
ischema_key_col.table_schema,
ischema_key_col.table_name,
ischema_key_col.column_name
FROM
INFORMATION_SCHEMA.KEY_COLUMN_USAGE ischema_key_col
),
index_info AS (
SELECT
sys.schemas.name AS index_schema,
sys.indexes.name AS index_name,
sys.index_columns.key_ordinal AS ordinal_position,
sys.schemas.name AS table_schema,
sys.objects.name AS table_name,
sys.columns.name AS column_name
FROM
sys.indexes
INNER JOIN
sys.objects ON
sys.objects.object_id = sys.indexes.object_id
INNER JOIN
sys.schemas ON
sys.schemas.schema_id = sys.objects.schema_id
INNER JOIN
sys.index_columns ON
sys.index_columns.object_id = sys.objects.object_id
AND sys.index_columns.index_id = sys.indexes.index_id
INNER JOIN
sys.columns ON
sys.columns.object_id = sys.indexes.object_id
AND sys.columns.column_id = sys.index_columns.column_id
)
SELECT
fk_info.constraint_schema,
fk_info.constraint_name,
fk_info.ordinal_position,
fk_info.constrained_column,
constraint_info.table_schema AS referred_table_schema,
constraint_info.table_name AS referred_table_name,
constraint_info.column_name AS referred_column,
fk_info.match_option,
fk_info.update_rule,
fk_info.delete_rule
FROM
fk_info INNER JOIN constraint_info ON
constraint_info.constraint_schema =
fk_info.unique_constraint_schema
AND constraint_info.constraint_name =
fk_info.unique_constraint_name
AND constraint_info.ordinal_position = fk_info.ordinal_position
UNION
SELECT
fk_info.constraint_schema,
fk_info.constraint_name,
fk_info.ordinal_position,
fk_info.constrained_column,
index_info.table_schema AS referred_table_schema,
index_info.table_name AS referred_table_name,
index_info.column_name AS referred_column,
fk_info.match_option,
fk_info.update_rule,
fk_info.delete_rule
FROM
fk_info INNER JOIN index_info ON
index_info.index_schema = fk_info.unique_constraint_schema
AND index_info.index_name = fk_info.unique_constraint_name
AND index_info.ordinal_position = fk_info.ordinal_position
ORDER BY constraint_schema, constraint_name, ordinal_position
]
[parameters: ('astencube_users', 'dbo')]
(Background on this error at: https://sqlalche.me/e/14/f405)

Wading through your code above we find this exact error message:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Ambiguous column name 'constraint_schema'. (209) (SQLExecDirectW)")
So when we look to see where you've referenced constraint_schema in your code, there are a few places.
For most of them you have correctly used 2-part naming to avoid any ambiguity.
However your final order by clause does not:
ORDER BY constraint_schema, constraint_name, ordinal_position
P.S. the best way to debug this stuff yourself in the future is to take the SQL code and run it directly against the SQL Server i.e. eliminate Python from the equation as it just muddies the waters.

the problem was because of the SqlAlchemy version.The version in my local machine was 1.4.25 when I install newest version to any where I got the sql error shown above.So this problem was happened because of the version.Thanks.

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: mytable

Sorry for the messy code, I got a little lost in trial and error stage (I run out of coffee).
I try to build a dashboard. to track my orders, revenue etc. to do this, I created a database that include all the related data.
After that, I download a "flask dashboard template" and started editing it.
I can collect all my order data and able using it in the index.html via curly bracelets.
When I try to sum all revenue rows in IDLE;
from flask import Flask, render_template, request, jsonify
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy.sql import text
from sqlalchemy import func
engine = create_engine('sqlite:///transactions.db', echo=True)
conn = engine.connect()
global b
a = text('SELECT * FROM mytable')
b = conn.execute(a).fetchall()
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + 'transactions.db'
db = SQLAlchemy(app)
revenue = db.engine.execute('select sum(total) from mytable').scalar()
revenue = int(revenue)
print(revenue)
it works without an issue, but when I try to implement it in my route file like this;
# -*- encoding: utf-8 -*-
from app.home import blueprint
from flask import Flask, render_template, redirect, url_for
from flask_login import login_required, current_user
from app import login_manager
from jinja2 import TemplateNotFound
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy.sql import text
# custom block
from sqlalchemy import create_engine
engine = create_engine('sqlite:///transactions.db', echo=True)
conn = engine.connect()
global b
global revenue
a = text('SELECT * FROM mytable')
b = conn.execute(a).fetchall()
app = Flask(__name__)
dborder = SQLAlchemy(app)
db = SQLAlchemy(app)
#revenue
engin = create_engine('sqlite:///transactions.db', echo=True)
coni = engine.connect()
db1 = SQLAlchemy(app)
revenue = db1.engine.execute('select sum(total) from mytable').scalar()
revenue = int(revenue)
#revenue
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + 'transactions.db'
class mytable(dborder.Model):
name = dborder.Column(dborder.Integer, primary_key=True)
add = dborder.Column(dborder.String)
# end custom block
#blueprint.route('/index')
#login_required
def index():
#if not current_user.is_authenticated:
# return redirect(url_for('base_blueprint.login'))
global b
orderlist = b
return render_template('index.html', orderlist = orderlist, revenue = revenue)
#blueprint.route('/<template>')
def route_template(template):
if not current_user.is_authenticated:
return redirect(url_for('base_blueprint.login'))
try:
return render_template(template + '.html')
except TemplateNotFound:
return render_template('page-404.html'), 404
except:
return render_template('page-500.html'), 500
I'm getting an error;
2020-05-29 00:41:13,977 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2020-05-29 00:41:13,977 INFO sqlalchemy.engine.base.Engine ()
2020-05-29 00:41:13,978 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2020-05-29 00:41:13,978 INFO sqlalchemy.engine.base.Engine ()
2020-05-29 00:41:13,979 INFO sqlalchemy.engine.base.Engine SELECT * FROM mytable
2020-05-29 00:41:13,979 INFO sqlalchemy.engine.base.Engine ()
C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flask_sqlalchemy\__init__.py:808: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
warnings.warn(
C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flask_sqlalchemy\__init__.py:829: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
warnings.warn(FSADeprecationWarning(
C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flask_sqlalchemy\__init__.py:829: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
warnings.warn(FSADeprecationWarning(
C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flask_sqlalchemy\__init__.py:829: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
warnings.warn(FSADeprecationWarning(
Traceback (most recent call last):
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1283, in _execute_context
self.dialect.do_execute(
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: mytable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run.py", line 21, in <module>
app = create_app(config_mode)
File "C:\Users\Ozgur\Documents\python-colleect\admin panel\flask-material-dashboard\app\__init__.py", line 81, in create_app
register_blueprints(app)
File "C:\Users\Ozgur\Documents\python-colleect\admin panel\flask-material-dashboard\app\__init__.py", line 24, in register_blueprints
module = import_module('app.{}.routes'.format(module_name))
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Users\Ozgur\Documents\python-colleect\admin panel\flask-material-dashboard\app\home\routes.py", line 44, in <module>
revenue = db1.engine.execute('select sum(total) from mytable').scalar()
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 2244, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1012, in execute
return self._execute_text(object_, multiparams, params)
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1181, in _execute_text
ret = self._execute_context(
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1323, in _execute_context
self._handle_dbapi_exception(
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1517, in _handle_dbapi_exception
util.raise_(
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\util\compat.py", line 178, in raise_
raise exception
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\base.py", line 1283, in _execute_context
self.dialect.do_execute(
File "C:\Users\Ozgur\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sqlalchemy\engine\default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: mytable
[SQL: select sum(total) from mytable]
(Background on this error at: http://sqlalche.me/e/e3q8)

I think you have to put
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + 'transactions.db'
before
db1 = SQLAlchemy(app)
Explanation
when your code wants to execute this line:
revenue = db1.engine.execute('select sum(total) from mytable').scalar()
SQLAlchemy needs to know the database that contains mytable. So you have to specify the URI to your database before executing this line.

I don't use flask but I had the same OperationalError using sqlalchemy (ORM mode), Python 3.7.3 and sqlite3.
Perhaps consider this : https://docs.sqlalchemy.org/en/14/tutorial/metadata.html, section "Emitting DDL to the database".
In my case I wrote :
from sqlalchemy import MetaData
metadata = MetaData()
from sqlalchemy.orm import declarative_base
Base = declarative_base()
engine = create_engine("sqlite:///test.db")
(classes declarations)
metadata.create_all(engine)
Then I get an OperationalError.
I've just added "Base." to solve my problem :
Base.metadata.create_all(engine)

How to fix incorrect SQLAlchemy metadata after migration

I have defined and created a SQLite database using SQLAlchemy, and added some data. This database has been working fine. However, I made a change to a column name in one the models ("User") and migrated this change using alembic. The current model looks like this, where the old name "analogy" was changed to "metaphor":
class User(Base):
__tablename__ = "user"
id = Column(Integer, primary_key=True)
username = Column(String(32), nullable=False, index=True, unique=True)
registered_on = Column(DateTime, index=True, default=datetime.now())
metaphor = Column(String(16))
The migration seemed to go fine, and I also doublechecked using the SQLite command line to confirm that the schema actually changed.
sqlite > .schema
CREATE TABLE "user" (
id INTEGER NOT NULL,
username VARCHAR(32) NOT NULL,
registered_on DATETIME,
metaphor VARCHAR(16),
PRIMARY KEY (id),
UNIQUE (wc_id)
);
CREATE UNIQUE INDEX ix_user_username ON user (username);
CREATE INDEX ix_user_registered_on ON user (registered_on);
But now when I try to query the table in Python, I get the following error, referencing the old column name and saying that there is no such column
>>> from background.database import db_session
>>> from data.models import User
>>> users = db_session.query(User).all()
Traceback (most recent call last):
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: user.analogy
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2843, in all
return list(self)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
return self._execute_and_instances(context)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/export/scratch/pt/power/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: user.goal_period [SQL: 'SELECT user.id AS user_id, user.username AS user_username, user.registered_on AS user_registered_on, user.analogy AS user_analogy \nFROM user'] (Background on this error at: http://sqlalche.me/e/e3q8)
After tracing this error, it looks like the name change to the model is not reflected in the metadata imported from SQLAlchemy. So, SQLAlchemy is still using the metadata from the prior version of the table, while the actual database schema and the defined model have a different column name.
I've tried to delete all caches (.pyc, py_cache) to see if that was the source of the problem.
However, after doing this, there were still no changes to the metadata.
when I open a new interprer session and query the metadata, it shows the old schema, not the new one.
>>> from SQLAlchemy import MetaData
>>> metadata= MetaData()
>>> metadata.tables['user']
Table('user', MetaData(bind=None), Column('id', Integer(), table=<user>, primary_key=True, nullable=False), Column('username', String(length=32), table=<user>, nullable=False), Column('registered_on', DateTime(), table=<user>, default=ColumnDefault(datetime.datetime(2019, 6, 5, 12, 33, 57, 11763))), Column('goal_period', String(length=16), table=<user>, default=ColumnDefault('daily'))), schema=None)
Why is the metadata is not updated, and how do fix this problem?

Connection problems with SQLAlchemy and multiple processes

I'm using PostgreSQL and SQLAlchemy in a project that consists of a main process which launches child processes. All of these processes access the database via SQLAlchemy.
I'm experiencing repeatable connection failures: The first few child processes work correctly, but after a while a connection error is raised. Here's an MWCE:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import sessionmaker
DB_URL = 'postgresql://user:password#localhost/database'
Base = declarative_base()
class Dummy(Base):
__tablename__ = 'dummies'
id = Column(Integer, primary_key=True)
value = Column(Integer)
engine = None
Session = None
session = None
def init():
global engine, Session, session
engine = create_engine(DB_URL)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
def cleanup():
session.close()
engine.dispose()
def target(id):
init()
try:
dummy = session.query(Dummy).get(id)
dummy.value += 1
session.add(dummy)
session.commit()
finally:
cleanup()
def main():
init()
try:
dummy = Dummy(value=1)
session.add(dummy)
session.commit()
p = multiprocessing.Process(target=target, args=(dummy.id,))
p.start()
p.join()
session.refresh(dummy)
assert dummy.value == 2
finally:
cleanup()
if __name__ == '__main__':
i = 1
while True:
print(i)
main()
i += 1
On my system (PostgreSQL 9.6, SQLAlchemy 1.1.4, psycopg2 2.6.2, Python 2.7, Ubuntu 14.04) this yields
1
2
3
4
5
6
7
8
9
10
11
Traceback (most recent call last):
File "./fork_test.py", line 64, in <module>
main()
File "./fork_test.py", line 55, in main
session.refresh(dummy)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1422, in refresh
only_load_props=attribute_names) is None:
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident
return q.one()
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2756, in one
ret = self.one_or_none()
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in one_or_none
ret = list(self)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2797, in __iter__
return self._execute_and_instances(context)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2820, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
exc_info
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/vagrant/latest-sqlalchemy/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[SQL: 'SELECT dummies.id AS dummies_id, dummies.value AS dummies_value \nFROM dummies \nWHERE dummies.id = %(param_1)s'] [parameters: {'param_1': 11074}]
This is repeatable and always crashes at the same iteration.
I'm creating a new engine and session after the fork as recommended by the SQLAlchemy documentation and elsewhere. Interestingly, the following slightly different approach does not crash:
import contextlib
import multiprocessing
import sqlalchemy
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import sessionmaker
DB_URL = 'postgresql://user:password#localhost/database'
Base = declarative_base()
class Dummy(Base):
__tablename__ = 'dummies'
id = Column(Integer, primary_key=True)
value = Column(Integer)
#contextlib.contextmanager
def get_session():
engine = sqlalchemy.create_engine(DB_URL)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
try:
yield session
finally:
session.close()
engine.dispose()
def target(id):
with get_session() as session:
dummy = session.query(Dummy).get(id)
dummy.value += 1
session.add(dummy)
session.commit()
def main():
with get_session() as session:
dummy = Dummy(value=1)
session.add(dummy)
session.commit()
p = multiprocessing.Process(target=target, args=(dummy.id,))
p.start()
p.join()
session.refresh(dummy)
assert dummy.value == 2
if __name__ == '__main__':
i = 1
while True:
print(i)
main()
i += 1
Since the original code is more complex and cannot simply be switched over to the latter version I'd like to understand why one of these works and the other doesn't.
The only obvious difference is that the crashing code uses global variables for the engine and the session -- these are shared via copy-on-write with the child processes. However, since I reset them directly after the fork I don't understand how that could be a problem.
Update
I re-ran the two code pieces with the latest SQLAlchemy (1.1.5) using both Python 2.7 and Python 3.4. On both the results are basically as described above. However, on Python 2.7 the crash of the first code piece now happens in the 13th iteration (reproducibly) while on 3.4 it already happens in the third iteration (also reproducibly). The second code piece runs without problems on both versions. Here's the traceback from 3.4:
1
2
3
Traceback (most recent call last):
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "fork_test.py", line 64, in <module>
main()
File "fork_test.py", line 55, in main
session.refresh(dummy)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 1424, in refresh
only_load_props=attribute_names) is None:
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident
return q.one()
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2749, in one
ret = self.one_or_none()
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2719, in one_or_none
ret = list(self)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2790, in __iter__
return self._execute_and_instances(context)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2813, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
exc_info
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/vagrant/latest-sqlalchemy-3.4/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[SQL: 'SELECT dummies.id AS dummies_id, dummies.value AS dummies_value \nFROM dummies \nWHERE dummies.id = %(param_1)s'] [parameters: {'param_1': 3397}]
Here's the PostgreSQL log (it's the same for 2.7 and 3.4):
2017-01-18 10:59:36 UTC [22429-1] LOG: database system was shut down at 2017-01-18 10:59:35 UTC
2017-01-18 10:59:36 UTC [22429-2] LOG: MultiXact member wraparound protections are now enabled
2017-01-18 10:59:36 UTC [22428-1] LOG: database system is ready to accept connections
2017-01-18 10:59:36 UTC [22433-1] LOG: autovacuum launcher started
2017-01-18 10:59:36 UTC [22435-1] [unknown]#[unknown] LOG: incomplete startup packet
2017-01-18 11:00:10 UTC [22466-1] user#db LOG: SSL error: decryption failed or bad record mac
2017-01-18 11:00:10 UTC [22466-2] user#db LOG: could not receive data from client: Connection reset by peer
(Note that the message about the incomplete startup packet is harmless)

Quoting "How do I use engines / connections / sessions with Python multiprocessing, or os.fork()?" with added emphasis:
The SQLAlchemy Engine object refers to a connection pool of existing database connections. So when this object is replicated to a child process, the goal is to ensure that no database connections are carried over.
and
However, for the case of a transaction-active Session or Connection being shared, there’s no automatic fix for this; an application needs to ensure a new child process only initiate new Connection objects and transactions, as well as ORM Session objects.
The issue stems from the forked child process inheriting the live global session, which is holding on to a Connection. When target calls init, it overwrites the global references to engine and session, thus decreasing their refcounts to 0 in the child, forcing them to finalize. If you for example one way or another create another reference to the inherited session in the child, you prevent it from being cleaned up – but don't do that. After main has joined and returns to business as usual it is trying to use the now potentially finalized – or otherwise out of sync – connection. As to why this causes an error only after some amount of iterations I'm not sure.
The only way to handle this situation using globals the way you do is to
Close all sessions
Call engine.dispose()
before forking. This will prevent connections from leaking to the child. For example:
def main():
global session
init()
try:
dummy = Dummy(value=1)
session.add(dummy)
session.commit()
dummy_id = dummy.id
# Return the Connection to the pool
session.close()
# Dispose of it!
engine.dispose()
# ...or call your cleanup() function, which does the same
p = multiprocessing.Process(target=target, args=(dummy_id,))
p.start()
p.join()
# Start a new session
session = Session()
dummy = session.query(Dummy).get(dummy_id)
assert dummy.value == 2
finally:
cleanup()
Your second example does not trigger finalization in the child, and so it only seems to work, though it might be as broken as the first, as it is still inheriting a copy of the session and its connection defined locally in main.