I create a connection between the two models of the DB(sqlite) Operation and Contragent:
class Operation(db.Model):
__tablename__ = "operation"
id = db.Column(db.Integer, primary_key=True)
date_operation = db.Column(db.DateTime, index=True, nullable=True)
contragent_id = db.Column(db.Integer, db.ForeignKey('contragent.id'))# add db.ForeignKey('contragent.id')
def __repr__(self):
return '<Operation {}>'.format(self.code)
class Contragent(db.Model):
__tablename__ = "contragent"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(360))
operations = db.relationship('Operation', backref='operation', lazy='dynamic')# add this line
def __repr__(self):
return '<Contragent {}>'.format(self.name)
After I make changes to the models, I create a new database migration and
I apply changes in the database, while getting an error:
(venv) C:\Users\User\testapp>flask db upgrade
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade d0ac0532c134 -> 285707aa6265,
Operation table
ERROR [root] Error: No support for ALTER of constraints in SQLite dialect
In the run_migrations_online () method of the env.py file, I added render_as_batch = True,but the error still remains:
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
render_as_batch=True, # add this line
**current_app.extensions['migrate'].configure_args
)
After that I tried to set the value of the variable render_as_batch:
config.get_main_option('sqlalchemy.url').startswith('sqlite:///')
...
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
render_as_batch=config.get_main_option('sqlalchemy.url').startswith('sqlite:///'), # add this line
**current_app.extensions['migrate'].configure_args
)
....
The error still happens!
Why does the error occur even after setting the variable render_as_batch with the recommended values and what are the options to make changes to the database model?
Related
I'm having this issue, where sqlalchemy does not recognize the database, even though it is declared with declarative_base. After trying to run a simple query of session.query(AppGeofencing).all() I get sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1046, 'No database selected').
The table is declared as
Base = declarative_base()
AppBase = declarative_base(metadata=MetaData(schema='app'))
class AppGeofencing(AppBase):
__tablename__ = 'geofencing'
id = Column(INTEGER, primary_key=True, autoincrement=True)
name = Column(VARCHAR(45))
polygon = Column(Geometry('POLYGON'))
def __init__(self, name=None, polygon=None):
self.name = name
self.polygon = polygon
The case is only with this table, because I have also done similarly for other tables, and they work just fine.
After enabling the logging for sqlalchemy I can see that is does indeed create the correct query
INFO:sqlalchemy.engine.Engine:SELECT app.geofencing.id AS app_geofencing_id, app.geofencing.name AS app_geofencing_name, ST_AsEWKB(app.geofencing.polygon) AS app_geofencing_polygon
FROM app.geofencing
but somehow it cannot determine the database to use?
Does anyone have any idea, what could cause such issue?
Problem
I'm using Alembic autogenerate to migrate some model changes. I run alembic revision/upgrade once and it properly creates my table and adds an alembic_version table to my database. When I go to run the revision/upgrade command again, it tries to recreate the table despite no changes being made to the model
alembic.command.revision(cfg, autogenerate=True)
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'alias.alias'
As you can see here it's attempting to add the table alias.alias even though it already exists in my database and was created by Alembic in the first revision/upgrade command.
Predictably, when I attempt to run the second upgrade I get the error
psycopg2.errors.DuplicateTable: relation "alias" already exists
Current setup
env.py
import sys
import os
sys.path.insert(0, '/tmp/')
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
from models.Base import Base
from models import Alias
config = context.config
fileConfig(config.config_file_name)
target_metadata = Base.metadata
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
Alias.py
from sqlalchemy import Column, Integer, String
from models.Base import Base
class Alias(Base):
__tablename__ = 'alias'
__table_args__ = {'schema': 'alias'}
id = Column(Integer, primary_key=True)
chart_config = Column(String)
name = Column(String, nullable=False, unique=True)
display_name = Column(String)
datasets = Column(String)
Base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Expected outcome
How do I get alembic to detect that the alias.alias table already exists? It should autogenerate an empty revision. The model Alias.py is completely static during my 2 runs of revision/upgrade
Solved by editing alembic's env.py to include schemas
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata,
include_schemas = True # Include this
)
with context.begin_transaction():
context.run_migrations()
I'm working on a project that uses SQLite as a database and Alembic as a database migration tool. It includes spatial data and therefore, spatial extensions and geoalchemy2 are included in the project. I'm using autogenerate command and it detects some changes that don't exist in the geometry columns.
Here is the simplified structure of the project:
# Model
sqlite_naming_convention = {
"ix": "ix_%(column_0_label)s",
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(column_0_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s",
}
Metadata = MetaData(naming_convention=sqlite_naming_convention)
BaseSpatiaLite = declarative_base(metadata=Metadata)
class Geometries(BaseSpatiaLite):
__tablename__ = "Geometries"
geometry_id = Column(Integer, primary_key=True)
geometry = Column(
geoalchemy2.types.Geometry(geometry_type="GEOMETRY", srid=4326, management=True),
nullable=False,
)
name = Column(String(length=150), nullable=False)
Alembic's env.py is as follows:
# env.py
...
def run_migrations_online():
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
# Enables Spatialite extension
listen(connectable, "connect", load_spatialite)
# Creates Spatial tables if they don't exist
create_spatial_tables_for_sqlite(connectable)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
First migration script which creates the Geometry table:
...
def upgrade():
op.create_table(
"Geometries",
sa.Column("geometry_id", sa.Integer(), nullable=False),
sa.Column("geometry", geoalchemy2.types.Geometry(management=True), nullable=False),
sa.Column("name", sa.String(length=150), nullable=False),
sa.PrimaryKeyConstraint("geometry_id"),
)
def downgrade():
op.drop_table(
"Geometries",
)
After this migration script is run, the table is created correctly:
When I run autogenerate command again, it should have found no changes. However, it generates a migration script which has an arbitrary type change:
def upgrade():
with op.batch_alter_table("Geometries", schema=None) as batch_op:
batch_op.alter_column(
"geometry",
existing_type=sa.NUMERIC(),
type_=geoalchemy2.types.Geometry(srid=4326, management=True),
nullable=False,
)
def downgrade():
with op.batch_alter_table("Geometries", schema=None) as batch_op:
batch_op.alter_column(
"geometry",
existing_type=geoalchemy2.types.Geometry(srid=4326, management=True),
type_=sa.NUMERIC(),
nullable=True,
)
I know that I might set compare_type argument to False but I'd like to auto-detect the type changes. Is there any way to tell Alembic that the type of geometry column is Geometry and there is no change at all?
I found a solution. I'm sharing it here in case some other people might face this error: (https://alembic.sqlalchemy.org/en/latest/autogenerate.html#comparing-types)
It's possible to implement a custom compare_type function and use it in env.py. In my case, geometry columns were interpreted as sqlalchemy.Integer or sqalchemy.NUMERIC types. That's why I added an if clause which returns False if inspected_type is NUMERIC or Integer and metadata_type is geoalchemy2.types.Geometry.
# add it to env.py
def custom_compare_type(
context,
inspected_column,
metadata_column,
inspected_type,
metadata_type
):
# return False if the metadata_type is the same as the inspected_type
# or None to allow the default implementation to compare these
# types. a return value of True means the two types do not
# match and should result in a type change operation.
if (isinstance(inspected_type, NUMERIC) or isinstance(inspected_type, Integer)) and isinstance(
metadata_type, Geometry
):
return False
return None
When you change compare_type=True to compare_type=custom_compare_type, Alembic should drop detecting arbitrary type changes for geometry columns!
Note: Alembic still detects a nullability change but it's not related to compare_type issue.
When I add a new column (nullable=False) to an existing table, I will need to update the migration revision file by manually to first add the column with nullable=True, and then update all existing records to set the column, after that alter the column to nullable=False. But I encountered an error of "ALTER": syntax error.
Here is the test script (test.py):
#!/usr/bin/env python
import os
import flask_migrate
from flask import Flask
from flask_script import Manager
from flask_sqlalchemy import SQLAlchemy
basedir = os.path.abspath(os.path.dirname(__file__))
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] =\
'sqlite:///' + os.path.join(basedir, 'data.sqlite')
app.config['SQLALCHEMY_COMMIT_ON_TEARDOWN'] = True
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
manager = Manager(app)
db = SQLAlchemy(app)
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
migrate = flask_migrate.Migrate(app, db)
manager.add_command('db', flask_migrate.MigrateCommand)
if __name__ == '__main__':
manager.run()
First I initialized the db and upgrade it to latest:
$ python test.py db init && python test.py db migrate && python test.py db upgrade
Creating directory /tmp/test/migrations ... done
Creating directory /tmp/test/migrations/versions ... done
Generating /tmp/test/migrations/env.pyc ... done
Generating /tmp/test/migrations/script.py.mako ... done
Generating /tmp/test/migrations/env.py ... done
Generating /tmp/test/migrations/alembic.ini ... done
Generating /tmp/test/migrations/README ... done
Please edit configuration/connection/logging settings in '/tmp/test/migrations/alembic.ini' before proceeding.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'users'
Generating /tmp/test/migrations/versions/86805d015930_.py ... done
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 86805d015930, empty message
Then I updated the model to add the new column 'email' which is nullable=False:
<snip>
...
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(64), nullable=False) # this is the new column
Then generate the migration revision file:
$ python test.py db migrate -m "add name"
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added column 'users.name'
Generating /tmp/test/migrations/versions/c89371227a53_add_name.py ... done
Since the name column is non-nullable, need to update the migration file by manual, update it as bellow:
$ cat /tmp/test/migrations/versions/c89371227a53_add_name.py
from alembic import op
import sqlalchemy as sa
revision = 'c89371227a53'
down_revision = '45a51b6df68c'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('users', sa.Column('name', sa.String(length=64), nullable=True))
op.execute("""
UPDATE users SET name="foobar"
""")
op.alter_column('users', 'name', nullable=False)
def downgrade():
op.drop_column('users', 'name')
Now run the migration:
$ python test.py db upgrade
I got an error as below:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "ALTER": syntax error [SQL: u'ALTER TABLE users ALTER COLUMN name SET NOT NULL']
How can I fix this or how should I do the migration for such cases?
My env is:
Flask==0.12.1
Flask-Migrate==2.0.3
Flask-Script==2.0.5
Flask-SQLAlchemy==2.2
SQLAlchemy==1.1.9
alembic==0.9.1
Just figured out the reason, it's because I'm using sqlite and sqlite lacks of ALTER support, a workaround for this is using the batch operation migrations http://alembic.zzzcomputing.com/en/latest/batch.html
I have two tables: Eca_users and Eca_user_emails, one user can have many emails. I recive json with users and their emails. And I wont to load them into MS SQL database. Users can update their emails, so in this json I can get the same users with new (or changed) emails.
My code
# some import here
Base = declarative_base()
class Eca_users(Base):
__tablename__ = 'eca_users'
sql_id = sqlalchemy.Column(sqlalchemy.Integer(), primary_key = True)
first_id = sqlalchemy.Column(sqlalchemy.String(15))
name = sqlalchemy.Column(sqlalchemy.String(200))
main_email = sqlalchemy.Column(sqlalchemy.String(200))
user_emails = relationship("Eca_user_emails", backref=backref('eca_users'))
class Eca_user_emails(Base):
__tablename__ = 'user_emails'
sql_id = sqlalchemy.Column(sqlalchemy.Integer(), primary_key = True)
email_address = Column(String(200), nullable=False)
status = Column(String(10), nullable=False)
active = Column(DateTime, nullable=True)
sql_user_id = Column(Integer, ForeignKey('eca_users.sql_id'))
def main()
engine = sqlalchemy.create_engine('mssql+pymssql://user:pass/ECAusers?charset=utf8')
Session = sessionmaker()
Session.configure(bind = engine)
session = Session()
#then I get my json, parse it and...
query = session.query(Eca_users).filter(Eca_users.first_id == str(user_id))
if query.count() == 0:
# not interesting now
else:
for exstUser in query:
exstUser.name = name #update user info
exstUser.user_emails = [:] # empty old emails
# creating new Email obj
newEmail = Eca_user_emails(email_address = email_record['email'],
status = email_record['status'],
active = active_date)
exstUser.user_emails.append(newEmail) # and I get error here because autoflush
session.commit()
if __name__ == '__main__':
main()
Error message:
sqlalchemy.exc.IntegrityError: ...
[SQL: 'UPDATE user_emails SET sql_user_id=%(sql_user_id)s WHERE user_emails.sql_id = %(user_emails_sql_id)s'] [parameters: {'sql_user_id': None, 'user_emails_sql_id': Decimal('1')}]
Can't find any idea why this sql_user_id is None :(
When I chek exstUser and newEmail objects in debugger - it looks like everething fine. I mean all the reference is OK. The session obj and it's dirty attribute looks also OK in the debugger (sql_user_id is set for Eca_user_emails obj).
And what is most strange for me - this code worked absolutely fine when it was without a main function, just all code after the classes declaration. But after I wrote main declaration and put all code here I started to get this error.
I am completely new to Python so maybe this is one of stupid mistakes...
Any ideas how to fix it and what is the reason? Thanks for reading this :)
By the way: Python 3.4, sqlalchemy 1.0, SQL Server 2012
sql_user_id is None because by default SQLAlchemy clears out the foreign key when you delete a child object across a relationship, that is, when you clear exstUser.user_emails SQLAlchemy sets sql_user_id to None for all those instances. If you want SQLAlchemy to issue DELETEs for Eca_user_emails instances when they are detached from Eca_users, you need to add delete-orphan cascade option to the user_emails relationship. If you want SQLAlchemy to issue DELETEs for Eca_user_emails instances when a Eca_users instance is deleted, you need to add the delete cascade option to the user_emails relationship.
user_emails = relationship("Eca_user_emails", backref=backref('eca_users'), cascade="save-update, merge, delete, delete-orphan")
You can find more information about cascades in the SQLAlchemy docs