Problem
I'm using Alembic autogenerate to migrate some model changes. I run alembic revision/upgrade once and it properly creates my table and adds an alembic_version table to my database. When I go to run the revision/upgrade command again, it tries to recreate the table despite no changes being made to the model
alembic.command.revision(cfg, autogenerate=True)
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'alias.alias'
As you can see here it's attempting to add the table alias.alias even though it already exists in my database and was created by Alembic in the first revision/upgrade command.
Predictably, when I attempt to run the second upgrade I get the error
psycopg2.errors.DuplicateTable: relation "alias" already exists
Current setup
env.py
import sys
import os
sys.path.insert(0, '/tmp/')
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
from models.Base import Base
from models import Alias
config = context.config
fileConfig(config.config_file_name)
target_metadata = Base.metadata
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
Alias.py
from sqlalchemy import Column, Integer, String
from models.Base import Base
class Alias(Base):
__tablename__ = 'alias'
__table_args__ = {'schema': 'alias'}
id = Column(Integer, primary_key=True)
chart_config = Column(String)
name = Column(String, nullable=False, unique=True)
display_name = Column(String)
datasets = Column(String)
Base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Expected outcome
How do I get alembic to detect that the alias.alias table already exists? It should autogenerate an empty revision. The model Alias.py is completely static during my 2 runs of revision/upgrade
Solved by editing alembic's env.py to include schemas
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata,
include_schemas = True # Include this
)
with context.begin_transaction():
context.run_migrations()
Related
I have multiple schemas in my database, and several models per schema. Flask-migrate (which is Alembic) is unable to detect changes in any schema besides the public schema. Running
flask db migrate
followed by
flask db upgrade
will yield an error every time because the tables are already created. How can I configure alembic to recognize other schemas besides the public schema?
Modify your env.py file created by Alembic so that the context.configure function is called using the include_schemas=True option. Ensure that this is done in both your offline and online functions.
Here are my modified run_migrations_offline and run_migrations_online functions.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url, target_metadata=get_metadata(), literal_binds=True, include_schemas=True
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
# this callback is used to prevent an auto-migration from being generated
# when there are no changes to the schema
# reference: http://alembic.zzzcomputing.com/en/latest/cookbook.html
def process_revision_directives(context, revision, directives):
if getattr(config.cmd_opts, 'autogenerate', False):
script = directives[0]
if script.upgrade_ops.is_empty():
directives[:] = []
logger.info('No changes in schema detected.')
connectable = get_engine()
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=get_metadata(),
process_revision_directives=process_revision_directives,
**current_app.extensions['migrate'].configure_args,
include_schemas=True
)
with context.begin_transaction():
context.run_migrations()
I'd like to connect to a second external database during my migration to move some of its data into my local database. What's the best way to do this?
Once the second db has been added to the alembic context (which I am not sure of how to do), how can run SQL statements on the db during my migration?
This is what my env.py looks like right now:
from alembic import context
from sqlalchemy import engine_from_config, pool
from logging.config import fileConfig
from migration_settings import database_url
import models
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
target_metadata = models.Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = database_url or config.get_main_option("sqlalchemy.url")
context.configure(url=url, target_metadata=target_metadata, literal_binds=True, version_table_schema='my_schema')
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
config_overrides = {'url': database_url} if database_url is not None else {}
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool, **config_overrides)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
version_table_schema='my_schema'
)
connection.execute('CREATE SCHEMA IF NOT EXISTS my_schema')
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
I create a connection between the two models of the DB(sqlite) Operation and Contragent:
class Operation(db.Model):
__tablename__ = "operation"
id = db.Column(db.Integer, primary_key=True)
date_operation = db.Column(db.DateTime, index=True, nullable=True)
contragent_id = db.Column(db.Integer, db.ForeignKey('contragent.id'))# add db.ForeignKey('contragent.id')
def __repr__(self):
return '<Operation {}>'.format(self.code)
class Contragent(db.Model):
__tablename__ = "contragent"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(360))
operations = db.relationship('Operation', backref='operation', lazy='dynamic')# add this line
def __repr__(self):
return '<Contragent {}>'.format(self.name)
After I make changes to the models, I create a new database migration and
I apply changes in the database, while getting an error:
(venv) C:\Users\User\testapp>flask db upgrade
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade d0ac0532c134 -> 285707aa6265,
Operation table
ERROR [root] Error: No support for ALTER of constraints in SQLite dialect
In the run_migrations_online () method of the env.py file, I added render_as_batch = True,but the error still remains:
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
render_as_batch=True, # add this line
**current_app.extensions['migrate'].configure_args
)
After that I tried to set the value of the variable render_as_batch:
config.get_main_option('sqlalchemy.url').startswith('sqlite:///')
...
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
render_as_batch=config.get_main_option('sqlalchemy.url').startswith('sqlite:///'), # add this line
**current_app.extensions['migrate'].configure_args
)
....
The error still happens!
Why does the error occur even after setting the variable render_as_batch with the recommended values and what are the options to make changes to the database model?
When I add a new column (nullable=False) to an existing table, I will need to update the migration revision file by manually to first add the column with nullable=True, and then update all existing records to set the column, after that alter the column to nullable=False. But I encountered an error of "ALTER": syntax error.
Here is the test script (test.py):
#!/usr/bin/env python
import os
import flask_migrate
from flask import Flask
from flask_script import Manager
from flask_sqlalchemy import SQLAlchemy
basedir = os.path.abspath(os.path.dirname(__file__))
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] =\
'sqlite:///' + os.path.join(basedir, 'data.sqlite')
app.config['SQLALCHEMY_COMMIT_ON_TEARDOWN'] = True
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
manager = Manager(app)
db = SQLAlchemy(app)
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
migrate = flask_migrate.Migrate(app, db)
manager.add_command('db', flask_migrate.MigrateCommand)
if __name__ == '__main__':
manager.run()
First I initialized the db and upgrade it to latest:
$ python test.py db init && python test.py db migrate && python test.py db upgrade
Creating directory /tmp/test/migrations ... done
Creating directory /tmp/test/migrations/versions ... done
Generating /tmp/test/migrations/env.pyc ... done
Generating /tmp/test/migrations/script.py.mako ... done
Generating /tmp/test/migrations/env.py ... done
Generating /tmp/test/migrations/alembic.ini ... done
Generating /tmp/test/migrations/README ... done
Please edit configuration/connection/logging settings in '/tmp/test/migrations/alembic.ini' before proceeding.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'users'
Generating /tmp/test/migrations/versions/86805d015930_.py ... done
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 86805d015930, empty message
Then I updated the model to add the new column 'email' which is nullable=False:
<snip>
...
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(64), nullable=False) # this is the new column
Then generate the migration revision file:
$ python test.py db migrate -m "add name"
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added column 'users.name'
Generating /tmp/test/migrations/versions/c89371227a53_add_name.py ... done
Since the name column is non-nullable, need to update the migration file by manual, update it as bellow:
$ cat /tmp/test/migrations/versions/c89371227a53_add_name.py
from alembic import op
import sqlalchemy as sa
revision = 'c89371227a53'
down_revision = '45a51b6df68c'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('users', sa.Column('name', sa.String(length=64), nullable=True))
op.execute("""
UPDATE users SET name="foobar"
""")
op.alter_column('users', 'name', nullable=False)
def downgrade():
op.drop_column('users', 'name')
Now run the migration:
$ python test.py db upgrade
I got an error as below:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "ALTER": syntax error [SQL: u'ALTER TABLE users ALTER COLUMN name SET NOT NULL']
How can I fix this or how should I do the migration for such cases?
My env is:
Flask==0.12.1
Flask-Migrate==2.0.3
Flask-Script==2.0.5
Flask-SQLAlchemy==2.2
SQLAlchemy==1.1.9
alembic==0.9.1
Just figured out the reason, it's because I'm using sqlite and sqlite lacks of ALTER support, a workaround for this is using the batch operation migrations http://alembic.zzzcomputing.com/en/latest/batch.html
I have the following alembic migration:
revision = '535f7a49839'
down_revision = '46c675c68f4'
from alembic import op
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from datetime import datetime
Session = sessionmaker()
Base = declarative_base()
metadata = sa.MetaData()
# This table definition works
organisations = sa.Table(
'organisations',
metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('creator_id', sa.Integer),
sa.Column('creator_staff_member_id', sa.Integer),
)
"""
# This doesn't...
class organisations(Base):
__tablename__ = 'organisations'
id = sa.Column(sa.Integer, primary_key=True)
creator_id = sa.Column(sa.Integer)
creator_staff_member_id = sa.Column(sa.Integer)
"""
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session._model_changes = {} # if you are using Flask-SQLAlchemy, this works around a bug
print(session.query(organisations).all())
raise Exception("don't succeed")
def downgrade():
pass
Now the query session.query(organisations).all() works when I use the imperatively-defined table (the one not commented out). But if I use the declarative version, which as far as I understand should be equivalent, I get an error:
sqlalchemy.exc.AmbiguousForeignKeysError: Could not determine join
condition between parent/child tables on relationship
StaffMember.organisation - there are multiple foreign key paths
linking the tables. Specify the 'foreign_keys' argument, providing a
list of those columns which should be counted as containing a foreign
key reference to the parent table.
Now I understand what this error means: I have two foreign keys from organisations to staff_members in my actual models. But why does alembic care about these, and how does it even know they exist? How does this migration know that something called StaffMember exists? As far as I understand, alembic should only know about the models you explicitly tell it about in the migration.
Turns out the problem was with my Flask-script setup I was using to call alembic. The command I was using to call alembic was importing the code to initialise my Flask app which was itself importing my actual models.