I have a flask application and trying to make it multi-tenant using multiple schemas in a single database.
When an alteration needed to the database like adding a column, adding a table, and other alterations. I need to migrate through each to schemas. I changed my migrations/env.py like below
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
engine = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool)
# schemas = set([prototype_schema,None])
connection = engine.connect()
context.configure(
connection=connection,
target_metadata=target_metadata,
include_schemas=True, #schemas,
# include_object=include_schemas([None,prototype_schema])
include_object=include_schemas([None])
)
try:
domains = ['public', 'test', 'some_schema_name']
for domain in domains:
connection.execute('set search_path to "{}", public'.format(domain))
with context.begin_transaction():
context.run_migrations()
finally:
connection.close()
The migrations are only affecting the first schema in the array. Here the public only gets migrated. I need to migrate across all schemas.
Related
I have multiple schemas in my database, and several models per schema. Flask-migrate (which is Alembic) is unable to detect changes in any schema besides the public schema. Running
flask db migrate
followed by
flask db upgrade
will yield an error every time because the tables are already created. How can I configure alembic to recognize other schemas besides the public schema?
Modify your env.py file created by Alembic so that the context.configure function is called using the include_schemas=True option. Ensure that this is done in both your offline and online functions.
Here are my modified run_migrations_offline and run_migrations_online functions.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url, target_metadata=get_metadata(), literal_binds=True, include_schemas=True
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
# this callback is used to prevent an auto-migration from being generated
# when there are no changes to the schema
# reference: http://alembic.zzzcomputing.com/en/latest/cookbook.html
def process_revision_directives(context, revision, directives):
if getattr(config.cmd_opts, 'autogenerate', False):
script = directives[0]
if script.upgrade_ops.is_empty():
directives[:] = []
logger.info('No changes in schema detected.')
connectable = get_engine()
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=get_metadata(),
process_revision_directives=process_revision_directives,
**current_app.extensions['migrate'].configure_args,
include_schemas=True
)
with context.begin_transaction():
context.run_migrations()
I am using flask_sqlalchemy to manage a database for a small web app. I have been using sqlite locally, but am now migrating to a postgresql database as it is the case in production environment.
Currently, the app has some logic where if the database does not exist, I read in some csv files and populate the tables:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
path_to_db = 'data/database.db'
db = SQLAlchemy()
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{path_to_db}'
db.init_app(app)
if not path.exists(path_to_db):
db.create_all(app)
# Read in csv files and add to database
Now I have created a postgresql database called database using pgAdmin and changed the code to:
path_to_db = 'database'
#...
app.config['SQLALCHEMY_DATABASE_URI'] = f'postgresql://postgres:postgres#localhost:5432/{path_to_db}'
# >> How do I check whether this database is empty?
# if not path.exists(path_to_db):
# Read in csv files and add to database
In the original sqlite code, it creates a database called database.db which I then check if exists. If it doesn't I create one, and read in csv files. However I am confused about on how to implement this logic with postgresql, and how to check if the database is empty.
Any suggestions?
I'd like to connect to a second external database during my migration to move some of its data into my local database. What's the best way to do this?
Once the second db has been added to the alembic context (which I am not sure of how to do), how can run SQL statements on the db during my migration?
This is what my env.py looks like right now:
from alembic import context
from sqlalchemy import engine_from_config, pool
from logging.config import fileConfig
from migration_settings import database_url
import models
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
target_metadata = models.Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = database_url or config.get_main_option("sqlalchemy.url")
context.configure(url=url, target_metadata=target_metadata, literal_binds=True, version_table_schema='my_schema')
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
config_overrides = {'url': database_url} if database_url is not None else {}
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool, **config_overrides)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
version_table_schema='my_schema'
)
connection.execute('CREATE SCHEMA IF NOT EXISTS my_schema')
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
I have push a python-django project to heroku and it works well. In my view.py file of django model, I added function that could connect to the local mysql database to retrieve data from the mysql. The function is the view.py is as followed:
#login_required
def results(request):
data=[]
data1 = []
owner = request.user
owner = str(owner)
db = MySQLdb.connect(user='root', db='aaa', passwd='xxxxx', host='localhost')
cursor = db.cursor()
cursor.execute("SELECT search_content, id, title, author, institute, FROM result_split where username = '%s'" % (owner))
data = cursor.fetchall()
db.close()
return render(request, "webdevelop/results.html", {"datas": data})
But when I try to open the page that show the data from mysql database in the deployed heroku website, it show the error:"OperationalError at /results/
(2003, "Can't connect to MySQL server on 'localhost' ([Errno 111] Connection refused)")". How could I have this heroku project to connect to my local mysql database? Or I should choose alternative?
Firstly, you need to ensure that the user and password you're using to connect to MySQL is correct and that the user has the correct privileges to work with the selected database.
Then you can check that mysql is accepting connections on localhost.
As for directly addressing the Connection Refused exception, check things like the mysql socket used to communicate with localhost applications like your Django project. The socket must exist and be configured in MySQL.
I also recommend taking a look at something like SQLAlchemy for Python which will help you interact directly with the database using Python objects. For example,
Connecting to the database:
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship, scoped_session, mapper
from config import DB_URL
"""Database Declaration"""
metadata = MetaData()
Base = declarative_base(name='Base', mapper=mapper, metadata=metadata)
engine = create_engine(DB_URL, pool_recycle=1800)
Session = sessionmaker(bind=engine, autocommit=False, autoflush=True)
session = scoped_session(Session)
You can now use session variable to perform queries and updates using its inherited functions from the SQLAlchemy Session class.
SQLAlchemy also includes a declarative model for telling Python what your tables look like. For example,
class Clinic(Base):
__tablename__ = 'clinic'
clinic_id = Column(Integer, primary_key=True)
clinic_name = Column(VARCHAR)
address = Column(VARCHAR)
city = Column(VARCHAR)
zip = Column(VARCHAR)
phone = Column(VARCHAR)
user_id = Column(VARCHAR)
These examples work well for my projects in Flask and should work well enough in Django.
In our project we have multiple databases and we use alembic for migration.
I know that alembic is supposed to be used only for database structure migration, but we also use it for data migration as it's convenient to have all database migration code in one place.
My problem is that alembic works on one database at a time. So if I have databases DB1 and DB2, alembic will first run all migrations for DB1 and after that all migrations for DB2.
The problems start when we migrate data between databases. Say, if in I'm in revision N of DB1 try to access data in DB2, the migration can fail because DB2 can be on revision zero or N-X.
Question: is it possible to run alembic migrations one by one for all databases instead of running all migrations for DB1 and then running all for DB2?
My current env.py migration function:
def run_migrations_online():
"""
for the direct-to-DB use case, start a transaction on all
engines, then run all migrations, then commit all transactions.
"""
engines = {}
for name in re.split(r',\s*', db_names):
engines[name] = rec = {}
cfg = context.config.get_section(name)
if not 'sqlalchemy.url' in cfg:
cfg['sqlalchemy.url'] = build_url(name)
rec['engine'] = engine_from_config(
cfg,
prefix='sqlalchemy.',
poolclass=pool.NullPool)
for name, rec in engines.items():
engine = rec['engine']
rec['connection'] = conn = engine.connect()
rec['transaction'] = conn.begin()
try:
for name, rec in engines.items():
logger.info("Migrating database %s" % name)
context.configure(
connection=rec['connection'],
upgrade_token="%s_upgrades" % name,
downgrade_token="%s_downgrades" % name,
target_metadata=target_metadata.get(name))
context.run_migrations(engine_name=name)
for rec in engines.values():
rec['transaction'].commit()
except:
for rec in engines.values():
rec['transaction'].rollback()
raise
finally:
for rec in engines.values():
rec['connection'].close()
While I haven't tested this myself, I have been reading https://alembic.sqlalchemy.org/en/latest/api/script.html
It seems feasible that you could use ScriptDirectory to iterate through all the revisions, check if each db needs to apply that revision, and then rather than context.run_migrations you could manually call command.upgrade(config, revision) to apply that one revision.