I'm trying to alter a column name. First attempt was with this script:
meta = MetaData()
users = Table('users', meta,
Column('id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
def upgrade(migrate_engine):
meta.bind = migrate_engine
users.c.id.alter(name='id')
def downgrade(migrate_engine):
meta.bind = migrate_engine
users.c.id.alter(name='user_id')
Running migrate.py test on my dev database (sqlite) works and so does upgrading and downgrading. But when deploying it to my test environment on Heroku (where PostgreSQL 8.3 is used) I get a trace when I try to upgrade. Gist is this message:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "id" does not exist
I then tried to use users.c.user_idin the upgrade method. That fails in both environments.:
AttributeError: user_id
The workaround I'm using now is this script:
meta_old = MetaData()
meta_new = MetaData()
users_old = Table('users', meta_old,
Column('user_id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
users_new = Table('users', meta_new,
Column('id', Integer, primary_key=True),
Column('name', String(50), unique=True),
Column('email', String(120), unique=True)
)
def upgrade(migrate_engine):
meta_old.bind = migrate_engine
users_old.c.user_id.alter(name='id')
def downgrade(migrate_engine):
meta_new.bind = migrate_engine
users_new.c.id.alter(name='user_id')
It's already recommended practice to copy-paste the model to the sqlalchemy-migrate scripts. But this extra duplications gets a bit too much for me. Anyone knows how this should be done. Assuming it's a bug, I'd like suggestions on how to DRY up the workaround some.
This one also works:
from alembic import op
....
def upgrade(migrate_engine):
op.alter_column('users', 'user_id', new_column_name='id')
def downgrade(migrate_engine):
op.alter_column('users', 'id', new_column_name='user_id')
Turns out there's an even DRY:er solution to this than I had hoped for. Introspection! Like so:
def upgrade(migrate_engine):
meta = MetaData(bind=migrate_engine)
users = Table('users', meta, autoload=True)
users.c.user_id.alter(name='id')
def downgrade(migrate_engine):
meta = MetaData(bind=migrate_engine)
users = Table('users', meta, autoload=True)
users.c.id.alter(name='user_id')
Works like a charm!
I bet that it can't generate any SQL because your metadata references are getting mixed up. You seem to be using two different metadata objects in your Table classes, and that's really not good. You only need one. The metadata tracks stale-ness of objects, whether it needs to issue queries for object updates, foreign key constraints, etc. and it needs to know about all your tables and relationships.
Change to use a single MetaData object, and pass echo=True to your sqlalchemy.create_engine call and it will print the SQL query that it's using to standard output. Try executing that query yourself while logged in as the same role (user) to Postgres. You may find that it's a simple permissions issue.
Regarding copy-pasting: I think Django has a good convention of placing Table and declarative classes in their own module and importing them. However, because you have to pass a MetaData object to the Table factory, that complicates matters. You can use a singleton/global metadata object, or just convert to declarative.
For a while I chose to implement one-argument functions that returned Table objects given a metadata and cached the result--in effect implementing a singleton model class. Then I decided that was silly and switched to declarative.
Related
I am exporting a pandas df to postgres using the SA core. Here is the basic script:
engine = db.create_engine(f'postgresql://data-catalogue:{dbpwd}#postgres-data-catalogue-dev/data-catalogue')
metadata = db.MetaData(schema="abn")
eshc_underlyers = db.Table('eshc_underlyers', metadata,
db.Column('description', db.String),
db.Column('isin', db.String),
db.Column('ul_product', db.String, primary_key=True),
db.Column('reference_product', db.String),
db.Column('haircut_base', db.String, primary_key=True),
db.Column('base_cur',db.String),
db.Column('business_date', db.DateTime, primary_key=True),
db.Column('account', db.String, primary_key=True),
)
metadata.create_all(engine)
with engine.connect() as conn:
strUlDf.to_sql(name='eshc_underlyers', con=conn , if_exists='append', index = False)
When this runs it creates both an "abn" schema and a "public" schema, but the public one is not needed. Also when interrogating the DB the "abn" schema shows the correct composite key being applied, but the "public" schema has none applied at all. The effect of this is that I can run this same script over and over and it will ignore the constraints and allow the duplicates to load into the public schema. At the same time select * from abn.eshc_underlyers returns mothing. Alternatively if I remove schema="abn" the public default schema works correctly and constraints are observed, but ofc this is not what I need.
Not an expert in python or postgres so feeling my way a little here.
ok, figured this out. You have to pass the schema into BOTH the Metadata() statement AND the to_sql statement. In this case the public schema is not created at all.
I wrote a module which is to create an empty database file
def create_database():
engine = create_engine("sqlite:///myexample.db", echo=True)
metadata = MetaData(engine)
metadata.create_all()
But in another function, I want to open myexample.db database, and create tables to it if it doesn't already have that table.
EG of the first, subsequent table I would create would be:
Table(Variable_TableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date),
Column('Volume', Float))
(Since it is initially an empty database, it will have no tables in it, but subsequently, I can add more tables to it. Thats what i'm trying to say.)
Any suggestions?
I've managed to figure out what I intended to do. I used engine.dialect.has_table(engine, Variable_tableName) to check if the database has the table inside. IF it doesn't, then it will proceed to create a table in the database.
Sample code:
engine = create_engine("sqlite:///myexample.db") # Access the DB Engine
if not engine.dialect.has_table(engine, Variable_tableName): # If table don't exist, Create.
metadata = MetaData(engine)
# Create a table with the appropriate Columns
Table(Variable_tableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date), Column('Country', String),
Column('Brand', String), Column('Price', Float),
# Implement the creation
metadata.create_all()
This seems to be giving me what i'm looking for.
Note that in 'Base.metadata' documentation it states about create_all:
Conditional by default, will not attempt to recreate tables already
present in the target database.
And if you can see that create_all takes these arguments: create_all(self, bind=None, tables=None, checkfirst=True), and according to documentation:
Defaults to True, don't issue CREATEs for tables already present in
the target database.
So if I understand your question correctly, you can just skip the condition.
The accepted answer prints a warning that engine.dialect.has_table() is only for internal use and not part of the public API. The message suggests this as an alternative, which works for me:
import os
import sqlalchemy
# Set up a connection to a SQLite3 DB
test_db = os.getcwd() + "/test.sqlite"
db_connection_string = "sqlite:///" + test_db
engine = create_engine(db_connection_string)
# The recommended way to check for existence
sqlalchemy.inspect(engine).has_table("BOOKS")
See also the SQL Alchemy docs.
For those who define the table first in some models.table file, among other tables.
This is a code snippet for finding the class that represents the table we want to create ( so later we can use the same code to just query it )
But together with the if written above, I still run the code with checkfirst=True
ORMTable.__table__.create(bind=engine, checkfirst=True)
models.table
class TableA(Base):
class TableB(Base):
class NewTableC(Base):
id = Column('id', Text)
name = Column('name', Text)
form
Then in the form action file:
engine = create_engine("sqlite:///myexample.db")
if not engine.dialect.has_table(engine, table_name):
# Added to models.tables the new table I needed ( format Table as written above )
table_models = importlib.import_module('models.tables')
# Grab the class that represents the new table
# table_name = 'NewTableC'
ORMTable = getattr(table_models, table_name)
# checkfirst=True to make sure it doesn't exists
ORMTable.__table__.create(bind=engine, checkfirst=True)
engine.dialect.has_table does not work for me on cx_oracle.
I am getting AttributeError: 'OracleDialect_cx_oracle' object has no attribute 'default_schema_name'
I wrote a workaround function:
from sqlalchemy.engine.base import Engine
def orcl_tab_or_view_exists(in_engine: Engine, in_object: str, in_object_name: str,)-> bool:
"""Checks if Oracle table exists in current in_engine connection
in_object: 'table' | 'view'
in_object_name: table_name | view_name
"""
obj_query = """SELECT {o}_name FROM all_{o}s WHERE owner = SYS_CONTEXT ('userenv', 'current_schema') AND {o}_name = '{on}'
""".format(o=in_object, on=in_object_name.upper())
with in_engine.connect() as connection:
result = connection.execute(obj_query)
return len(list(result)) > 0
This is the code working for me to create all tables of all model classes defined with Base class
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
class YourTable(Base):
__tablename__ = 'your_table'
id = Column(Integer, primary_key = True)
DB_URL="mysql+mysqldb://<user>:<password>#<host>:<port>/<db_name>"
scoped_engine = create_engine(DB_URL)
Base = declarative_base()
Base.metadata.create_all(scoped_engine)
I have the following alembic migration:
revision = '535f7a49839'
down_revision = '46c675c68f4'
from alembic import op
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from datetime import datetime
Session = sessionmaker()
Base = declarative_base()
metadata = sa.MetaData()
# This table definition works
organisations = sa.Table(
'organisations',
metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('creator_id', sa.Integer),
sa.Column('creator_staff_member_id', sa.Integer),
)
"""
# This doesn't...
class organisations(Base):
__tablename__ = 'organisations'
id = sa.Column(sa.Integer, primary_key=True)
creator_id = sa.Column(sa.Integer)
creator_staff_member_id = sa.Column(sa.Integer)
"""
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session._model_changes = {} # if you are using Flask-SQLAlchemy, this works around a bug
print(session.query(organisations).all())
raise Exception("don't succeed")
def downgrade():
pass
Now the query session.query(organisations).all() works when I use the imperatively-defined table (the one not commented out). But if I use the declarative version, which as far as I understand should be equivalent, I get an error:
sqlalchemy.exc.AmbiguousForeignKeysError: Could not determine join
condition between parent/child tables on relationship
StaffMember.organisation - there are multiple foreign key paths
linking the tables. Specify the 'foreign_keys' argument, providing a
list of those columns which should be counted as containing a foreign
key reference to the parent table.
Now I understand what this error means: I have two foreign keys from organisations to staff_members in my actual models. But why does alembic care about these, and how does it even know they exist? How does this migration know that something called StaffMember exists? As far as I understand, alembic should only know about the models you explicitly tell it about in the migration.
Turns out the problem was with my Flask-script setup I was using to call alembic. The command I was using to call alembic was importing the code to initialise my Flask app which was itself importing my actual models.
I have a database package which contains some modules namely student_table and a db module,
in student_table I have definition it's definition like this
from sqlalchemy import Table, MetaData, String, Column, Integer
metadata = MetaData()
class User(object):
def __init__(self, user_id, name):
self.user_id = user_id
self.name = name
user_table = Table('twitter_user', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100))
)
and in db.py I have database create functions, which like this
def prepareDB():
"""
sets global variables based to access database
"""
read_settings()
engine = create_engine('mysql://'+setting_data["database_username"]+':'+setting_data["database_password"]+'#'+setting_data["database_host"]+'/'+setting_data["database_name"]+'?charset=utf8')
Session = sessionmaker(bind=engine)
global session
session = Session()
metadata.create_all(engine)
mapper(database.User, database.user_table)
my problem is metadata is required by db module to create engine and also by student module for definition however I don't see a way of doing so without creating cyclic dependency.
what can I do rectify this situation.
I have used something equivalent to this.
holder = [None]
def getmetadata():
if holder[0] is None:
holder[0] = sqlalchemy.MetaData()
return holder[0]
user_tablef = lambda metadata: Table('twitter_user', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100))
)
# ...
user_table = user_tablef(getmetadata())
But this is not the cleanest thing in the world, and it's one reason a lot of people use the declarative style.
I just moved constant to separate file , now everything works fine.
I got a weird error when I try to create a Screen object. Before It worked without any problem, but I got this error when I added a new attribute to the User class. This attribute is related to Screen in a relation many to many through user_screens. This is the error:
"InvalidRequestError: One or more mappers failed to compile. Exception was probably suppressed within a hasattr() call. Message was: One or more mappers failed to compile. Exception was probably suppressed within a hasattr() call. Message was: Class 'zeppelinlib.screen.ScreenTest.Screen' is not mapped"
These are the classes:
class Screen(rdb.Model):
"""Set up screens table in the database"""
rdb.metadata(metadata)
rdb.tablename("screens")
id = Column("id", Integer, primary_key=True)
title = Column("title", String(100))
ip = Column("ip", String(20))
...
user_screens = Table(
"user_screens",
metadata,
Column("user_id", Integer, ForeignKey("users.id")),
Column("screen_id", Integer, ForeignKey("screens.id"))
)
class User(rdb.Model):
"""Set up users table in the database"""
rdb.metadata(metadata)
rdb.tablename("users")
id = Column("id", Integer, primary_key=True)
name = Column("name", String(50))
...
group = relationship("UserGroup", uselist=False)
channels = relationship("Channel", secondary=user_channels, order_by="Channel.titleView", backref="users")
mediaGroups = relationship("MediaGroup", secondary=user_media_groups, order_by="MediaGroup.title", backref="users")
screens = relationship("Screen", secondary=user_screens, backref="users")
I might not added new relation to user because I really don't know what the problem is...
Thanks in avance!
Try to specify the primary join via the primaryjoin keyword argument on one (don't know which one) of your relationships. Sometimes (in complex relationship graphs) SQLAlchemy has a hard time to figure out how it should go about. Worked for me more than once, albeit on 0.5.x.