I am working on implementing sets of sqlalchemy declarative models for multiple database schemas. These database schemas are highly similar in table design. I would like to be able to reuse as much of the common model code (column definitions, relations etc.) as possible across the sets of models for each schema.
(Please note, the schemas in question are 99% identical.)
As a small example:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Integer, String
Base = declarative_base()
class Person(Base):
__tablename__ = 'person'
name = Column(String, primary_key=True)
age = Column(Integer)
If I now want to introduce a new attribute Person column 'height' that only exists in one schema and not the other, how can I go about defining it whilst reusing the existing Person column definitions?
So far, I have tried using declarative mixins to define the common functionality and then mix these into classes that inherit from different schema-specific bases. However, this is unsatisfactory because even the majority of identical models have to be declared as mixins, and a declarative class declared once for schema (inheriting from the appropriate base and the mixin). Given that the schemas are similar this seems like a lot of boilerplate code.
Is there a better way to do it?
Related
My problem is as follows:
I have 3 classes that all inherent attributes from an abstract class - id, entrytime, expirytime, owner. These classes also have other attributes that are unique to themselves, so they are stored in different tables.
I wish to query over all these classes and search by the shared attributes, for example a select search where expirytime > datetime.now without having to iterate through each class and use a query.filter function.
I thought this would work by querying a class with __abstract__ = True but sqlalchemy doesn't like to query without having a table associated with said class. I have considered joined table inheritance but that seems overly complex for this since I do not need to store the abstract class in a separate table, only use it to query shared attributes.
I'm experimenting with SQLModel (https://sqlmodel.tiangolo.com/) and I get to the point that I had to create a composite index between several fields and I can't how to do it using SQLModel library.
Db Model
The only work around I found was to use directly sqlalchemy Index, rather than index=true (from SQLModel documentation when creating indexes for unique fields - )
class Jump(SQLModel, table=True):
"""
SQL Table abstraction: Jump
Contains data belonging to a connection between a questionnaire-version and another
questionnaire-version
"""
origin_name: str = Field(primary_key=True)
origin_version: int = Field()
destination_name: str = Field()
__table_args__ = (
Index(
"compound_index_origin_name_version_destination_name", "origin_name", "origin_version", "destination_name"
),
)
This is not a "workaround". This is exactly how it is supposed to be done (as of now). The idea behind SQLModel presumably is to provide a toolkit for constructing table models that is very familiar to people coming from SQLAlchemy, while also providing most of the goodies coming from Pydantic models.
In some cases, SQLModel obviously does things differently and in some regards it tries to simplify existing interfaces. E.g. providing the foreign_key parameter on the Field constructor, so that you don't need to import and instantiate ForeignKey from SQLAlchemy.
But in this case, I really don't see the point in trying to change the existing tools. SQLAlchemy declarative ORM models allow you to set composite indices and other table parameters via the __table_args__ class-attribute. SQLModel's meta class inherits this feature. So why reinvent the wheel? 🙂
Unless you have an idea how to simplify this further. In that case, I am sure Sebastián will be more than happy about a corresponding PR or a suggestion in the issue tracker.
I'd like to create a 1:n relationship between two tables dynamically. My DB model is mapped via SQLAlchemy but due to some special features of my application I can not use the default declarative way.
E.g.
class Foo(Base):
id = Column(Integer, autoincrement=True, primary_key=True)
flag = Column(Boolean)
class Bar(Base):
id = Column(Integer, autoincrement=True, primary_key=True)
foo_id = Column(Integer, ForeignKey('foo.id'))
# declarative version:
# foo = relationship(Foo)
So I want to add relationship named "foo" to the mapped class "Bar" after Bar was defined and SQLAlchemy did its job of defining a mapper etc.
Update 2017-09-05: Why is this necessary for me? (I thought I could omit this because I think it mostly distracts from the actual problem to solve but since there were comments abouts it...)
First of all I don't have a single database but hundreds/thousands. Data in old databases must not be altered in any way but I want a single source code to access old data (even though data structure and calculation rules change significantly).
Currently we use multiple model definitions. Later definitions extend/modify previous ones. Often we manipulate SQLAlchemy models dynamically. We try not to have code in mapped classes because we think it will be much harder ensuring correctness of that code after changing a table many times (code must work in every intermediate step).
In many cases we extend tables (mapped classes) programatically in model X after it was initially defined in model X-1. Adding columns to an existing SQLAlchemy ORM class is manageable. Now we are adding a new reference column an existing table and a relationship() provides a nicer Python API.
Well, my question above is again a nice example of SQLAlchemy's super powers (and my limited understanding):
Bar.__mapper__.add_property('foo', relationship('Foo'))
Likely I was unable to get this working initially because some of my surrounding code mixed adding relationships and columns. Also there is one important difference to declaring columns:
Column('foo', Integer)
For columns the first parameter can be the column name but you can not use this for relationships. relationship('foo', 'Foo') triggers exceptions when passing it to .add_property().
This is a follow-up to a post I opened regarding the creation of dynamic tables and columns using SQLALCHEMY. SQLAlchemy create dynamic tables and columns
Most of the tutorials that show how to create dynamic Tables and columns in Python and SQLAlchemy focus on creating a class object - using Declarative, such that every use of a class creates a table with the pre-defined columns and column metadata defined in that class (see below). The problem with this way of creating tables is that at least as I understand it, does not allow me to create columns dynamically as well. I am reading an API and attaining metadata for tables which can change daily, therefore I need a method of creating these tables dynamically - see my orignal post: SQLAlchemy create dynamic tables and columns
The following construct for instance does not allow for creating dynamic columns for tables, as far as I understand it, such like my original post:
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
password = Column(String)
The reason I ask about class objects in this regard, is because the tutorials that show how to query the database require the use of a class object in the query.
So for instance, every method I've researched of querying the database to find the lastest value in each table (maximum id assuming it is the primary key or maximum timestamp) seems to require the use of a class object which currently I am not doing.
Session = sessionmaker(bind=engine)
session = Session()
User.query.order_by('timestamp desc').limit(1)
Is it possible to Is it possible to construct a class object that will create dynamic columns? Using a class object ?
If not, Is there another best-practice for querying the database using SQLAlchemy to find the latest record in a table ?
See this blog post on SQLAlchemy - Great tutorial, i've read.
For example, there are such models:
class User(Base):
photo_id = Column(ForeignKey('photo.id'))
class Group(Base):
photo_id = Column(ForeignKey('photo.id'))
class Photo(Base):
__tablename__ = 'photo'
user = relationship('User', backref='photo')
group = relationship('Group', backref='photo')
But in last model relationship to User and Group is not good because in one case first relationship will be None and in other case second relationship will be None (because photo owner can be only user or group, but not both)... And if there will be more than 2 models with foreignkeys to model Photo - situation will be even worse.
How to do such relationship correct?
Thanks in advance!
If your User and Group are not stored in the same table, there is nothing wrong to defined them with two relationship. These two relationship means two different SQL query,
and you actually needs these two different query in your case.
If your User and group can be stored in the same table, you can use inheritance.
and create a relationshop to the parent table
http://docs.sqlalchemy.org/en/latest/orm/inheritance.html
or create a view for that
http://docs.sqlalchemy.org/en/rel_0_7/core/schema.html#reflecting-views
Use table inheritance: http://docs.sqlalchemy.org/en/rel_0_7/orm/extensions/declarative.html#joined-table-inheritance
I recommend this slide to you: http://www.slideshare.net/tyler4long/quickorm . It is about quick_orm, which is base on SQLAlchemy. You will see how the same problem is resolved by means of table inheritance.
Slide 7: many models should have relationship with "comments"
Slide 8: add a parent class named "Commentable" to solve the problem.
The syntax is different from SQLAlchemy, but you can get the main idea.
I do not think there is one correct way of modeling this kind of relationships. Cardinality, navigability are also facts to consider.
To a solution very similar to your modeling problem, see Generic Associations examples. The examples might look somewhat complicated at first, but if you read Mike's blog on Polymorphic Associations with SQLAlchemy it should be pretty clear what is happening there. You will end up with somewhat different model, and navigating back from Photo to the correct parent by single attribute (parent or owner) might not be achievable, but do you really need to navigate the relationship from the side of Photo?