Consider a many-to-many relationship of Things where the order of the children (or parents) matters. So the association object ("Link") has an additional property "position". When I fill the structure, I obviously can't use the "association-jumping" append() method because I need to explicitly access the Link object to set the "position" attribute. Sad but understandable.
What I'm struggling with now is to retrieve the children of a Thing in the order given by the Link's position property. Is that in any way possible? If not, the association_proxy is not of any use to me in either populating nor retrieving the data which makes me wonder what it is good for at all. I've tried to wrap my head around the use cases in the documentation but they don't seem to apply for my situation.
Here's a self-contained example:
from sqlalchemy import create_engine, CHAR, Column, Integer,\
String, ForeignKey
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Thing(Base):
__tablename__ = 'thing'
id = Column(Integer, primary_key=True)
name = Column(String(255))
up = association_proxy('coll_up', 'up')
down = association_proxy('coll_down', 'down')
class Link(Base):
__tablename__ = 'link'
id_up = Column(Integer, ForeignKey('thing.id'),
primary_key=True)
id_down = Column(Integer, ForeignKey('thing.id'),
primary_key=True)
position = Column(Integer)
up = relationship('Thing', primaryjoin=id_up == Thing.id,
backref='coll_down')
down = relationship('Thing', primaryjoin=id_down == Thing.id,
backref='coll_up')
if __name__ == '__main__':
# I know that for convenient append()ing stuff I'd need an __init__()
# method, but see comments below
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)
if __name__ == '__main__':
db = Session()
root = Thing(name='Root thing')
db.add(root)
# inserting "reversed" to have something to sort
for i in reversed(range(5)):
t = Thing(name='Thing %d' % i)
# If Link had an __init__ method I could use root.append(t), but...
db.add(t)
# ...in order to set position, I need to explicitly use the association
# object anyway
l = Link(up=root, down=t, position=i)
db.add(l)
db.commit()
# OK, so far the association_proxy construct hasn't been of any use. Let's
# see about retrieving the data...
root = db.query(Thing).filter_by(name='Root thing').one()
for thing in root.down:
print(thing.name)
# This, as expected. prints the elements in the order in which they were
# inserted (which is "reversed"). How can I sort them by the "position"
# property of the "Link" element, or is the associaton object useless in
# this scenario?
I'd suggest investigating the order_by parameter of relationships. You can can order the ORM's organization of related object by the child's properties. (Ordering will not work without session.commit()).
Assuming you want to order by Link.position:
class Link(Base):
__tablename__ = 'link'
id_up = Column(Integer, ForeignKey('thing.id'), primary_key=True)
id_down = Column(Integer, ForeignKey('thing.id'), primary_key=True)
position = Column(Integer)
# Note the syntax for order_by -- the ORM model is referred to as a string
up = relationship(
'Thing', primaryjoin=id_up == Thing.id,
order_by='Link.position',
backref='coll_down'
)
down = relationship(
'Thing',
primaryjoin=id_down == Thing.id,
order_by='Link.position',
backref='coll_up'
)
If you need more rigid ordering, meaning, ordering before the child is committed to the session, you can define an orderinglist on the relationship which "intercept" and reorganize relationships in the ORM. (Ordering will work without session.commit()).
from sqlalchemy.ext.orderinglist import ordering_list
class Link(Base):
...
up = relationship(
'Thing',
primaryjoin=id_up == Thing.id,
order_by='Link.position',
collection_class=ordering_list('position')
backref='coll_down'
)
...
As a next step, if you're looking to organize your position based on ids (perhaps position = id_up - id_down), I'd suggest looking into sqlalchemy's event API. An elegant approach would be trigger recalculation of relevant attributes based on a relevant update. (Ex: id_up is increased by 1, increase position by 1).
Related
What I'm trying to do seems simple. I'd like to have a parent Organization which has child Groups. However, one group will be the "main_group". Each Organization should have a main group. Each Group should have a reference to the Organization in which it belongs.
I've been able to create_all and use these Models but in my tests when I do a drop_all, I get a
sqlalchemy.exc.CircularDependencyError: Can't sort tables for DROP; an unresolvable foreign key dependency exists between tables: groups, organizations. Please ensure that the ForeignKey and ForeignKeyConstraint objects involved in the cycle have names so that they can be dropped using DROP CONSTRAINT.
Here is a minimum example to show what I'm trying to do. I've left some commented lines in to show what all I've tried.
from sqlalchemy.orm import relationship
from sqlalchemy.sql.schema import ForeignKey
from sqlalchemy.sql.sqltypes import Boolean, Date, Float
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Integer, ForeignKey, String, Column
Base = declarative_base()
class Organization(Base):
__tablename__ = "organizations"
id = Column(Integer(), primary_key=True)
name = Column(String(100))
### Define one-to-one with Group
main_group_id = Column(Integer, ForeignKey("groups.id"))
main_group = relationship(
"Group",
# uselist=False,
# back_populates="organization",
foreign_keys=[main_group_id],
primaryjoin="Group.id==Organization.main_group_id",
post_update=True
# backref="organization"
)
####
## Defines the one-to-many collection with Group
# groups = relationship(
# "Group",
# uselist=True,
# # foreign_keys="Group.organization_id",
# primaryjoin="Group.id==Organization.main_group_id",
# # back_populates="organization",
# )
class Group(Base):
__tablename__ = "groups"
id = Column(Integer, primary_key=True)
name = Column(String(100))
organization_id = Column(Integer, ForeignKey("organizations.id"), nullable=False)
organization = relationship(
"Organization",
uselist=False,
# backref="groups",
foreign_keys=[organization_id],
primaryjoin="Group.organization_id==Organization.id",
# primaryjoin="Group.id==Organization.main_group_id",
post_update=True,
)
from sqlalchemy import create_engine
from sqlalchemy.orm.session import sessionmaker
conn_string = "sqlite:///example.sql"
engine = create_engine(conn_string)
Base.metadata.create_all(engine) # here we create all tables
Session = sessionmaker(bind=engine)
session = Session()
new_org = Organization(name="my new org")
print(new_org)
session.add(new_org)
org = session.query(Organization).get(1)
new_group = Group(name="my main group", organization=org, organization_id=org.id)
new_org.main_group = new_group
session.commit()
Base.metadata.drop_all(engine)
From The Docs
There is actually an example pretty similar to what you want here, which uses an explicitly named foreign key, fk_favorite_entry:
rows-that-point-to-themselves-mutually-dependent-rows
That doesn't seem to fully solve the drop warning, which maybe seems to be dialect dependent. You can fully solve it with use_alter=True which is explained here:
creating-dropping-foreign-key-constraints-via-alter
Likely Solution
Best case would be to name the atypical constraint and to set use_alter=True on it as well.
class Organization(Base):
# ...
main_group_id = Column(Integer, ForeignKey("groups.id", name="fk_main_group", use_alter=True))
# ...
Consider the following example code (using SQLAlchemy 1.4):
import os
from sqlalchemy import Column, ForeignKey, Integer, String, select
from sqlalchemy.orm import backref, declarative_base, relationship
Base = declarative_base()
class Parent(Base):
__tablename__ = "parent"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String)
type = Column(String)
class Child(Base):
__tablename__ = "child"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String)
parent_id = Column(Integer, ForeignKey("parent.id"))
parent = relationship("Parent", backref=backref("children"))
def select_parent_name(statement):
return statement.join(Child.parent).add_columns(Parent.name)
def filter_by_parent_name(statement, parent_name):
return statement.join(Child.parent).where(Parent.name == parent_name)
def build_query():
statement = select(Child.id)
if os.getenv("SELECT_PARENT_NAME", True):
statement = select_parent_name(statement)
if os.getenv("FILTER_BY_PARENT", True):
statement = filter_by_parent_name(statement, "foo")
return statement
if __name__ == "__main__":
print(str(build_query()))
This produces invalid SQL, with the same JOIN clause represented twice:
SELECT child.id, parent.name
FROM child JOIN parent ON parent.id = child.parent_id JOIN parent ON parent.id = child.parent_id
WHERE parent.name = :name_1
If executed, it will result in:
(MySQLdb._exceptions.OperationalError) (1066, "Not unique table/alias: 'parent'")
It's a stripped-down trivial example, but the point I'm trying to demonstrate is that I'm building up an SQL statement by passing it around to different functions, each of which has a different responsibility and might require the addition of a JOIN which could have already been added to the statement.
Is there an easy way to suppress duplicate JOINs like this? Or to inspect the statement to see if the redundant JOIN is already present? Ideally this information would be easily determined from the statement object itself, rather than having to maintain and pass around that state separately.
In SQLAlchemy>=1.4 the joined tables can be found in statement._setup_joins:
joined_tables = [joins[0].parent.entity for joins in statement._setup_joins]
For SQLAlchemy<1.4 the joined tables can be found in statement._join_entities:
joined_tables = [mapper.class_ for mapper in statement._join_entities]
Reference for SQLAlchemy<1.4: Can I inspect a sqlalchemy query object to find the already joined tables?
We have 1 table with a large amount of data and DBA's partitioned it based on a particular parameter. This means I ended up with Employee_TX, Employee_NY kind of table names. Earlier the models.py was simple as in --
class Employee(Base):
__tablename__ = 'Employee'
name = Column...
state = Column...
Now, I don't want to create 50 new classes for the newly partitioned tables as anyways my columns are the same.
Is there a pattern where I can create a single class and then use it in query dynamically? session.query(<Tablename>).filter().all()
Maybe some kind of Factory pattern or something is what I'm looking for.
So far I've tried by running a loop as
for state in ['CA', 'TX', 'NY']:
class Employee(Base):
__qualname__ = __tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
but this doesn't work and I get a warning as - SAWarning: This declarative base already contains a class with the same class name and module name as app_models.employee, and will be replaced in the string-lookup table.
Also it can't find the generated class when I do from app_models import Employee_TX
This is a flask app with PostgreSQL as a backend and sqlalchemy is used as an ORM
Got it by creating a custom function like -
def get_model(state):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
return MyModel
And then from my services.py, I just call with get_model(TX)
Whenever you think of dynamically constructing classes think of type() with 3 arguments (see this answer for a demonstration, and the docs more generally).
In your case, it's just a matter of constructing the classes and keeping a reference to them so you can access them again later.
Here's an example:
from sqlalchemy import Column, Integer, String
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
# this produces the set of common attributes that each class should have
def attribute_factory():
return dict(
id=Column(Integer, primary_key=True),
name=Column(String, nullable=False),
state=Column(String, nullable=False),
CLASS_VAR=12345678,
)
states = ["CA", "TX", "NY"]
# here we map the state abbreviation to the generated model, notice the templated
# class and table names
model_map = {
state: type(
f"Employee_{state}",
(Base,),
dict(**attribute_factory(), __tablename__=f"Employee_{state}"),
)
for state in states
}
engine = create_engine("sqlite:///", echo=True)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(engine)
if __name__ == "__main__":
# inserts work
s = Session()
for state, model in model_map.items():
s.add(model(name="something", state=state))
s.commit()
s.close()
# queries work
s = Session()
for state, model in model_map.items():
inst = s.query(model).first()
print(inst.state, inst.CLASS_VAR)
In order to handle a growing database table, we are sharding on table name. So we could have database tables that are named like this:
table_md5one
table_md5two
table_md5three
All tables have the exact same schema.
How do we use SQLAlchemy and dynamically specify the tablename for the class that corresponds to this? Looks like the declarative_base() classes need to have tablename pre-specified.
There will eventually be too many tables to manually specify derived classes from a parent/base class. We want to be able to build a class that can have the tablename set up dynamically (maybe passed as a parameter to a function.)
OK, we went with the custom SQLAlchemy declaration rather than the declarative one.
So we create a dynamic table object like this:
from sqlalchemy import MetaData, Table, Column
def get_table_object(self, md5hash):
metadata = MetaData()
table_name = 'table_' + md5hash
table_object = Table(table_name, metadata,
Column('Column1', DATE, nullable=False),
Column('Column2', DATE, nullable=False)
)
clear_mappers()
mapper(ActualTableObject, table_object)
return ActualTableObject
Where ActualTableObject is the class mapping to the table.
In Augmenting the Base you find a way of using a custom Base class that can, for example, calculate the __tablename__ attribure dynamically:
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
The only problem here is that I don't know where your hash comes from, but this should give a good starting point.
If you require this algorithm not for all your tables but only for one you could just use the declared_attr on the table you are interested in sharding.
Because I insist to use declarative classes with their __tablename__ dynamically specified by given parameter, after days of failing with other solutions and hours of studying SQLAlchemy internals, I come up with the following solution that I believe is simple, elegant and race-condition free.
def get_model(suffix):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'table_{suffix}'.format(suffix=suffix)
id = Column(Integer, primary_key=True)
name = Column(String)
...
return MyModel
Since they have their own class_registry, you will not get that warning saying:
This declarative base already contains a class with the same class name and module name as mypackage.models.MyModel, and will be replaced in the string-lookup table.
Hence, you will not be able to reference them from other models with string lookup. However, it works perfectly fine to use these on-the-fly declared models for foreign keys as well:
ParentModel1 = get_model(123)
ParentModel2 = get_model(456)
class MyChildModel(BaseModel):
__tablename__ = 'table_child'
id = Column(Integer, primary_key=True)
name = Column(String)
parent_1_id = Column(Integer, ForeignKey(ParentModel1.id))
parent_2_id = Column(Integer, ForeignKey(ParentModel2.id))
parent_1 = relationship(ParentModel1)
parent_2 = relationship(ParentModel2)
If you only use them to query/insert/update/delete without any reference left such as foreign key reference from another table, they, their base classes and also their class_registry will be garbage collected, so no trace will be left.
you can write a function with tablename parameter and send back the class with setting appropriate attributes.
def get_class(table_name):
class GenericTable(Base):
__tablename__ = table_name
ID= Column(types.Integer, primary_key=True)
def funcation(self):
......
return GenericTable
Then you can create a table using:
get_class("test").__table__.create(bind=engine) # See sqlachemy.engine
Try this
import zlib
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, BigInteger, DateTime, String
from datetime import datetime
BASE = declarative_base()
ENTITY_CLASS_DICT = {}
class AbsShardingClass(BASE):
__abstract__ = True
def get_class_name_and_table_name(hashid):
return 'ShardingClass%s' % hashid, 'sharding_class_%s' % hashid
def get_sharding_entity_class(hashid):
"""
#param hashid: hashid
#type hashid: int
#rtype AbsClientUserAuth
"""
if hashid not in ENTITY_CLASS_DICT:
class_name, table_name = get_class_name_and_table_name(hashid)
cls = type(class_name, (AbsShardingClass,),
{'__tablename__': table_name})
ENTITY_CLASS_DICT[hashid] = cls
return ENTITY_CLASS_DICT[hashid]
cls = get_sharding_entity_class(1)
print session.query(cls).get(100)
Instead of using imperative creating Table object, you can use usual declarative_base and make a closure to set a table name as the following:
def make_class(Base, table_name):
class User(Base):
__tablename__ = table_name
id = Column(Integer, primary_key=True)
name= Column(String)
return User
Base = declarative_base()
engine = make_engine()
custom_named_usertable = make_class(Base, 'custom_name')
Base.metadata.create_all(engine)
session = make_session(engine)
new_user = custom_named_usertable(name='Adam')
session.add(new_user)
session.commit()
session.close()
engine.dispose()
just you need to create class object for Base.
from sqlalchemy.ext.declarative import declarative_base, declared_attr
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name.lower()
Base = declarative_base(cls=Base)
In declarative approach, I want to exclude one property, its working properly when my column name and property name are same. But if I give different name then its not working.
Here is my sample code.
Base = declarative_base()
class tblUser(Base):
__tablename__ = 'tblUser'
User_Id = Column('User_Id', String(100), primary_key=True)
SequenceNo = Column('Sequence_No', Integer)
FullName = Column('FullName', String(50))
__mapper_args__ = {'exclude_properties' :['Sequence_No']}
user = tblUser()
user.User_Id = '1000001'
user.SequenceNo = 101
session.add(user)
session.commit()
In the above sample I don't want the SequenceNo property to be updated in database even if I assign some value to it. So I used exclude_properties but still its updating the value in db. But if I change the property name from SequenceNo to Sequence_No (same as the column name) then its working as per the behaviour. Can anyone help me?
Thanks
Adhi
Unfortunately, __mapper_args__ is probably the wrong approach. It is intended to control the reflection of an existing database table into a mapper, not make a column 'read-only'.
I think a better approach would be to use a hybrid property:
from sqlalchemy.ext.hybrid import hybrid_property
Base = declarative_base()
class tblUser(Base):
__tablename__ = 'tblUser'
User_Id = Column('User_Id', String(100), primary_key=True)
FullName = Column('FullName', String(50))
_Sequence_No = Column('Sequence_No', Integer)
_local_Sequence_No = None
#hybrid_property
#property
def SequenceNo(self):
if self._local_Sequence_No is not None:
return self._local_Sequence_No
return self._SequenceNo
#SequenceNo.setter
def SequenceNo(self, value):
self._local_Sequence_No = value
The original Sequence_No column is available via a private attribute, and the SequenceNo property intercepts writes and stores them on the instance to be re-used later, but not written to the database.