Develop Reference

SQLAlchemy Handling Multiple Paths In One Relationship

Please note: this question is related but separate from my other currently open question SQLAlchemy secondary join relationship on multiple foreign keys.
The SQLAlchemy documentation describes handling multiple join paths in a single class for multiple relationships:
from sqlalchemy import Integer, ForeignKey, String, Column
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
Base = declarative_base()
class Customer(Base):
__tablename__ = 'customer'
id = Column(Integer, primary_key=True)
name = Column(String)
billing_address_id = Column(Integer, ForeignKey("address.id"))
shipping_address_id = Column(Integer, ForeignKey("address.id"))
billing_address = relationship("Address")
shipping_address = relationship("Address")
class Address(Base):
__tablename__ = 'address'
id = Column(Integer, primary_key=True)
street = Column(String)
city = Column(String)
state = Column(String)
zip = Column(String)
Within the same section the documentation shows three separate ways to define the relationship:
billing_address = relationship("Address", foreign_keys=[billing_address_id])
billing_address = relationship("Address", foreign_keys="[Customer.billing_address_id]")
billing_address = relationship("Address", foreign_keys="Customer.billing_address_id")
As you can see in (1) and (2) SQLAlchemy allows you to define a list of foreign_keys. In fact, the documentation explicitly states:
In this specific example, the list is not necessary in any case as there’s only one Column we need: billing_address = relationship("Address", foreign_keys="Customer.billing_address_id")
But I cannot determine how to use the list notation to specify multiple foreign keys in a single relationship.
For the classes
class PostVersion(db.Model):
id = db.Column(db.Integer, primary_key=True)
...
tag_1_id = db.Column(db.Integer, db.ForeignKey("tag.id"))
tag_2_id = db.Column(db.Integer, db.ForeignKey("tag.id"))
tag_3_id = db.Column(db.Integer, db.ForeignKey("tag.id"))
tag_4_id = db.Column(db.Integer, db.ForeignKey("tag.id"))
tag_5_id = db.Column(db.Integer, db.ForeignKey("tag.id"))
class Tag(db.Model):
id = db.Column(db.Integer, primary_key=True)
tag = db.Column(db.String(127))
I have tried all of the following:
tags = db.relationship("Tag", foreign_keys=[tag_1_id, tag_2_id, tag_3_id, tag_4_id, tag_5_id]) resulting in
sqlalchemy.exc.AmbiguousForeignKeysError: Could not determine join condition between parent/child tables on relationship AnnotationVersion.tags - there are multiple foreign key paths linking the tables. Specify the 'foreign_keys' argument, providing a list of those columns which should be counted as containing a foreign key reference to the parent table.
tags = db.relationship("Tag", foreign_keys="[tag_1_id, tag_2_id, tag_3_id, tag_4_id, tag_5_id]") resulting in
sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper|AnnotationVersion|annotation_version, expression '[tag_1_id, tag_2_id, tag_3_id, tag_4_id, tag_5_id]' failed to locate a name ("name 'tag_1_id' is not defined"). If this is a class name, consider adding this relationship() to the class after both dependent classes have been defined.
And many others variations on the list style, using quotes inside and outside, using Table names and Class names.
I've actually solved the problem in the course of this question. Since there seems to be no direct documentation, I'll answer it myself instead of deleting this question.

The key is to define the relationship on a primary join and specify the uselist parameter.
tags = db.relationship("Tag", primaryjoin="or_(PostVersion.tag_1_id==Tag.id,"
"PostVersion.tag_2_id==Tag.id, PostVersion.tag_3_id==Tag.id,"
"PostVersion.tag_4_id==Tag.id, PostVersion.tag_5_id==Tag.id)",
uselist=True)

Many-to-many with flask and sqlalchemy in python

I am trying to do a many-to-many relationship table with more information than just the two ids. It doesnt work. I obviously standardised my table with generic names. But the use case could be like a user table , with a post table and a likes relationship table, something like that. when i do these table, python gave me that :
sqlalchemy.exc.NoForeignKeysError: Could not determine join condition between parent/child tables on relationship Table2.rela1 - there are no foreign keys linking these tables. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or specify a 'primaryjoin' expression.
So here is my relationship table
class Relationship(db.Model):
id = db.Column(db.Integer, primary_key = True)
table1_id = db.Column(db.Integer, db.ForeignKey('Table1.id')),
table2_id = db.Column(db.Integer, db.ForeignKey('Table2.id')),
value = db.Column(db.Integer), #a number
date = db.Column(db.DateTime) #actual time of the entry
here is my table1
class Table1(db.Model):
id = db.Column(db.Integer, primary_key = True)
rela1 = db.relationship('Relationship', backref = 'rel1', lazy = 'dynamic')
rela2 = db.relationship('Table2', backref = 'rel2', lazy = 'dynamic')
here is my table 2
class Table2(db.Model):
id = db.Column(db.Integer, primary_key = True)
table1_id = db.Column(db.Integer, db.ForeignKey('table1.id'))
rela1 = db.relationship('Relationship', backref = 'rel3', lazy = 'dynamic')
Thanks for helping me.
If that can be solved then my second issue is creating a function to compute the total 'value' per Table2 object, regardless of the which table1 object post the value. something like a select with sum and group by table2.id , but i dont really understand how to do it with python and flask and sqlalchemy.
Thanks.
EDIT1
Using http://docs.sqlalchemy.org/en/rel_0_9/orm/relationships.html#association-object
i changed to
class Relationship(db.Model):
__tablename__ = 'relationship'
table1_id = db.Column(db.Integer, db.ForeignKey('Table1.id'), primary_key = True),
table2_id = db.Column(db.Integer, db.ForeignKey('Table2.id'), primary_key = True),
value = db.Column(db.Integer), #a number
date = db.Column(db.DateTime) #actual time of the entry
table2obj = db.relationship("Table2", backref="table2_assocs")
then
class Table2(db.Model):
__tablename__ = 'table2'
id = db.Column(db.Integer, primary_key = True)
table1_id = db.Column(db.Integer, db.ForeignKey('table1.id'))
and table 1 is unchanged except the addition of tablename
but now i get
sqlalchemy.exc.ArgumentError: Mapper Mapper|Relationship|relatioship could not assemble any primary key columns for mapped table 'Relationship'

there are two relationship declarations on Table1, but from the code you've posted, I can see only one path to Table2, through Relationship.table2_id.
It looks like you are trying to have a many-to-many relationship with extra metadata on the intermediate relation. This is called the association object pattern in the sqlalchemy docs. For that, everything you've got so far, except the Table1.rela2 is good, you pretty well have it. That "rogue" relationship needs to be removed.
to get the convenience of having a real relationship from Table1 to Table2, you need to use an association proxy.

Long chained exists query with multiple one-to-many mappings in the chain

Edit: Following piece seems to be the right way:
session.query(User).join("userGadgets", "gadget", "components","gadgetComponentMetals")
Original:
I have the following tables configured:
class User(Base):
__tablename__ = "user"
id = Column(Integer, primary_key=True)
name = Column(String)
class Gadget(Base):
__tablename__ = "gadget"
id = Column(Integer, primary_key=True)
brand = Column(String)
class UserGadget(Base):
__tablename__ = "user_gadget"
user_id = Column(Integer, ForeignKey('user.id'), primary_key=True)
gadget_id = Column(Integer, ForeignKey('gadget.id'), primary_key=True)
user = relationship("User", backref=backref('userGadgets', order_by=user_id))
gadget = relationship("Gadget", backref=backref('userGadgets', order_by=gadget_id))
class GadgetComponent(Base):
__tablename__ = "gadget_component"
id = Column(String, primary_key=True)
gadget_id = Column(Integer,ForeignKey('gadget.id'))
component_maker = Column(String)
host = relationship("Gadget", backref=backref('components', order_by=id))
class ComponentUsingMetal(Base):
__tablename__ = "component_metal"
id = Column(Integer, primary_key=True)
component_id = Column(Integer, ForeignKey('GadgetComponent.id'))
metal = Column(String)
component = relationship("GadgetComponent", backref=backref('gadgetComponentMetals', order_by=id))
I want to find all user names for users who own gadgets having at least one component containing some kind of metal. SQL query for this will be something along the lines of:
SELECT distinct u.name FROM user u join user_gadget ug on (u.id = ug.user_id) join gadget_component gc on (ug.gadget_id = gc.id) join component_metal cm on (gc.id = cm.component_id) order by u.name
I have tried various versions along the line of: session.query(User).filter(User.userGadgets.any(UserGadget.gadget.components.any(GadgetComponent.gadgetComponentMetals.exists())))
I get the below error:
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with UserGadget.gadget has an attribute 'gadgetComponents'
Any ideas on what I am doing wrong or is there a better way to do this kind of query in SQLAlchemy?

the join() is the better way to go here since any() is going to produce lots of expensive nested subqueries. but the mistake you made with the "any" is using syntax like: UserGadget.gadget.components. SQLAlchemy doesn't continue the namespace of attributes in a series like that, e.g. there is no UserGadget.gadget.components; there is just UserGadget.gadget and Gadget.components, separately. Just like SQL won't let you say, "SELECT * from user_gadget.gadget_id.gadget.component_id" or something, SQLAlchemy needs you to tell it how you want to join together multiple tables that you're querying from. With the any() here it would be something like any(and_(UserGadget.gadget_id == GadgetComponent.gadget_id)), but using JOIN is better in any case.

Filtering SQLAlchemy query on attribute_mapped_collection field of relationship

I have two classes, Tag and Hardware, defined with a simple parent-child relationship (see the full definition at the end).
Now I want to filter a query on Tag using the version field in Hardware through an attribute_mapped_collection, eg:
def get_tags(order_code=None, hardware_filters=None):
session = Session()
query = session.query(Tag)
if order_code:
query = query.filter(Tag.order_code == order_code)
if hardware_filters:
for k, v in hardware_filters.iteritems():
query = query.filter(getattr(Tag.hardware, k).version == v)
return query.all()
But I get:
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with Tag.hardware has an attribute 'baseband
The same thing happens if I strip it back by hard-coding the attribute, eg:
query.filter(Tag.hardware.baseband.version == v)
I can do it this way:
query = query.filter(Tag.hardware.any(artefact=k, version=v))
But why can't I filter directly through the attribute?
Class definitions
class Tag(Base):
__tablename__ = 'tag'
tag_id = Column(Integer, primary_key=True)
order_code = Column(String, nullable=False)
version = Column(String, nullable=False)
status = Column(String, nullable=False)
comments = Column(String)
hardware = relationship(
"Hardware",
backref="tag",
collection_class=attribute_mapped_collection('artefact'),
)
__table_args__ = (
UniqueConstraint('order_code', 'version'),
)
class Hardware(Base):
__tablename__ = 'hardware'
hardware_id = Column(Integer, primary_key=True)
tag_id = Column(String, ForeignKey('tag.tag_id'))
product_id = Column(String, nullable=True)
artefact = Column(String, nullable=False)
version = Column(String, nullable=False)

when you compose a query, we're ultimately composing SQL. What SQL do you think would be rendered from an expression such as filter(Tag.hardware.baseband)? There's no simple answer to that, and SQLAlchemy never makes any guesses as to how traversing along multiple paths like that would work. The usage of attribute_mapped_collection is only relevant to the in-Python manipulation of objects and has no effect on how SQL is rendered for this attribute. Hence you need to use a construct that maps directly to SQL, in this case the ANY() seems like a good choice.

sqlalchemy: referencing label()'d column in a filter or clauselement

I'm trying to perform a query that works across a many->many relation ship between bmarks and tags with a secondary table of bmarks_tags. The query involves several subqueries and I have a need to DISTINCT a column. I later want to join that to another table via the DISTINCT'd ids.
I've tried it a few ways and this seems closest:
tagid = alias(Tag.tid.distinct())
test = select([bmarks_tags.c.bmark_id],
from_obj=[bmarks_tags.join(DBSession.query(tagid.label('tagid'))),
bmarks_tags.c.tag_id == tagid])
return DBSession.execute(qry)
But I get an error:
⇝ AttributeError: '_UnaryExpression' object has no attribute 'named_with_column'
Does anyone know how I can perform the join across the bmarks_tags.tag_id and the result of the Tag.tid.distinct()?
Thanks
Schema:
# this is the secondary table that ties bmarks to tags
bmarks_tags = Table('bmark_tags', Base.metadata,
Column('bmark_id', Integer, ForeignKey('bmarks.bid'), primary_key=True),
Column('tag_id', Integer, ForeignKey('tags.tid'), primary_key=True)
)
class Tag(Base):
"""Bookmarks can have many many tags"""
__tablename__ = "tags"
tid = Column(Integer, autoincrement=True, primary_key=True)
name = Column(Unicode(255), unique=True)

Something like this should work:
t = DBSession.query(Tag.tid.distinct().label('tid')).subquery('t')
test = select([bmarks_tags.c.bmark_id], bmarks_tags.c.tag_id == t.c.tid)
return DBSession.execute(test)

It is hard to tell what you are trying to accomplish, but since you are using orm anyways (and there is not much reason anymore to go with bare selects in sa these days), you should probably start by establishing a many-to-many relation:
bmarks_tags = Table('bmark_tags', Base.metadata,
Column('bmark_id', Integer, ForeignKey('bmarks.bid'), primary_key=True),
Column('tag_id', Integer, ForeignKey('tags.tid'), primary_key=True)
)
class Tag(Base):
"""Bookmarks can have many many tags"""
__tablename__ = "tags"
tid = Column(Integer, primary_key=True)
name = Column(Unicode(255), unique=True)
class BMark(Base):
__tablename__ = 'bmarks'
bid = Column(Integer, primary_key=True)
tags = relation(Tag, secondary=bmarks_tags, backref="bmarks")
Then get your query and go from there:
query = DBSession.query(BMark).join(BMark.tags)
If not, give us the actual sql you are trying to make sqlalchemy emit.