In a Flask app that uses SQLAlchemy, I have tables for users and assets. The users belong to zero to multiple groups, and the access to each asset is allowed to zero to several of these groups. These relationships are modeled with association tables (the basic structure is shown below). I can query the database and the association tables work as intended.
The part where I am struggling is to, given a user, retrieve the assets that this user is allowed to access. I understand that I need to join on groups.
In SQL, the following statement gives the result that I need:
select * from user as u
join association_user_group as aug on u.id == aug.user_id
join association_asset_group as aag on aag.group_id = aug.group_id
where username='some_name';
However, I can't figure out how to translate this to Flask-SQLAlchemy, leveraging its benefits (which I like in many contexts).
For simplicity, let's assume the user under consideration is User.query.first() (in my code I have a reference to this object).
The basic database definition is as follows:
association_user_group = db.Table(
'association_user_group',
db.Column('user_id', db.Integer, db.ForeignKey('user.id')),
db.Column('group_id', db.Integer, db.ForeignKey('group.id'))
)
association_asset_group = db.Table(
'association_asset_group',
db.Column('asset_id', db.Integer, db.ForeignKey('asset.id')),
db.Column('group_id', db.Integer, db.ForeignKey('group.id'))
)
class User(db.Model):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
...
class Asset(db.Model):
__tablename__ = 'asset'
id = db.Column(db.Integer, primary_key=True)
...
class Group(db.Model):
__tablename__ = 'group'
id = db.Column(db.Integer, primary_key=True)
...
users = db.relationship(
'User',
secondary=association_user_group,
backref=db.backref('groups', lazy='dynamic'))
asset = db.relationship(
'Asset',
secondary=association_asset_group,
backref=db.backref('groups', lazy='dynamic'))
Update: In the meanwhile, the following kind of works, but it does not make use of the db.relationships defined in the classes, which seems like a shame:
db.session.query(association_asset_group) \
.join(association_user_group,
association_user_group.c.group_id
== association_asset_group.c.group_id) \
.filter(association_user_group.c.user_id == u.id)
This seems to do the job:
q = db.session.query(Asset) \
.join(Group, User.groups) \
.join(Asset, Group.assets) \
.filter(User.id == u.id) \
.distinct()
But I have to say that I find it difficult to grasp the glue that flask-sqlalchemy provides under the hood (and when not).
Related
I for the life of me cannot figure out why this self-referential many-to-many will not be happy:
minor_contains = db.Table(
'minor_contains',
db.Column('parent_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True),
db.Column('contains_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True))
class Minor(db.Model):
__tablename__ = 'minors'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String())
...
contains = db.relationship(
"Minor",
secondary=minor_contains,
primaryjoin="id == minor_contains.c.parent_id",
secondaryjoin="id == minor_contains.c.contains_id",
backref="contained_by",
lazy='dynamic')
I've tried reworking it a few different ways based on examples I've seen for SQLAlchemy and for Flask-SQLAlchemy, but I consistently end up where either I get the following error message or I end up in an infinite loop somewhere.
E ArgumentError: Could not locate any simple equality expressions involving locally mapped foreign key columns for primary join condition 'minor_contains.parent_id = :parent_id_1' on relationship Minor.contains. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or are annotated in the join condition with the foreign() annotation. To allow comparison operators other than '==', the relationship can be marked as viewonly=True.
UPDATE
I'm really failing to understand the error message because it shows the column from the join table being compared to itself, where the condition on the join should be the PK of the minors table compared to the FK in the join table.
I'll also add a version of this that just hangs forever. You'll see I've been reworking variable names and such just rewriting it over and over hoping if I take a fresh stab at it, I'll somehow be smarter the second or fifth time around.
minor_contains = db.Table(
'minor_contains',
db.Column('parent_minor_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True),
db.Column('contains_minor_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True))
class Minor(db.Model):
__tablename__ = 'minors'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String())
...
contains = db.relationship(
"Minor",
secondary=minor_contains,
primaryjoin=id==minor_contains.c.parent_minor_id,
secondaryjoin=id==minor_contains.c.contains_minor_id,
backref=db.backref("minor_contains", lazy='dynamic'))
I think it's necessary to specify the model name in the join conditions.
contains = db.relationship(
"Minor",
secondary=minor_contains,
primaryjoin="Minor.id == minor_contains.c.parent_id",
secondaryjoin="Minor.id == minor_contains.c.contains_id",
backref=db.backref('minor_contains', lazy='dynamic'),
lazy='dynamic')
I had the same issue and it fixed the problem. I found a useful answer here :
link
Have you tried doing this?
minor_contains = db.Table(
'minor_contains',
db.Column('parent_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True),
db.Column('contains_id', db.Integer, db.ForeignKey('minors.id'),
primary_key=True))
class Minor(db.Model):
__tablename__ = 'minors'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String())
...
contains = db.relationship(
"Minor",
secondary=minor_contains,
primaryjoin="id == minor_contains.c.parent_id",
secondaryjoin="id == minor_contains.c.contains_id",
backref=db.backref('minor_contains', lazy='dynamic'),
lazy='dynamic')
When I try to join a many-to-many table and group it by the main-id I am getting duplicates when I add the second many-to-many table.
Here is how my models look like:
Models
user
class User(UserMixin, db.Model):
id = db.Column(db.Integer, primary_key=True)
user_fistName = db.Column(db.String(64))
...
student_identifier
student_identifier = db.Table('student_identifier',
db.Column('class_id', db.Integer, db.ForeignKey('class.class_id')),
db.Column('id', db.Integer, db.ForeignKey('user.id'))
)
class
class Class(db.Model):
sqlite_autoincrement=True
class_id = db.Column(db.Integer, primary_key=True)
class_name = db.Column(db.String(128), unique=True)
mm_children = db.relationship('User', secondary=student_identifier, backref=db.backref('classes'))
class_course_identifier
class_course_identifier = db.Table('class_course_identifier',
db.Column('course_id', db.Integer, db.ForeignKey('course.course_id')),
db.Column('class_id', db.Integer, db.ForeignKey('class.class_id'))
)
database structure
Well I am using SQLAlchemy to select the desired tables with the data I want. with this session.query
db.session.query(
Class.class_id,
Class.class_name,
func.group_concat(User.user_fistName),
func.group_concat(Course.course_name)
).filter(Class.courses, User.classes).group_by(Class.class_id)
the problem is that I am getting duplicates of both the courses AND names, so if the course has two users it will print the students and the course two times.
Here is how it is looking:
wrong view
And here is how it should look:
correct view
the problem
the problem is coming when I am adding the second many-to-many table, for example users/student-identifier. If I remove the line where I "join" it, I am getting the duplicates. Is there anyway to correct this? Or should I use RAW-SQL instead(and if yes, how?)
Found out the solution, and it is quite simple.
RAW SQL
SELECT
class.class_id,
class.class_name,
GROUP_CONCAT(DISTINCT course.course_name),
GROUP_CONCAT(DISTINCT user.user_fistName)
FROM
class
JOIN class_course_identifier ON class.class_id = class_course_identifier.class_id
JOIN course ON class_course_identifier.course_id = course.course_id
JOIN student_identifier ON class.class_id = student_identifier.class_id
JOIN user ON student_identifier.id = user.id
GROUP BY class.class_id
SQLAlchemy
db.session.query(
Class.class_id,
Class.class_name,
func.group_concat(User.user_fistName.distinct()),
func.group_concat(Course.course_name.distinct())
).filter(Class.courses, User.classes).group_by(Class.class_id)
Simply add the distinct() to the desired column you want to be unique
I am using Flask-SQLAlchemy to to query my Postgres database.
I am currently trying to query for suggestions of titles with the following query:
res = Title.query.filter(Titles.name.ilike(searchstring)).limit(20)
So far so good.
Now I would like to order the results by the number of "subscribers" each Title object has.
I am aware of the following SO question: SQLAlchemy ordering by count on a many to many relationship however its solution did not work for me.
I am receiving the following error:
ProgrammingError: (ProgrammingError) column "publishers_1.id" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...itles_name, count(titles_users.user_id) AS total, publishers...
(Publisher has a one-to-many relationship with Title models)
I understand that the answer has something to do with subqueries
Below is a simplified example of my two models.
# Many-to-Many relationship for user and titles
titles_users = db.Table('titles_users',
db.Column('user_id', db.Integer, db.ForeignKey('users.id')),
db.Column('title_id', db.Integer, db.ForeignKey('titles.id'))
)
class User(db.Model, UserMixin):
__tablename__ = 'users'
# ids
id = db.Column(db.Integer, primary_key=True)
# Attributes
email = db.Column(db.String(255), unique=True)
full_name = db.Column(db.String(255))
pull_list = db.relationship(
'Title',
secondary=titles_users,
backref=db.backref('users', lazy='dynamic'),
lazy='joined'
)
class Title(db.Model):
__tablename__ = 'titles'
#: IDs
id = db.Column(db.Integer(), primary_key=True)
#: Attributes
name = db.Column(db.String(255))
Code below would work for the model you describe:
q = (db.session.query(Title, func.count(names_users.c.user_id).label("total"))
.filter(Title.name.ilike(searchstring))
.outerjoin(names_users).group_by(Title).order_by('total DESC')
)
for x in q:
print(x)
Your error, however, includes some other data like publisher or like. So if above is not helpful, you should add more information to your question.
I have two models in SQLAlchemy, having a many-to-many relationship
team_user_table = Table('team_user', Base.metadata,
Column('user_id', Integer, ForeignKey('users.id')),
Column('team_id', Integer, ForeignKey('teams.id'))
)
class User(Base):
""" The SQLAlchemy declarative model class for a User object. """
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
class Team(Base):
""" The SQLAlchemy declarative model class for a Team object. """
__tablename__ = 'teams'
id = Column(Integer, primary_key=True)
name = Column(Text, unique=True)
members = relationship("User",
secondary=team_user_table,
backref="memberteams")
I would like to query those users that are not member of a specific team.
In SQL (for example):
SELECT u.id,u.name FROM users u WHERE u.id NOT IN (SELECT tu.user_id FROM team_user tu WHERE tu.team_id=?);
How can I do this in SQLAlchemy?
This does what you want:
team_id = 1
query = session.query(User.id).filter(~User.memberteams.any(Team.id == team_id))
This is the SQL it outputs (on MySQL):
SELECT users.id AS users_id, users.name AS users_name
FROM users
WHERE NOT (EXISTS (SELECT 1
FROM team_user, teams
WHERE users.id = team_user.user_id AND teams.id = team_user.team_id AND teams.id = %s))
This is not exactly what your query looks like now, but I think this is the way it should be done. Check the docs for this.
I tested this on MySQL. However, I think it should work on any other.
To have the exact query you are looking for, you might need to look into using subqueries in combination with the filter statement. I think you will need explicit reference to the team_user table in your code to make it work.
I believe this should work and it does not use subqueries.
session.query(User).join(team_users_table).filter(team_users_table.team_id != OTHER_TEAM)
I've got a User and Group table with a many to many relationship
_usergroup_table = db.Table('usergroup_table', db.metadata,
db.Column('user_id', db.Integer, db.ForeignKey('user.id')),
db.Column('group_id', db.Integer, db.ForeignKey('group.id')))
class User(db.Model):
"""Handles the usernames, passwords and the login status"""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False, unique=True)
class Group(db.Model):
"""Used for unix-style access control."""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False)
users = db.relationship('User', secondary=_usergroup_table,
backref='groups')
Now i'd like to add a primary group to the user class. Of course I could just add a group_id column and a relationship to the Group class, but this has drawbacks. I'd like to get all groups when calling User.group, including primary_group. The primary group should always be part of the groups relationship.
Edit:
It seems the way to go is the association object
class User(db.Model, UserMixin):
"""Handles the usernames, passwords and the login status"""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False, unique=True)
primary_group = db.relationship(UserGroup,
primaryjoin="and_(User.id==UserGroup.user_id,UserGroup.primary==True)")
class Group(db.Model):
"""Used for unix-style access control."""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False)
class UserGroup(db.Model):
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
group_id = db.Column(db.Integer, db.ForeignKey('group.id'))
active = db.Column(db.Boolean, default=False)
user = db.relationship(User, backref='groups', primaryjoin=(user_id==User.id))
group = db.relationship(Group, backref='users', primaryjoin=(group_id==Group.id))
I could simplify this with the AssociationProxy, but how do I force only a single primary group per user?
The group_id approach you originally thought of has several advantages here to the "boolean flag" approach.
For one thing, it is naturally constrained so that there is only one primary group per user. For another, loading user.primary_group means the ORM can identify this related row by it's primary key, and can look locally in the identity map for it, or emit a simple SELECT by primary key, instead of emitting a query that has a hard-to-index WHERE clause with a boolean inside of it. Yet another is there's no need to get into the association object pattern which simplifies the usage of the association table and allows SQLAlchemy to handle loads and updates from/to this table more efficiently.
Below we use events, including a new version (as of 0.7.7) of #validates that catches "remove" events, to ensure object-level modifications to User.groups and User.primary_group are kept in sync. (If on an older version of 0.7 you can use the attribute "remove" event or the "AttributeExtension.remove" extension method if you're still on 0.6 or earlier). If you wanted to enforce this at the DB level you could possibly use triggers to verify the integrity you're looking for:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base= declarative_base()
_usergroup_table = Table('usergroup_table', Base.metadata,
Column('user_id', Integer, ForeignKey('user.id')),
Column('group_id', Integer, ForeignKey('group.id')))
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(60), nullable=False, unique=True)
group_id = Column(Integer, ForeignKey('group.id'), nullable=False)
primary_group = relationship("Group")
#validates('primary_group')
def _add_pg(self, key, target):
self.groups.add(target)
return target
#validates('groups', include_removes=True)
def _modify_groups(self, key, target, is_remove):
if is_remove and target is self.primary_group:
del self.primary_group
return target
class Group(Base):
__tablename__ = 'group'
id = Column(Integer, primary_key=True)
name = Column(String(60), nullable=False)
users = relationship('User', secondary=_usergroup_table,
backref=backref('groups', collection_class=set))
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
s = Session(e)
g1, g2, g3 = Group(name='g1'), Group(name='g2'), Group(name='g3')
u1 = User(name='u1', primary_group=g1)
u1.groups.update([g2, g3])
s.add_all([
g1, g2, g3, u1
])
s.commit()
u1.groups.remove(g1)
assert u1.primary_group is None
u1.primary_group = g2
s.commit()
How about a GroupMemberships model to hold the association, instead of _usergroup_table? A user could have many groups through Group Memberships, and a group membership can hold additional attributes, such as whether a given Group is the associated User's primary group.
EDIT
In order to enforce a limit of one primary group per user, I would use a validation in the User model, such that any attempt to assign more (or fewer) than one primary group would result in an error when the record is saved. I am not aware of a way of achieving the same result relying purely on the database's integrity system. There are any number of ways of coding the validation check - the documentation shows a nice approach using the validates() decorator.