I am using Flask-SQLAlchemy to to query my Postgres database.
I am currently trying to query for suggestions of titles with the following query:
res = Title.query.filter(Titles.name.ilike(searchstring)).limit(20)
So far so good.
Now I would like to order the results by the number of "subscribers" each Title object has.
I am aware of the following SO question: SQLAlchemy ordering by count on a many to many relationship however its solution did not work for me.
I am receiving the following error:
ProgrammingError: (ProgrammingError) column "publishers_1.id" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...itles_name, count(titles_users.user_id) AS total, publishers...
(Publisher has a one-to-many relationship with Title models)
I understand that the answer has something to do with subqueries
Below is a simplified example of my two models.
# Many-to-Many relationship for user and titles
titles_users = db.Table('titles_users',
db.Column('user_id', db.Integer, db.ForeignKey('users.id')),
db.Column('title_id', db.Integer, db.ForeignKey('titles.id'))
)
class User(db.Model, UserMixin):
__tablename__ = 'users'
# ids
id = db.Column(db.Integer, primary_key=True)
# Attributes
email = db.Column(db.String(255), unique=True)
full_name = db.Column(db.String(255))
pull_list = db.relationship(
'Title',
secondary=titles_users,
backref=db.backref('users', lazy='dynamic'),
lazy='joined'
)
class Title(db.Model):
__tablename__ = 'titles'
#: IDs
id = db.Column(db.Integer(), primary_key=True)
#: Attributes
name = db.Column(db.String(255))
Code below would work for the model you describe:
q = (db.session.query(Title, func.count(names_users.c.user_id).label("total"))
.filter(Title.name.ilike(searchstring))
.outerjoin(names_users).group_by(Title).order_by('total DESC')
)
for x in q:
print(x)
Your error, however, includes some other data like publisher or like. So if above is not helpful, you should add more information to your question.
Related
I have following DB models
class User(Base):
__tablename__ = "user"
user_id = Column("id", Integer(), primary_key=True)
groups = relationship(
"Group", back_populates="user", lazy="selectin", cascade="all, delete-orphan",
)
class Group(Base):
__tablename__ = "group"
group_id = Column("id", Integer(), primary_key=True)
user_id = Column(
Integer,
ForeignKey("user.id", ondelete="CASCADE"),
index=True,
)
division_id = Column(
String(96),
ForeignKey("division.id", onupdate="CASCADE"),
nullable=False,
)
name = Column(String(64), nullable=False)
user = relationship("User", back_populates="groups", lazy="selectin")
group = relationship("Division", back_populates="groups", lazy="selectin")
class Division(Base):
__tablename__ = "division"
division_id = Column("id", Integer, primary_key=True)
name = Column(String(64), nullable=False)
groups = relationship("Group", back_populates="group", lazy="selectin")
I want to fetch all the users ordered by their groups(can be something else as well, need to come from enduser), which I can easily achieve using the following query
session.query(User).join(Group).join(Division).order_by(Group.name).all()
And it might look that this works just fine but it doesn't, because since a user might have multiple groups, so in order to have correct result I first need to sort the groups for each user object i.e. something like sorted(User.group.order_by(Group.name) and then apply the order_by on the User model based on these sorted groups.
And the same thing can apply to division names as well. I know that we can provide default order_by fields while defining the relationship like below but that's not what I want since the order_by field need to come from enduser and can be any other field as well.
groups = relationship("Group", back_populates="user", lazy="selectin", cascade="all, delete-orphan",order_by=("Group.name"))
I can do this at data layer in python but that would not be ideal since there is already some ordering being done at DB layer.
So how can I achieve this at DB layer using SQLAlchemy or even with raw sql. Or is it even possible with sql?
I just can not wrap my head around the Many to Many Relationships in (Flask-)SQLAlchemy or how backrefs seem to apply to my problem.
Heres what I want do achieve:
n Users each have n (predefined) Assignments to do
Each User can Submit their work (Submission - belonging to one of 8 Assignments) multiple times.
Quick Example: Dummy User has 2 Assignments (e.g. Programm a For Loop), he/she submitted 2 code snippets (each graded individually) for the first assignment and none yet for the second.
So here is what I got so far in terms of Class Definition in SQLAlchemy:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class User(db.Model):
__tablename__ = "users"
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False)
assignments = db.relationship(
"Assignment", secondary="submissions", backref=db.backref("users", lazy=True)
)
class Assignment(db.Model):
__tablename__ = "assignments"
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(40), nullable=False)
class Submission(db.Model):
__tablename__ = "submissions"
id = db.Column(db.Integer, primary_key=True)
user_id = db.Column(db.Integer, db.ForeignKey("users.id"))
assignment_id = db.Column(db.Integer, db.ForeignKey("assignments.id"))
user = db.relationship(User, backref=db.backref("submissions"))
assignment = db.relationship(
Assignment,
backref=db.backref("submissions"),
)
What I get is these warning so I think I am missing/not understanding something here:
relationship 'Assignment.submissions' will copy column assignments.id to column submissions.assignment_id, which conflicts with relationship(s): 'Assignment.users' (copies assignments.id to submissions.assignment_id), 'User.assignments' (copies assignments.id to submissions.assignment_id).
relationship 'Submission.user' will copy column users.id to column submissions.user_id, which conflicts with relationship(s): 'Assignment.users' (copies users.id to submissions.user_id), 'User.assignments' (copies users.id to submissions.user_id) etc.
Thanks in Advance for your help!
Please read a warning section under the Association Object documentation, which describes your case where you are building separate relationships (direct to/from the association table and a secondary).
Based on your model, I assume that the many-to-many will be read-only as it will not allow you to access attributes on the "Submission" anyways, and I would mark it as such:
class User(db.Model):
__tablename__ = "users"
...
assignments = db.relationship(
"Assignment",
secondary="submissions",
backref=db.backref("users", lazy=True, viewonly=True),
viewonly=True,
)
Also Association Proxy might be useful.
I have a table with some text fields and a lot of many to many relationships. I'm having trouble querying only some of the fields. Here's how I query the table:
with db.session(raise_err=True) as session:
result = session.query(
ClientKnowledge
).options(
*[defaultload(getattr(ClientKnowledge, table.table.name))
.load_only(KqFactory().get_tables([table.table.name])[0].get_pk().name)
for table in ClientKnowledge.__mapper__.relationships]
).filter(
ClientKnowledge.id == 1
)
This works fine, SQLAlchemy is doing all the joins for me and uses the lazy=joinedload param well.
When I specify fields in the query parameters I have to do all the joins myself, but I don't want that; also tried to use with_entities() and to make the query from the other side of the relationship, but got the same result.
Is there a way I can query only specific fields without losing SQLAlchemy's ability to make the joins for me?
PS: I can't really give a complete working example since all my tables are dynamically generated. Please tell me if more context is needed.
EDIT:
here is an example of how my tables looks like
#as_declarative()
class RightTable1:
id = Column(Integer, primary_key=True)
desc = Text()
#as_declarative()
class RightTable2:
id = Column(Integer, primary_key=True)
desc = Text()
#as_declarative()
class Association1:
__tablename__ = 'association1'
left_id = Column(Integer, ForeignKey(ClientKnowledge.rel_field1), primary_key=True)
right_id = Column(Integer, ForeignKey(RightTable1.id), primary_key=True)
#as_declarative()
class Association2:
__tablename__ = 'association2'
left_id = Column(Integer, ForeignKey(ClientKnowledge.rel_field2), primary_key=True)
right_id = Column(Integer, ForeignKey(RightTable2.id), primary_key=True)
#as_declarative()
class ClientKnowledge:
id = Column(Integer, primary_key=True)
text_field = Text()
rel_field1 = relationship(
kq_table, secondary=Association1.__table__, lazy='joined',
backref=backref(RightTable1.__table__.name, lazy='joined')
)
rel_field2 = relationship(
kq_table, secondary=Association2.__table__, lazy='joined',
backref=backref(RightTable2.__table__.name, lazy='joined')
)
what i want is being able to query ClientKnowledge.rel_field1.
I have tried:
session.query(ClientKnowledge.rel_field1)
But i have to do all the join myself.
When I try to join a many-to-many table and group it by the main-id I am getting duplicates when I add the second many-to-many table.
Here is how my models look like:
Models
user
class User(UserMixin, db.Model):
id = db.Column(db.Integer, primary_key=True)
user_fistName = db.Column(db.String(64))
...
student_identifier
student_identifier = db.Table('student_identifier',
db.Column('class_id', db.Integer, db.ForeignKey('class.class_id')),
db.Column('id', db.Integer, db.ForeignKey('user.id'))
)
class
class Class(db.Model):
sqlite_autoincrement=True
class_id = db.Column(db.Integer, primary_key=True)
class_name = db.Column(db.String(128), unique=True)
mm_children = db.relationship('User', secondary=student_identifier, backref=db.backref('classes'))
class_course_identifier
class_course_identifier = db.Table('class_course_identifier',
db.Column('course_id', db.Integer, db.ForeignKey('course.course_id')),
db.Column('class_id', db.Integer, db.ForeignKey('class.class_id'))
)
database structure
Well I am using SQLAlchemy to select the desired tables with the data I want. with this session.query
db.session.query(
Class.class_id,
Class.class_name,
func.group_concat(User.user_fistName),
func.group_concat(Course.course_name)
).filter(Class.courses, User.classes).group_by(Class.class_id)
the problem is that I am getting duplicates of both the courses AND names, so if the course has two users it will print the students and the course two times.
Here is how it is looking:
wrong view
And here is how it should look:
correct view
the problem
the problem is coming when I am adding the second many-to-many table, for example users/student-identifier. If I remove the line where I "join" it, I am getting the duplicates. Is there anyway to correct this? Or should I use RAW-SQL instead(and if yes, how?)
Found out the solution, and it is quite simple.
RAW SQL
SELECT
class.class_id,
class.class_name,
GROUP_CONCAT(DISTINCT course.course_name),
GROUP_CONCAT(DISTINCT user.user_fistName)
FROM
class
JOIN class_course_identifier ON class.class_id = class_course_identifier.class_id
JOIN course ON class_course_identifier.course_id = course.course_id
JOIN student_identifier ON class.class_id = student_identifier.class_id
JOIN user ON student_identifier.id = user.id
GROUP BY class.class_id
SQLAlchemy
db.session.query(
Class.class_id,
Class.class_name,
func.group_concat(User.user_fistName.distinct()),
func.group_concat(Course.course_name.distinct())
).filter(Class.courses, User.classes).group_by(Class.class_id)
Simply add the distinct() to the desired column you want to be unique
I'm trying to perform a query that works across a many->many relation ship between bmarks and tags with a secondary table of bmarks_tags. The query involves several subqueries and I have a need to DISTINCT a column. I later want to join that to another table via the DISTINCT'd ids.
I've tried it a few ways and this seems closest:
tagid = alias(Tag.tid.distinct())
test = select([bmarks_tags.c.bmark_id],
from_obj=[bmarks_tags.join(DBSession.query(tagid.label('tagid'))),
bmarks_tags.c.tag_id == tagid])
return DBSession.execute(qry)
But I get an error:
⇝ AttributeError: '_UnaryExpression' object has no attribute 'named_with_column'
Does anyone know how I can perform the join across the bmarks_tags.tag_id and the result of the Tag.tid.distinct()?
Thanks
Schema:
# this is the secondary table that ties bmarks to tags
bmarks_tags = Table('bmark_tags', Base.metadata,
Column('bmark_id', Integer, ForeignKey('bmarks.bid'), primary_key=True),
Column('tag_id', Integer, ForeignKey('tags.tid'), primary_key=True)
)
class Tag(Base):
"""Bookmarks can have many many tags"""
__tablename__ = "tags"
tid = Column(Integer, autoincrement=True, primary_key=True)
name = Column(Unicode(255), unique=True)
Something like this should work:
t = DBSession.query(Tag.tid.distinct().label('tid')).subquery('t')
test = select([bmarks_tags.c.bmark_id], bmarks_tags.c.tag_id == t.c.tid)
return DBSession.execute(test)
It is hard to tell what you are trying to accomplish, but since you are using orm anyways (and there is not much reason anymore to go with bare selects in sa these days), you should probably start by establishing a many-to-many relation:
bmarks_tags = Table('bmark_tags', Base.metadata,
Column('bmark_id', Integer, ForeignKey('bmarks.bid'), primary_key=True),
Column('tag_id', Integer, ForeignKey('tags.tid'), primary_key=True)
)
class Tag(Base):
"""Bookmarks can have many many tags"""
__tablename__ = "tags"
tid = Column(Integer, primary_key=True)
name = Column(Unicode(255), unique=True)
class BMark(Base):
__tablename__ = 'bmarks'
bid = Column(Integer, primary_key=True)
tags = relation(Tag, secondary=bmarks_tags, backref="bmarks")
Then get your query and go from there:
query = DBSession.query(BMark).join(BMark.tags)
If not, give us the actual sql you are trying to make sqlalchemy emit.