I am trying to filter like this:
#models.py
class A(models:Model):
all_letters = models.ManyToManyField(Letters, related_name="all_letters")
selected_letters = models.ManyToManyField(Letters, related_name="selected_letters")
#views.py
a = A.objects.get(pk=pk)
array_one = a.all_letters.all().exclude(a.selected_letters.all())
I know you can't filter based on instances but is the only equivalent way of doing this using some sort of manual python searching?
You can filter out the items where the relation in reverse (which you also named selected_letters) do not point to a, so:
array_one = a.all_letters.exclude(selected_letters=a)
Due to an absolutely fantastic DB design, I've stumbled across an issue. I'm counting the number of reverse FK matches and ITS reverse FK matches (following the method in this SO question: Django QuerySet ordering by number of reverse ForeignKey matches). I tried doing:
assert 6 == Model.objects.annotate(counting=Count(f"blah1") + Count(f"blah1__blah2")).get(id=1).counting
But I'm getting 6 == 7 as in my query is always giving me an extra 1. Why is that?
Edit: the models:
class Model(models.Model):
...
class Blah1(models.Model):
parent = models.ForeignKey(Model, on_delete=models.CASCADE)
class Blah2(models.Model):
parent = models.ForeignKey(Blah1, on_delete=models.CASCADE)
Count('blah1') will count the same number of items as Count('blah1__blah2'), because you made a JOIN. Indeed the query looks like:
SELECT model.*, COUNT(blah1.id) + COUNT(blah2.id) AS counting
FROM model
LEFT OUTER JOIN blah1.parent = model.id
LEFT OUTER JOIN blah2.parent = blah1.id
COUNT does not care about the values, or the values being duplicates, it only will not count NULLs, but for the rest, COUNT(blah1.id) and COUNT(blah2.id) will probably be nearly identical.
You thus should count unique values for blah1:
from django.db.models import Count
Model.objects.annotate(
counting=Count('blah1', distinct=True) +
Count('blah1__blah2')
).get(id=1).counting
Here we will thus only count the distinct blah1 objects.
Simplest example:
class User(models.Model):
name = ...
class Group(models.Model):
members = models.ManyToManyField(User, through='GroupMembership')
class GroupMembership(models.Model):
user = ...
group = ...
I want to get list of Groups ordered by annotated field of members.
I'm using trigram search to filter and annotate User queryset.
To get annotated users I have something like that:
User.objects.annotate(...).annotate(similarity=...)
And now I'm trying to sort Groups queryset by Users' "similarity":
ann_users = User.objects.annotate(...).annotate(similarity=...)
qs = Group.objects.prefetch_related(Prefetch('members',
queryset=ann_users))
qs.annotate(similarity=Max('members__similarity')).order_by('similarity')
But it doesn't work, because prefetch_related does the ‘joining’ in Python; so I have the error:
"FieldError: Cannot resolve keyword 'members' into field."
I expect that you have a database function for similarity of names by trigram search and its Django binding or you create any:
from django.db.models import Max, Func, Value, Prefetch
class Similarity(Func):
function = 'SIMILARITY'
arity = 2
SEARCHED_NAME = 'searched_name'
ann_users = User.objects.annotate(similarity=Similarity('name', Value(SEARCHED_NAME)))
qs = Group.objects.prefetch_related(Prefetch('members', queryset=ann_users))
qs = qs.annotate(
similarity=Max(Similarity('members__name', Value(SEARCHED_NAME)))
).order_by('similarity')
The main query is compiled to
SELECT app_group.id, MAX(SIMILARITY(app_user.name, %s)) AS similarity
FROM app_group
LEFT OUTER JOIN app_groupmembership ON (app_group.id = app_groupmembership.group_id)
LEFT OUTER JOIN app_user ON (app_groupmembership.user_id = app_user.id)
GROUP BY app_group.id
ORDER BY similarity ASC;
-- params: ['searched_name']
It is not exactly what you want in the title, but the result is the same.
Notes: The efficiency how many times will be the SIMILARITY function evaluated depends on the database query optimizer. The query plan by EXPLAIN command will be an interesting your answer, if the original idea by raw query in some simplified case is better.
I have a model, which looks something like this:
class Agent(models.Model):
class Meta:
ordering = ['first_name', 'last_name', ]
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
... some other fields ...
class Point(models.Model):
agent = models.ForeignKey(Agent)
... some other fields ...
When I select some points I want them to be ordered by agent names, like this:
points = Point.objects.filter(...smth...).order_by(-agent)
So the resulting queryset is ordered by agent names, from Z to A
or
points = Point.objects.filter(...smth...).order_by(agent)
or even without ordering
The question is:
How to get a queryset or a list of unique agent objects from points queryset, preserving the ordering?
Preferrably, without any more order_by() statements, because the view may or may not be explicitly aware of ordering type
Something like set(points.values_list('agent', flat=True)), but with objects instead of pk-s?
You can use your own idea and nest it like so (where points is any QuerySet of Point):
Agent.objects.filter(id__in=points.values_list('agent', flat=True))
Django ORM will translate this into a single db query. This should have Agent's default order. In order to preserve the agent order in points, you could do:
[p.agent for p in points.distinct('agent')]
This workaround, obviously, is not a queryset and distinct with field arguments is only supported in Postgres.
Use Agent objects manager, and QuerySet.distinct to get unique agents:
Agent.objects.filter(point__condition=...).distinct().order_by('-pk')
NOTE: condition=... should be adjusted to point__condition=....
This should work.
Agent.objects.filter(id__in=Point.objects.values_list('agent_id', flat=True)).order_by('-first_name')
Update:
Point.objects.filter(agent_id__in=Agent.objects.values_list('id', flat=True)).distinct('agent').order_by('-agent__first_name')
I'd like to append or chain several Querysets in Django, preserving the order of each one (not the result). I'm using a third-party library to paginate the result, and it only accepts lists or querysets. I've tried these options:
Queryset join: Doesn't preserve ordering in individual querysets, so I can't use this.
result = queryset_1 | queryset_2
Using itertools: Calling list() on the chain object actually evaluates the querysets and this could cause a lot of overhead. Doesn't it?
result = list(itertools.chain(queryset_1, queryset_2))
How do you think I should go?
This solution prevents duplicates:
q1 = Q(...)
q2 = Q(...)
q3 = Q(...)
qs = (
Model.objects
.filter(q1 | q2 | q3)
.annotate(
search_type_ordering=Case(
When(q1, then=Value(2)),
When(q2, then=Value(1)),
When(q3, then=Value(0)),
default=Value(-1),
output_field=IntegerField(),
)
)
.order_by('-search_type_ordering', ...)
)
If the querysets are of different models, you have to evaluate them to lists and then you can just append:
result = list(queryset_1) + list(queryset_2)
If they are the same model, you should combine the queries using the Q object and 'order_by("queryset_1 field", "queryset_2 field")'.
The right answer largely depends on why you want to combine these and how you are going to use the results.
So, inspired by Peter's answer this is what I did in my project (Django 2.2):
from django.db import models
from .models import MyModel
# Add an extra field to each query with a constant value
queryset_0 = MyModel.objects.annotate(
qs_order=models.Value(0, models.IntegerField())
)
# Each constant should basically act as the position where we want the
# queryset to stay
queryset_1 = MyModel.objects.annotate(
qs_order=models.Value(1, models.IntegerField())
)
[...]
queryset_n = MyModel.objects.annotate(
qs_order=models.Value(n, models.IntegerField())
)
# Finally, I ordered the union result by that extra field.
union = queryset_0.union(
queryset_1,
queryset_2,
[...],
queryset_n).order_by('qs_order')
With this, I could order the resulting union as I wanted without changing any private attribute while only evaluating the querysets once.
I'm not 100% sure this solution works in every possible case, but it looks like the result is the union of two QuerySets (on the same model) preserving the order of the first one:
union = qset1.union(qset2)
union.query.extra_order_by = qset1.query.extra_order_by
union.query.order_by = qset1.query.order_by
union.query.default_ordering = qset1.query.default_ordering
union.query.get_meta().ordering = qset1.query.get_meta().ordering
I did not test it extensively, so before you use that code in production, make sure it behaves like expected.
If you need to merge two querysets into a third queryset, here is an example, using _result_cache.
model
class ImportMinAttend(models.Model):
country=models.CharField(max_length=2, blank=False, null=False)
status=models.CharField(max_length=5, blank=True, null=True, default=None)
From this model, I want to display a list of all the rows such that :
(query 1) empty status go first, ordered by countries
(query 2) non empty status go in second, ordered by countries
I want to merge query 1 and query 2.
#get all the objects
queryset=ImportMinAttend.objects.all()
#get the first queryset
queryset_1=queryset.filter(status=None).order_by("country")
#len or anything that hits the database
len(queryset_1)
#get the second queryset
queryset_2=queryset.exclude(status=None).order_by("country")
#append the second queryset to the first one AND PRESERVE ORDER
for query in queryset_2:
queryset_1._result_cache.append(query)
#final result
queryset=queryset_1
It might not be very efficient, but it works :).
For Django 1.11 (released on April 4, 2017) use union() for this, documentation here:
https://docs.djangoproject.com/en/1.11/ref/models/querysets/#django.db.models.query.QuerySet.union
Here is the Version 2.1 link to this:
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#union
the union() function to combine multiple querysets together, rather than the or (|) operator. This avoids a very inefficient OUTER JOIN query that reads the entire table.
If two querysets has common field, you can order combined queryset by that field. Querysets are not evaluated during this operation.
For example:
class EventsHistory(models.Model):
id = models.IntegerField(primary_key=True)
event_time = models.DateTimeField()
event_id = models.IntegerField()
class EventsOperational(models.Model):
id = models.IntegerField(primary_key=True)
event_time = models.DateTimeField()
event_id = models.IntegerField()
qs1 = EventsHistory.objects.all()
qs2 = EventsOperational.objects.all()
qs_combined = qs2.union(qs1).order_by('event_time')