Develop Reference

Django and use calculated values in QuerySet

I have a function to convert net to gross price like this.
taxRate is the tax value. e.g. 23, 8, 5, 0
def gross(netValue, taxRate: int, currencyPrecision=2):
if taxRate > 0:
return round(netValue * (100 + taxRate) / 100, currencyPrecision)
else:
return round(netValue, currencyPrecision)
In my model I have a class that is order items with fields: netPrice and taxId.
class OrderPosition(models.Model):
posNumber = models.AutoField(auto_created=True, primary_key=True, editable=False, blank=False)
order = models.ForeignKey(OrderHeader, on_delete=models.PROTECT)
product = models.ForeignKey(Product, on_delete=models.PROTECT)
quantity = models.DecimalField(blank=False, max_digits=3, decimal_places=0)
netPrice = MoneyField(blank=False, max_digits=6, decimal_places=2, default=Money("0", "PLN"))
taxId = models.IntegerField(blank=False, default=0)
The value of the entire order with a given ID can be calculated as follows:
def getNetValue(self):
posList = OrderPosition.objects.filter(order_id=self.orderID)
if posList:
return str(posList.aggregate(netValue=Sum(F('quantity') * F('netPrice'),
output_field=MoneyField()))['netValue'])
else:
return "0"
Question: Can I use my function "gross()" in a query to calculate the gross value of an order in a similar way to calculate the net worth?
I suppose the aggregation function is performed on the database side (mySQL in my case) and we can only use what is understandable at the database and SQL level.

Try this:
net_value = OrderPosition.objects.filter(order_id=self.order_id).aggregate(
net_value=Sum(F('quantity') * F('net_price'), output_field=MoneyField())
)['net_value']
tax_rate = OrderPosition.objects.filter(order_id=self.orderID) \
.values('taxId')[0]['taxId']
gross_value = gross(net_value, tax_rate)
we need to iterate over the order item and calculate the gross for each item like this:
total_gross_value = Money(0, "PLN")
for order_item in OrderPosition.objects.filter(order_id=self.order_id):
net_price = order_item.net_price
tax_rate = order_item.tax_id
gross_price = gross(net_price, tax_rate)
total_gross_value += gross_price
we can use annotate method on queryset like this:
from django.db.models import F, Func
class GrossValue(Func):
function = 'ROUND'
template = '%(function)s(%(expressions)s * (100 + %(tax_rate)s) / 100, 2)'
def __init__(self, expression, tax_rate, **extra):
super().__init__(expression, tax_rate=tax_rate, **extra)
OrderPosition.objects.filter(order_id=self.order_id).annotate(
gross_price=GrossValue(F('net_price'), F('tax_id'))
).aggregate(Sum('gross_price'))
have a nice day ;)

Django - Annotate with Case/When/Value and related objects

I have the two following models:
class Post(models.Model):
content = models.TextField()
class Vote(models.Model):
UP_VOTE = 0
DOWN_VOTE = 1
VOTE_TYPES = (
(UP_VOTE, "Up vote"),
(DOWN_VOTE, "Down vote"),
)
post = models.ForeignKey(Post, related_name="votes")
vote_type = models.PositiveSmallIntegerField(choices=VOTE_TYPES)
I would like to have a score property on Post that returns the sum of the values of the votes to that post, counting votes with UP_VOTE type as 1 and those with DOWN_VOTE as -1.
This is what I’ve tried:
# inside Post
#property
def score(self):
return (
self.votes.all()
.annotate(
value=Case(
When(vote_type=Vote.DOWN_VOTE, then=Value(-1)),
When(vote_type=Vote.UP_VOTE, then=Value(1)),
default=Value("0"),
output_field=models.SmallIntegerField(),
)
)
.aggregate(Sum("value"))["value__sum"]
)
However, this yields None. More specifically, without dereferencing ["value__sum"], this returns {'value__sum': None}.
Is using Case-When-Value the correct approach to my use case? If so, what’s wrong with the code I posted?

The sum of an empty set will be NULL/None by default. As of django-4.0, you can work with the default=… parameter [Django-doc]:
from django.db.models import F, Sum
#property
def score(self):
return self.votes.aggregate(total=Sum(-2*F('vote_type') + 1, default=0))['total']
Prior to django-4.0, you can work with Coalesce [Django-doc]:
from django.db.models import F, Sum, Value
from django.db.models.functions import Coalesce
#property
def score(self):
return self.votes.aggregate(
total=Coalesce(Sum(-2*F('vote_type') + 1), Value(0))
)['total']
although in this simple case, you can just replace None by 0 at the Django/Python layer:
from django.db.models import F, Sum
#property
def score(self):
return self.votes.aggregate(total=Sum(-2*F('vote_type') + 1))['total'] or 0
It might be better to use the "score" of a vote as value, so:
class Vote(models.Model):
UP_VOTE = 1
DOWN_VOTE = -1
# …
vote_type = models.SmallIntegerField(choices=VOTE_TYPES)
This will make the aggregation logic simpler, and will make it easier to later allow for example voting +5, -10, etc.

Django: Count related model where an annotation on the related has a specific value and store count in an annotation (or simply: count subquery)

I have two models Pick and GamePick. GamePick has a ForeignKey relation to Pick, which is accessible on Pick.game_picks.
I have setup GamePick with a custom queryset and manger so that when ever I retrieve a GamePick with the manager objects is is annotated with a field is_correct based on the values of other fields.
Now what I want to be able to do is count the how many correct GamePicks are pointing to a specific Pick.
One simple way is doing this with a method in Python:
class Pick(models.Model):
...
def count_correct(self):
return self.game_picks.filter(is_correct=True).count()
So far so good.
But now, I would like to annotate each Pick with that count, say as correct_count. This is so I can order the Pick with something like Pick.objects.all().order_by("correct_count").
Now how would I do this?
This is where I am:
correct_game_picks = GamePick.objects.filter(
pick=models.OuterRef("pk"),
is_correct=True
)
picks = Pick.objects.annotate(
correct_count=models.Count(correct_game_picks.values("pk"))
)
This is what pick.query gives me:
SELECT
"picks_pick"."id",
"picks_pick"."picker",
"picks_pick"."pot_id",
COUNT((
SELECT U0."id" FROM "picks_gamepick" U0
INNER JOIN "games_game" U1 ON (U0."game_id" = U1."id")
WHERE ((U0."picked_team_id" = U1."winning_team_id") AND U0."pick_id" = "picks_pick"."id")
)) AS "correct_count"
FROM "picks_pick"
GROUP BY "picks_pick"."id", "picks_pick"."picker", "picks_pick"."pot_id"
I am not good at SQL, but it seems like it should be correct.
In my test, it returns 1 when it should 2 for two correct GamePick belonging to a Pick.
Does anybody have any pointers?
Btw, if I remove the .values("pk") I get this error:
E django.db.utils.OperationalError: sub-select returns 5 columns - expected 1
I am not sure why it matters how many column I have when I want to count rows.
As feedback suggests that this is hard to debug without knowing the models, here they are:
class Pot(models.Model):
name = models.CharField(max_length=250, null=False, blank=False)
class Team(models.Model):
name = models.CharField(max_length=250, null=False, blank=False)
class Game(models.Model):
teams = models.ManyToManyField(
Team,
related_name="+",
)
winning_team = models.ForeignKey(
Team,
on_delete=models.CASCADE,
related_name="+",
blank=True,
null=True,
)
class Pick(models.Model):
picker = models.CharField(max_length=100, help_text="Name of the person picking")
# This is the method is would like to replace with an annotation
def count_correct_method(self):
return self.game_picks.filter(is_correct=True).count()
class GamePickQueryset(models.QuerySet):
def annotate_is_correct(self):
return self.annotate(
is_correct=models.ExpressionWrapper(
models.Q(picked_team=models.F("game__winning_team")),
output_field=models.BooleanField(),
)
)
class GamePickManager(models.Manager):
def get_queryset(self):
queryset = GamePickQueryset(self.model, using=self._db)
queryset = queryset.annotate_is_correct()
return queryset
GamePickMangerFromQueryset = GamePickManager.from_queryset(GamePickQueryset)
class GamePick(models.Model):
pick = models.ForeignKey(
Pick, on_delete=models.CASCADE, related_name="game_picks", null=True, blank=True
)
game = models.ForeignKey(Game, on_delete=models.CASCADE,
related_name="game_picks")
picked_team = models.ForeignKey(
Team, on_delete=models.CASCADE, related_name="+", null=True, blank=False
)
objects = GamePickMangerFromQueryset()
With these models, I am running this as a test in which I am trying to get the annotation working
team_1 = Team(name="Test Team 1")
team_1.save()
team_2 = Team(name="Test Team 2")
team_2.save()
team_3 = Team(name="Test Team 3")
team_3.save()
team_4 = Team(name="Test Team 4")
team_4.save()
team_5 = Team(name="Test Team 5")
team_5.save()
team_6 = Team(name="Test Team 6")
team_6.save()
assert Team.objects.count() == 6
pot = Pot(name="Test Pot")
pot.save()
assert Pot.objects.count() == 1
assert Pot.objects.first() == pot
game_1 = Game(pot=pot)
game_1.save()
game_1.teams.add(team_1, team_2)
game_1.winning_team = team_1
game_1.save()
game_2 = Game(pot=pot)
game_2.save()
game_2.teams.add(team_3, team_4)
game_2.winning_team = team_3
game_2.save()
game_3 = Game(pot=pot)
game_3.save()
game_3.teams.add(team_5, team_6)
game_3.winning_team = team_5
game_3.save()
assert Game.objects.count() == 3
assert pot.games.count() == 3
assert pot.games.all()[0].winning_team == team_1
assert pot.games.all()[1].winning_team == team_3
assert pot.games.all()[2].winning_team == team_5
pick = Pick(picker="Tester", pot=pot)
pick.save()
assert Pick.objects.count() == 1
game_pick_1 = GamePick(pick=pick, game=game_1, picked_team=team_1)
game_pick_1.save()
game_pick_2 = GamePick(pick=pick, game=game_2, picked_team=team_3)
game_pick_2.save()
game_pick_3 = GamePick(pick=pick, game=game_3, picked_team=team_6)
game_pick_3.save()
assert GamePick.objects.count() == 3
assert pick.game_picks.count() == 3
assert pick.game_picks.all()[0].is_correct == True
assert pick.game_picks.all()[1].is_correct == True
assert pick.game_picks.all()[2].is_correct == False
assert pick.count_correct() == 2
from django.db import models
correct_game_picks = GamePick.objects.filter(
pick=models.OuterRef("pk"),
is_correct=True,
)
pick = Pick.objects.all().annotate(
correct_count=models.Count(
# models.Q(game_picks__in=correct_game_picks)
models.Q(game_picks__picked_team=models.F("game_picks__game__winning_team"))
)
)[0]
assert pick.correct_count == 2
In this test I get 3 == 2. For some reason, it is counting all the game_picks not only the ones that fulfill the expression.
Really don't know what to do with that anymore...
I just realized (thanks to #BradMeinsberger), since I am doing that __in expression, I should not really need the OuterRef.
So the annotation can be just this:
correct_game_picks = GamePick.objects.filter(
is_correct=True,
)
pick = Pick.objects.all().annotate(
correct_count=models.Count(
models.Q(game_picks__in=correct_game_picks)
)
)[0]
But now the kicker: without the OuterRef I can evaluate the correct game picks separately:
assert correct_game_picks.count() == 2
assert pick.correct_count == 2
The first assert passes but the second does not with 3 == 2 😧
How can there be more than 2 in a list of 2?
Is there some kind of duplicate happening?
Now I can through a distinct=True into the Count and it passes 🎉
Let's test another combination e.g. only 1 correct game pick:
game_pick_1 = GamePick(pick=pick, game=game_1, picked_team=team_1)
game_pick_1.save()
game_pick_2 = GamePick(pick=pick, game=game_2, picked_team=team_4)
game_pick_2.save()
game_pick_3 = GamePick(pick=pick, game=game_3, picked_team=team_6)
game_pick_3.save()
assert GamePick.objects.count() == 3
assert pick.game_picks.count() == 3
assert pick.game_picks.all()[0].is_correct == True
assert pick.game_picks.all()[1].is_correct == False
assert pick.game_picks.all()[2].is_correct == False
assert pick.count_correct() == 1
from django.db import models
correct_game_picks = GamePick.objects.filter(
is_correct=True,
)
pick = Pick.objects.all().annotate(
correct_count=models.Count(
models.Q(game_picks__in=correct_game_picks),
distinct=True
)
)[0]
assert correct_game_picks.count() == 1
assert pick.correct_count == 1
💥 2 == 1
😭

In the SQL you generate the subquery inside the COUNT aggregate is joining to a games_game table that isn't anywhere else in your question. It looks like it's doing this to figure out if the pick is correct where elsewhere in your question you have a column on GamePick called is_correct that is used for this.
Here is how you would do it assuming you have the is_correct column and ignoring the games_game table
from django.db.models import Subquery, OuterRef, Count
subquery = GamePick.objects.filter(
pick=OuterRef('id'),
is_correct=True
).values(
'pick_id' # Necessary to get the proper group by
).annotate(
count=Count('pk')
).values(
'id' # Necessary to select only one column
)
picks = Pick.objects.annotate(correct_count=Subquery(subquery))
You can get the same thing using the django-sql-utils package. pip install django-sql-utils and then
from sql_util.utils import SubqueryCount
from django.db.models import Q
subquery = SubqueryCount('game_pick', filter=Q(is_correct=True))
picks=Pick.objects.annotate(correct_count=subquery)
If you need to determine if the pick is correct using the games_game table, I think you would replace is_correct=True (in both examples above) with
game__winning_team_id=F('picked_team_id')
I'm not 100% certain since I can't see those models/columns.

Just got it!
I guess I was making it more complicated than it needed to be.
correct_game_picks = GamePick.objects.filter(
pick=models.OuterRef("pk"),
is_correct=True
)
picks = Pick.objects.annotate(
correct_count=models.Count(
models.Q(game_picks__in=correct_game_picks)
)
)
and the resulting SQL:
SELECT
"picks_pick"."id",
"picks_pick"."picker",
"picks_pick"."pot_id",
COUNT(
"picks_gamepick"."id" IN (
SELECT U0."id" FROM "picks_gamepick" U0
INNER JOIN "games_game" U1 ON (U0."game_id" = U1."id")
WHERE ((U0."picked_team_id" = U1."winning_team_id") AND U0."pick_id" = "picks_pick"."id"))
) AS "correct_count"
FROM "picks_pick"
LEFT OUTER JOIN "picks_gamepick" ON ("picks_pick"."id" = "picks_gamepick"."pick_id")
GROUP BY "picks_pick"."id", "picks_pick"."picker", "picks_pick"."pot_id"
This seemingly unrelated blog post I came a across when searching for "Django subquery count" pointed me in the right direction:
https://mattrobenolt.com/the-django-orm-and-subqueries/
Nope. The above does not work. For some reason it only counts the number of game picks... 🤦‍♂️
Guess a proper look into the docs is always helpful:
correct_game_picks = GamePick.objects.filter(
is_correct=True,
)
picks = Pick.objects.all().annotate(
correct_count=models.Count(
"game_picks", # The field to count needs to be mentioned specifically
filter=models.Q(game_picks__in=correct_game_picks), # ... and you can define a filter to limit the number of rows in the aggregate
distinct=True. # Prevent duplicates! Important for counting rows
)
)
The aggregate filter is what is was looking for: https://docs.djangoproject.com/en/3.2/ref/models/querysets/#aggregate-filter
This is the generated SQL:
SELECT
"picks_pick"."id",
"picks_pick"."picker",
"picks_pick"."pot_id",
COUNT(
DISTINCT "picks_gamepick"."id"
) FILTER (
WHERE "picks_gamepick"."id" IN (
SELECT U0."id" FROM "picks_gamepick" U0
INNER JOIN "games_game" U1 ON (U0."game_id" = U1."id")
WHERE (U0."picked_team_id" = U1."winning_team_id")
)
)
AS "correct_count"
FROM "picks_pick"
LEFT OUTER JOIN "picks_gamepick" ON ("picks_pick"."id" = "picks_gamepick"."pick_id")
GROUP BY "picks_pick"."id", "picks_pick"."picker", "picks_pick"."pot_id"

How to limit top N of each group in Django ORM by using Postgres Window functions or Lateral Joins?

I have following Post, Category & PostScore Model.
class Post(models.Model):
category = models.ForeignKey('Category', on_delete=models.SET_NULL, related_name='category_posts', limit_choices_to={'parent_category': None}, blank=True, null=True)
status = models.CharField(max_length=100, choices=STATUS_CHOICES, default='draft')
deleted_at = models.DateTimeField(null=True, blank=True)
...
...
class Category(models.Model):
title = models.CharField(max_length=100)
parent_category = models.ForeignKey('self', on_delete=models.SET_NULL,
related_name='sub_categories', null=True, blank=True,
limit_choices_to={'parent_category': None})
...
...
class PostScore(models.Model):
post = models.OneToOneField(Post, on_delete=models.CASCADE, related_name='post_score')
total_score = models.DecimalField(max_digits=8, decimal_places=5, default=0)
...
...
So what i want is to write a query which returns N number of posts (Posts) of each distinct category (Category) sorted by post score (denoted by total_score column in PostScore model) in descending manner. So that i have atmost N records of each category with highest post score.
So i can achieve the above mentioned thing by the following raw query which gives me top 10 posts having highest score of each category :
SELECT *
FROM (
SELECT *,
RANK() OVER (PARTITION BY "post"."category_id"
ORDER BY "postscore"."total_score" DESC) AS "rank"
FROM
"post"
LEFT OUTER JOIN
"postscore"
ON
("post"."id" = "postscore"."post_id")
WHERE
("post"."deleted_at" IS NULL AND "post"."status" = 'accepted')
ORDER BY
"postscore"."total_score"
DESC
) final_posts
WHERE
rank <= 10
What i have achieved so far using Django ORM:
>>> from django.db.models.expressions import Window
>>> from django.db.models.functions import Rank
>>> from django.db.models import F
>>> posts = Post.objects.annotate(
rank=Window( expression=Rank(),
order_by=F('post_score__total_score').desc(),
partition_by[F('category_id')]
)). \
filter(status='accepted', deleted_at__isnull=True). \
order_by('-post_score__total_score')
which roughly evaluates to
>>> print(posts.query)
>>> SELECT *,
RANK() OVER (PARTITION BY "post"."category_id"
ORDER BY "postscore"."total_score" DESC) AS "rank"
FROM
"post"
LEFT OUTER JOIN
"postscore"
ON
("post"."id" = "postscore"."post_id")
WHERE
("post"."deleted_at" IS NULL AND "post"."status" = 'accepted')
ORDER BY
"postscore"."total_score"
DESC
So basically what is missing that i need to limit each group (i.e category) results by using “rank” alias.
Would love to know how this can be done ?
I have seen one answer suggested by Alexandr on this question, one way of achieving this is by using Subquery and in operator . Although it satisfies the above condition and outputs the right results but the query is very slow.
Anyway this would be the query if I go by Alexandr suggestions:
>>> from django.db.models import OuterRef, Subquery
>>> q = Post.objects.filter(status='accepted', deleted_at__isnull=True,
category=OuterRef('category')).order_by('-post_score__total_score')[:10]
>>> posts = Post.objects.filter(id__in=Subquery(q.values('id')))
So i am more keen in completing the above raw query (which is almost done just misses the limit part) by using window function in ORM. Also, i think this can be achieved by using lateral join so answers in this direction are also welcomed.

So I have got a workaround using RawQuerySet but the things is it returns a django.db.models.query.RawQuerySet which won't support methods like filter, exclude etc.
>>> posts = Post.objects.annotate(rank=Window(expression=Rank(),
order_by=F('post_score__total_score').desc(),
partition_by=[F('category_id')])).filter(status='accepted',
deleted_at__isnull=True)
>>> sql, params = posts.query.sql_with_params()
>>> posts = Post.objects.raw(""" SELECT * FROM ({}) final_posts WHERE
rank <= %s""".format(sql),[*params, 10],)
I'll wait for the answers which provides a solution which returns a QuerySet object instead, otherwise i have to do by this way only.

How to filter django model based on other non related model

Please refer to the code below
Transaction models
class Transaction(models.Model)
current_product_code = models.CharField(....)
previous_product_code = models.CharField(....)
#property
def status(self):
c_price = Product.objects.get(code=self.current_product_code).price
p_price = Product.objects.get(code=self.previous_product_code).price
if c_price == p_price:
return "Due"
elif c_price > p_price:
return "Upgrade"
else:
return "Downgrade"
Product model
class Product(models.Model):
code = models.CharField(....)
price = models.DecimalField(....)
My question: How can i obtain/filter transactions with upgrade/downgrade/due status. I am trying to create a custom admin filter which filter transaction based on their status but i fail what to put inside .filter() , check the method below
def queryset(self, request, queryset):
value = self.value()
if value == 'Upgrade':
return queryset.filter(***** HERE *****)
elif value == 'Downgrade':
return queryset.filter(***** HERE *****)
elif value == 'Unknown':
return queryset.filter(***** HERE *****)
return queryset

You really should use ForeignKey between Product and Transaction (for both: current_product_code and previous_product_code). This will allow you to use those relations in your querysets with ease.
My proposed models structure looks like this:
class Product(models.Model):
code = models.CharField(....)
price = models.DecimalField(....)
class Transaction(models.Model)
# You have to define related_name for at least one of relations below.
# Without that, automatically generated ones will clash.
# Also don't foget to change `on_delete` to suit your needs.
current_product = models.ForeignKey(Product, related_name="current_transactions", on_delete=models.CASCADE)
previous_product = models.ForeignKey(Product, related_name="previous_transactions", on_delete=models.CASCADE)
#property
def status(self):
# also, no need to do additional queries here manually. You can improve
# it further by using `select_related` when querying for transactions.
c_price = self.current_product.price
p_price = self.previous_product.price
if c_price == p_price:
return "Due"
elif c_price > p_price:
return "Upgrade"
else:
return "Downgrade"
With that model structure, finding specific types of transactions will be easier:
upgrade_transactions = Transaction.objects.filter(current_product__price__gt=F('previous_product__price'))
downgrade_transactions = Transaction.objects.filter(current_product__price__lt=F('previous_product__price'))
due_transactions = Transaction.objects.filter(current_product__price=F('previous_product__price'))

I think you could try to use Subquery, OuterRef and .annotate():
if value == 'Upgrade':
return queryset.annotate(
previous_price=Subquery(
Product.objects.filter(
code=OuterRef("previous_product_code")
)[:1]
),
current_price=Subquery(
Product.objects.filter(
code=OuterRef("current_product_code")
)[:1]
),
).filter(current_price__gt=F("previous_price"))
...

Remember that filter() operation, in the end, is a SQL operation and we should take care on the performance issues.
So my advice is: if you need to filter by status, update the status on Product model everytime a transaction is saved. Your application will be faster and will have a cleaner code.