Develop Reference

Django Annotated Query to Count Only Latest from Reverse Relationship

Problem Overview
Given the models
class Candidate(BaseModel):
name = models.CharField(max_length=128)
class Status(BaseModel):
name = models.CharField(max_length=128)
class StatusChange(BaseModel):
candidate = models.ForeignKey("Candidate", related_name="status_changes")
status = models.ForeignKey("Status", related_name="status_changes")
created_at = models.DateTimeField(auto_now_add=True, blank=True)
And SQL Tables:
candidates
+----+--------------+
| id | name |
+----+--------------+
| 1 | Beth |
| 2 | Mark |
| 3 | Mike |
| 4 | Ryan |
+----+--------------+
status
+----+--------------+
| id | name |
+----+--------------+
| 1 | Review |
| 2 | Accepted |
| 3 | Rejected |
+----+--------------+
status_change
+----+--------------+-----------+------------+
| id | candidate_id | status_id | created_at |
+----+--------------+-----------+------------+
| 1 | 1 | 1 | 03-01-2019 |
| 2 | 1 | 2 | 05-01-2019 |
| 4 | 2 | 1 | 01-01-2019 |
| 5 | 3 | 1 | 01-01-2019 |
| 6 | 4 | 3 | 01-01-2019 |
+----+--------------+-----------+------------+
I want to get the get the total number of candidates with a given status, but only the latest status_change is counted.
In other words, StatusChange is used to track history of status, but only the latest is considered when counting current status of candidates.
SQL Solution
Using SQL, I was able to achieve it using Group BY and COUNT.
(SQL untested)
SELECT
status.id as status_id
, status.name as status_name
, COUNT(*) as status_count
FROM
(
SELECT
status_id,
Max(created_at) AS latest_status_change
FROM
status_change
GROUP BY status_id
)
AS last_status_count
INNER JOIN
last_status_count AS status
ON (last_status_count.status_id = status.id)
GROUP BY status.name
ORDER BY status_count DESC;
last_status_count
+-----------+-------------+--------+
| status_id | status_name | count |
+-----------+-------------+--------+
| 1 | Review | 2 | # <= Does not include instance from candidate 1
| 2 | Accepted | 1 | # because status 2 is latest
| 3 | Rejected | 1 |
+-----------+-------------+--------+
Attempted Django Solution
I need a view to return each status and their corresponding count -
eg [{ status_name: "Review", count: 2 }, ...]
I am not sure how to build this queryset, without pulling all records and aggregating in python.
I figured I need annotate() and possibly Subquery but I haven't been able to stitch it all together.
The closest I got is this, which counts the number of status change for each status but does counts non-latest changes.
queryset = Status.objects.all().annotate(case_count=Count("status_changes"))
I have found lot's of SO questions on aggregating, but I couldn't find a clear answer on aggregating and annotating "latest.
Thanks in advance.

We can perform a query where we first filter the last StatusChanges per Candidate and then count the statusses:
from django.db.models import Count, F, Max
Status.objects.filter(
status_changes__in=StatusChange.objects.annotate(
last=Max('candidate__status_changes__created_at')
).filter(
created_at=F('last')
)
).annotate(
nlast=Count('status_changes')
)
For the given sample data, this gives us:
>>> [(q.name, q.nlast) for q in qs]
[('Review', 2), ('Accepted', 1), ('Rejected', 1)]

How to make func.sum and group_by to output sum of the rows and merge the duplicate rows using sqlalchemy

I want to generate a table which sums the number of books that are sold and the total amount paid for that distinct book in a given period of time. I need it to show a report of books that are sold.
My subquery is:
bp = db.session.query(CustomerPurchase.book_category_id,
func.sum(CustomerPurchase.amount).label('amount'),
func.sum(CustomerPurchase.total_price).label('total_price'))\
.filter(CustomerPurchase.created_on >= start_date)\
.filter(CustomerPurchase.created_on <= end_date)\
.group_by(CustomerPurchase.book_category_id).subquery()
Combined query with a subquery:
cp = CustomerPurchase.query\
.join(bp, bp.c.category_id == CustomerPurchase.category_id)\
.distinct(bp.c.category_id)\
.order_by(bp.c.category_id)
My CustomerPurchase table looks like this and the output of my query looks the same:
id | book_category_id | book_title | amount | total_price |
---+------------------+------------+--------+-------------+
1 | 1 | Book A | 10 | 35.00 |
2 | 1 | Book A | 20 | 70.00 |
3 | 2 | Book B | 40 | 45.00 |
Desired output after the query run should be like this:
id | book_category_id | book_title | amount | total_price |
---+------------------+------------+--------+-------------+
1 | 1 | Book A | 30 | 105.00 |
2 | 2 | Book B | 40 | 45.00 |
Above query displays all the books that are sold to customer from CustomerPurchase table, but it doesn't SUM the amount and total_price nor it merges the duplicate
I have seen many examples but none of them worked for me. Any help is greatly appreciated! Thanks in advance!

So after a lot of research and trials I came up with a query which solved my problem. Basically I used add_column attribute in sqlalchemy which gave me exact rows that I wanted to display for my report.
bp = db.session.query(CustomerPurchase.book_store_category_id,
func.sum(CustomerPurchase.quantity).label('quantity'),
func.sum(CustomerPurchase.total_price).label('total'))\
.filter(CustomerPurchase.created_on >= start_date)\
.filter(CustomerPurchase.created_on <= end_date)
bp = bp.add_column(BookStore.book_amount)\
.filter(BookStore.category_id == CustomerPurchase.book_store_category_id)
bp = bp.add_columns(Category.category_name, Category.total_stock_amount)\
.filter(Category.id == CustomerPurchase.book_store_category_id)
bp = bp.add_column(Category.unit_cost)\
.filter(Category.id == CustomerPurchase.book_store_category_id)
bp = bp.add_column(Book.stock_amount)\
.filter(Book.category_id == CustomerPurchase.book_store_category_id)\
.group_by(BookStore.book_amount, CustomerPurchase.book_store_category_id, Category.category_name, Category.unit_cost, Category.total_stock_amount, Book.stock_amount)
bp = bp.all()

Django - How to combine 2 queryset and filter to get same element in both queryset?

I have a model:
class LocationItem(models.Model):
location = models.ForeignKey(Location, on_delete=models.CASCADE)
item = models.ForeignKey(Item, on_delete=models.CASCADE)
stock_qty = models.IntegerField(null=True)
Example: I have some data like this:
------------------------------
| ID | Item | Location | Qty |
------------------------------
| 1 | 1 | 1 | 10 |
------------------------------
| 2 | 2 | 1 | 5 |
------------------------------
| 3 | 1 | 2 | 2 |
------------------------------
| 4 | 3 | 1 | 4 |
------------------------------
| 5 | 3 | 2 | 20 |
------------------------------
I have 2 queryset to get items of each location:
location_1 = LocationItem.objects.filter(location_id=1)
location_2 = LocationItem.objects.filter(location_id=2)
Now I want to combine 2 queryset above into 1 and filter only same items in both 2 location such as result of this example above is [Item 1, Item 3] because item 1 and 3 belong to both location 1 and 2

You can combine django query set using following expression
location_1 = LocationItem.objects.filter(location_id=1)
location_2 = LocationItem.objects.filter(location_id=2)
location = location_1 | location_2
Above combine expression works on same model filter query set.
Try this one
from django.db.models import Count
dupes = LocationItem.objects.values('item__id').annotate(Count('id')).order_by().filter(id__count__gt=1)
LocationItem.objects.filter(item__=[i['item__id'] for i in dupes]).distinct('item__id')
May be above solution help.

If you want both conditions to be true, then you need the AND operator (&)
from django.db.models import Q
Q(location_1) & Q(location_2)

Try this:
location1_items_pk = LocationItem.objects.filter(
location_id=1
).values_list("item_pk", flat=true)
Result = Location.objects.filter(
item_pk__in=location1_items_pk, location_id=2
)

You can do this by piping the filters. The result of a filter is a queryset. So after the first filtering, the result will be [Item1, Item2 , Item3] and then second filter will be applied on the resulting queryset which leads [Item1, Item3]. For eg.
Item.objects.filter(locationitem_set__location = 1).filter(locationitem_set__location = 2)
P.S. Not tested. Hope this works.

Selecting minimum and grouping

I have the following table:
id | product_id | quantity
--------------------------
1 | 222 | 25
2 | 222 | 35
3 | 223 | 10
Now I want to select the lowest quantities grouped by product_id. In SQL this works as
SELECT product_id, MIN(quantity) FROM my_table GROUP BY product_id
The result of this query is the following
product_id | MIN(quantity)
--------------------------
222 | 25
223 | 10
However, how can I use Django's database models to do the same?
I tried
myModel.objects.filter(product__in=product_ids).annotate(Min("quantity")).values_list("product", "quantity__min")
This returns the full table.

objects = (MyModel.objects
.values('product')
.annotate(Min('quantity'))
# if you want to get values_list
.values_list('product', 'quantity__min'))
https://docs.djangoproject.com/en/1.8/topics/db/aggregation/#values

Django, how to make multiple annotate in single queryset?

using Django 1.7, Python 3.4 and PostgreSQL 9.1 I am having difficulties with annotate over queryset.
Here is my model:
class Payment(models.Model):
TYPE_CHOICES= (
('C', 'CREDIT'),
('D', 'DEBIT')
)
amount = models.DecimalField(max_digits=8, decimal_places=2, default=0.0)
customer = models.ForeignKey(Customer, null=False)
type=models.CharField(max_length=1, null=True, choices=TYPE_CHOICES)
class Customer(models.Model):
name = models.CharField(max_length=100, unique=True)
available_funds = models.DecimalField(max_digits=8, decimal_places=2, null=True, default=0.0)
total_funds = models.DecimalField(max_digits=8, decimal_places=2, null=True, default=0.0)
What I am trying to get is something like:
Customers:
Name | Total in | Total out | available funds | total funds
-----------------------------------------------------------------
cust 1 | 255 | 220 | 5 | 35
cust 2 | 100 | 120 | 0 | -20
cust 3 | 50 | 20 | 15 | 30
and some data:
Payments:
amount | customer | type
--------------------------
20 | cust 1 | D
10 | cust 1 | c
70 | cust 2 | D
20 | cust 2 | C
10 | cust 2 | D
25 | cust 1 | C
200 | cust 3 | D
10 | cust 3 | C
20 | cust 1 | D
i was trying this query set:
Customer.objects.select_related().filter(Q(payment__isnull=False)& Q(payment__type='D')).values('name').annotate(Sum('payment__amount'))
but i am getting only Debits.
I don't know how to create a list with customer,total in, total out, total funds, available funds.
Can anyone help me with this?

I think you're hitting a limit of what you can do with a single queryset. The reason I say this is that you're asking to do database aggregation on different sets of Payment records.
Let's look at your current queryset:
Customer.objects.select_related().filter(Q(payment__isnull=False)& Q(payment__type='D')).values('name').annotate(Sum('payment__amount'))
Ignoring the extraneous Q() calls, the filter call payment__type='D' means that the payment_amount will always only pertain to debits. If you change that to 'C', it'll always only pertain to credits. This query demonstrates a fundamental constraint imposed on you by Django's queryset language -- you can't really generate two different aggregations and annotate them into a single record.
Taking a detour off to raw SQL land to see how I'd write this query is another way of demonstrating the point. You'll note, of course, that I still am running two different Payment aggregations here! One for credits and one for debits.
SELECT
*
FROM
customer
INNER JOIN
(
SELECT SUM(amount) as total FROM Payment WHERE type='C' GROUP BY customer_id, type
) AS credits
ON credits.customer_id=customer.id
INNER JOIN
(
SELECT SUM(amount) as total FROM Payment WHERE type='D' GROUP BY customer_id, type
) AS debits
ON debits.customer_id=customer.id
That query will return data approximately of the form:
customer.id | customer.name | ... | credits.total | debits.total
----------------------------------------------------------------
1 | foo bar | | 20 | 30
2 | baz qux | | 30 | 20
If you try to use only one inner join/aggregation, you're forced to have group by both payment type and customer, resulting in a table like this:
customer_id | type | sum(amount)
--------------------------------
1 | C | 20
1 | D | 30
2 | C | 30
2 | D | 20
When you inner join this intermediate result with your customers, it should be immediately clear that debits and credits still are not unified into a single record.
Because you can't do this sort of select with inner joins in Django (as far as I know), you can't really do what you're trying to do in a single query. However, there are solutions to your problem.
In descending order of desirability (in my opinion, of course -- and based on what I consider obviousness/maintainability of your resulting code), the first is to just do multiple queries and unify the results manually.
You can also track credits/debits as a part of the Customer record. You're already tracking available funds this way (you're using F objects in your query to update/maintain these records, right?), so it's not really too much more onerous to maintain credit/debit summaries in similar fashion as well.
Lastly, and I don't think you should do this as I don't think there's a burning need to, you can perform a raw SQL query to get the results you need in one go.