I want to get all duplicates based on a case insensitive field value.
Basically to rewrite this SQL query
SELECT count(*), lower(name)
FROM manufacturer
GROUP BY lower(name)
HAVING count(*) > 1;
with Django ORM. I was hoping something like this would do the trick
from django.db.models import Count
from django.db.models.functions import Lower
from myapp.models import Manufacturer
qs = Manufacturer.objects.annotate(
name_lower=Lower('name'),
cnt=Count('name_lower')
).filter('cnt__gt'=1)
but of course it didn't work.
Any idea how to do this?
you can try it:
qs = Manufacturer.objects.annotate(lname=Lower('name')
).values('lname').annotate(cnt=Count(Lower('name'))
).values('lname', 'cnt').filter(cnt__gt=1).order_by('lname', 'cnt')
why should add the order_by ordering-or-order-by:
the sql query looks like:
SELECT
LOWER("products_manufacturer"."name") AS "lname",
COUNT(LOWER("products_manufacturer"."name")) AS "cnt"
FROM "products_manufacturer"
GROUP BY LOWER("products_manufacturer"."name")
HAVING COUNT(LOWER("products_manufacturer"."name")) > 1
ORDER BY "lname" ASC, "cnt" ASC
Related
I have an annotation like this: which displays the month wise count of a field
bar = Foo.objects.annotate(
item_count=Count('item')
).order_by('-item_month', '-item_year')
and this produces output like this:
html render
I would like to show the change in item_count when compared with the previous month item_count for each month (except the first month). How could I achieve this using annotations or do I need to use pandas?
Thanks
Edit:
In SQL this becomes easy with LAG function, which is similar to
SELECT item_month, item_year, COUNT(item),
LAG(COUNT(item)) OVER (ORDER BY item_month, item_year)
FROM Foo
GROUP BY item_month, item_year
(PS: item_month and item_year are date fields)
Do Django ORM have similar to LAG in SQL?
For these types of Query you need to use Window functions in django Orm
For Lag you can take the help of
https://docs.djangoproject.com/en/4.0/ref/models/database-functions/#lag
Working Query in Orm will look like this :
#models.py
class Review(models.Model):
user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='review_user', db_index=True)
review_text = models.TextField(max_length=5000)
rating = models.SmallIntegerField(
validators=[
MaxValueValidator(10),
MinValueValidator(1),
],
)
date_added = models.DateTimeField(db_index=True)
review_id = models.AutoField(primary_key=True, db_index=True)
This is just a dummy table to show you the use case of Lag and Window function in django
Because examples are not available for Lag function on Django Docs.
from django.db.models.functions import Lag, ExtractYear
from django.db.models import F, Window
print(Review.objects.filter().annotate(
num_likes=Count('likereview_review')
).annotate(item_count_lag=Window(expression=Lag(expression=F('num_likes')),order_by=ExtractYear('date_added').asc())).order_by('-num_likes').distinct().query)
Query will look like
SELECT DISTINCT `temp_view_review`.`user_id`, `temp_view_review`.`review_text`, `temp_view_review`.`rating`, `temp_view_review`.`date_added`, `temp_view_review`.`review_id`, COUNT(`temp_view_likereview`.`id`) AS `num_likes`, LAG(COUNT(`temp_view_likereview`.`id`), 1) OVER (ORDER BY EXTRACT(YEAR FROM `temp_view_review`.`date_added`) ASC) AS `item_count_lag` FROM `temp_view_review` LEFT OUTER JOIN `temp_view_likereview` ON (`temp_view_review`.`review_id` = `temp_view_likereview`.`review_id`) GROUP BY `temp_view_review`.`review_id` ORDER BY `num_likes` DESC
Also if you don't want to order_by on extracted year of date then you can use F expressions like this
print(Review.objects.filter().annotate(
num_likes=Count('likereview_review')
).annotate(item_count_lag=Window(expression=Lag(expression=F('num_likes')),order_by=[F('date_added')])).order_by('-num_likes').distinct().query)
Query for this :
SELECT DISTINCT `temp_view_review`.`user_id`, `temp_view_review`.`review_text`, `temp_view_review`.`rating`, `temp_view_review`.`date_added`, `temp_view_review`.`review_id`, COUNT(`temp_view_likereview`.`id`) AS `num_likes`, LAG(COUNT(`temp_view_likereview`.`id`), 1) OVER (ORDER BY `temp_view_review`.`date_added`) AS `item_count_lag` FROM `temp_view_review` LEFT OUTER JOIN `temp_view_likereview` ON (`temp_view_review`.`review_id` = `temp_view_likereview`.`review_id`) GROUP BY `temp_view_review`.`review_id` ORDER BY `num_likes` DESC
I have this SQL:
SELECT
stock_id, consignment_id, SUM(qty), SUM(cost)
FROM
warehouse_regсonsignmentproduct
WHERE
product_id = '1'
GROUP BY
stock_id, consignment_id
HAVING
SUM(qty) > 0
I used django ORM to create this query:
regСonsignmentProduct.objects
.filter(product='1')
.order_by('period')
.values('stock', 'consignment')
.annotate(total_qty=Sum('qty'), total_cost=Sum('cost'))
.filter(total_qty__gt=0)
But my django query returns an incorrect result.
I think, the problem is in "annotate"
Thanks!
You need to order by the values to force grouping, so:
regСonsignmentProduct.objects.filter(product='1').values(
'stock', 'consignment'
).annotate(
total_qty=Sum('qty'),
total_cost=Sum('cost')
).order_by('stock', 'consignment').filter(total_qty__gt=0)
I need to calculate period medians per seller ID (see simplyfied model below). The problem is I am unable to construct the ORM query.
Model
class MyModel:
period = models.IntegerField(null=True, default=None)
seller_ids = ArrayField(models.IntegerField(), default=list)
aux = JSONField(default=dict)
Query
queryset = (
MyModel.objects.filter(period=25)
.annotate(seller_id=Func(F("seller_ids"), function="unnest"))
.values("seller_id")
.annotate(
duration=Cast(KeyTextTransform("duration", "aux"), IntegerField()),
median=Func(
F("duration"),
function="percentile_cont",
template="%(function)s(0.5) WITHIN GROUP (ORDER BY %(expressions)s)",
),
)
.values("median", "seller_id")
)
ArrayField aggregation (seller_id) source
I think what I need to do is something along the lines below
select t.*, p_25, p_75
from t join
(select district,
percentile_cont(0.25) within group (order by sales) as p_25,
percentile_cont(0.75) within group (order by sales) as p_75
from t
group by district
) td
on t.district = td.district
above example source
Python 3.7.5, Django 2.2.8, Postgres 11.1
You can create a Median child class of the Aggregate class as was done by Ryan Murphy (https://gist.github.com/rdmurphy/3f73c7b1826cacee34f6c2a855b12e2e). Median then works just like Avg:
from django.db.models import Aggregate, FloatField
class Median(Aggregate):
function = 'PERCENTILE_CONT'
name = 'median'
output_field = FloatField()
template = '%(function)s(0.5) WITHIN GROUP (ORDER BY %(expressions)s)'
Then to find the median of a field use
my_model_aggregate = MyModel.objects.all().aggregate(Median('period'))
which is then available as my_model_aggregate['period__median'].
Here's what did the trick.
from django.db.models import F, Func, IntegerField
from django.db.models.aggregates import Aggregate
queryset = (
MyModel.objects.filter(period=25)
.annotate(duration=Cast(KeyTextTransform("duration", "aux"), IntegerField()))
.filter(duration__isnull=False)
.annotate(seller_id=Func(F("seller_ids"), function="unnest"))
.values("seller_id") # group by
.annotate(
median=Aggregate(
F("duration"),
function="percentile_cont",
template="%(function)s(0.5) WITHIN GROUP (ORDER BY %(expressions)s)",
),
)
)
Notice the median annotation employs Aggregate and not Func as in the question.
Also, order of annotate() and filter() clauses as well as order of annotate() and values() clauses matters a lot!
BTW the resulting SQL is without a nested select and join.
I have a database table with tweets in a jsonb field.
I have a query to get the tweets ordered by the most retweeted, this is what it looks like:
SELECT * FROM (
SELECT DISTINCT ON (raw->'retweeted_status'->'id_str')
raw->'retweeted_status' as status,
raw->'retweeted_status'->'retweet_count' as cnt
FROM tweet
WHERE (raw->'retweeted_status') is not null
ORDER BY raw->'retweeted_status'->'id_str', cnt DESC
) t
ORDER BY cnt DESC
I'm trying to create this query with sqlalchemy, this is where i got so far:
session.query(Tweet.raw['retweeted_status'],
Tweet.raw['retweeted_status']['retweet_count'].label('cnt'))\
.filter(~Tweet.raw.has_key('retweeted_status'))\
.distinct(Tweet.raw['retweeted_status']['id_str']).order_by(Tweet.raw['retweeted_status']['id_str'].desc()).subquery()
But how to go from that to order by cnt?
It may not produce the exact query you have shown but should point you in the right direction: you can use your label 'cnt' in order_by, like: .order_by('cnt').
Moreover you can use your label as an argument for sqlalchemy.desc function. Summing up:
from sqlalchemy import desc
q = (
session.query(
Tweet.raw['retweeted_status'],
Tweet.raw['retweeted_status']['retweet_count'].label('cnt')
)
.filter(~Tweet.raw.has_key('retweeted_status'))
.distinct(
Tweet.raw['retweeted_status']['id_str']
)
.order_by(desc('cnt'))
).subquery()
Additional hint: you can format your query nicely if you put it in parentheses.
You may want to read answers to a general question on python sqlalchemy label usage too.
This my is query:
SELECT kategoriharga,ongkoskirim,diskon,ratingproduk,ratingtoko,label
FROM
(SELECT *
FROM pohonkeputusan
where perdaerah='Kabupaten Toba Samosir'
order by label desc
) AS sub
GROUP BY
kategoriharga,ongkoskirim,diskon,ratingproduk,ratingtoko
How to make to be query set in Django?
I don't understand why you want to group by all fields. Try to use distinct:
Pohonkeputusan.objects.filter(perdaerah='Kabupaten Toba Samosir').order_by('-label').values_list('kategoriharga', 'ongkoskirim', 'diskon', 'ratingproduk', 'ratingtoko').distinct()