Develop Reference

How to handle 300 parameters in Django Model / Form?

I develop an app for creating products in online shop. Let's suppose I have 50 categories of products and each of these has some required parameters for product (like color, size, etc.).
Some parameters apper in all categories, and some are unique. That gives me around 300 parameters (fields) that should be defined in Django model.
I suppose it is not good idea to create one big database with 300 fields and add products that have 1-15 parameters there (leaving remaining fields empty). What would be the best way to handle it?
What would be the best way to display form that will ask only for parameters required in given category?

If you have to keep the Model structure as you have defined it here, I would create a "Product" "Category" "ProductCategory" tables.
Product table is as follows:
ProductID
ProductName
1
Shirt
2
Table
3
Vase
Category table is following
CategoryID
CategoryName
1
Size
2
Color
3
Material
ProductCategory
ID
ProductID
CategoryID
CategoryValue
1
1 (Shirt)
1 (Size)
Medium
2
2 (Table)
2 (Color)
Dark Oak
3
3 (Vase)
3 (Material)
Glass
3
3 (Vase)
3 (Material)
Plastic
This would be the easiest way, which wouldn't create 300 columns, would allow you to reuse categories across different types of products, but in the case of many products, would start to slowdown the database queries, as you would be joining 2 big tables. Product and ProductCategory
You could split it up in more major Categories such as "Plants", "Kitchenware" etc etc.

1-1. Create Models for each category
You can build 50 classes for describing your products in shop.
This is a simple and basic solution. It can also be an optimal solution if the domain logic varies from category to category.
class Furniture(Model):
price
size
color
...
class Clothes(Model):
price
gender
texture
...
1-2. Aggregate common fields into base class
If you have many common fields, introducing inheritance would be a great idea.
class Base(Model):
price
...
class Meta:
abstract = True
class Furniture(Base):
size
color
...
class Clothes(Base):
gender
texture
...
2-1. One BigTable
I guess this is what you were going to do.
I suppose it is not good idea to create one big database with 300 fields and add products that have 1-15 parameters there (leaving remaining fields empty).
Like you said, the rest of the field will remain, but it's a not bad idea unless domain logic is different by category.
class Product(Model):
price
size
color
gender
texture
...
2-2. One Table, but several models
Tables are in the data layer and models are in the domain layer. It does not have to be considered the same as the model.
You can build a proxy model to describe each category type.
Pros
simple data layer
available to deal with complex domain logic across different categories
Cons
code complexity due to proxy processing
various difficulties arising from the table and model not being one-on-one
class ProductProxyManager(Manager):
def get_queryset(self):
return (
super()
.get_queryset()
.filter(type=self.model.product_type)
.only(*(self.model.required_fields + self.model.base_fields))
)
class ProductType(enum.Enum):
Furniture = "furniture"
Clothes = "clothes"
class Product(Model):
type: ProductType
price
size
color
gender
texture
...
def __new__(cls, *args, **kwargs):
# get proxy name, either from kwargs or from args
type: ProductType = kwargs.get("type")
if type is None:
type_field_index = cls._meta.fields.index(cls._meta.get_field("type"))
proxy_name = args[type_field_index]
else:
proxy_name = type
# get proxy class, by name, from the block formwork
instance_class = Product.get_instance_class(proxy_name)
o = super().__new__(instance_class)
return o
#staticmethod
def get_instance_class(type: ProductType) -> Type["ProductType"]:
return {
ProductType.Furniture: Furniture,
ProductType.Clothes: Clothes,
}[type]
class Furniture(Product):
class Meta:
proxy = True
required_fields = ("size", "color")
objects = ProductProxyManager()
class Clothes(Product):
class Meta:
proxy = True
required_fields = ("gender", "texture")
objects = ProductProxyManager()
You can see further steps here. (I followed up to step 3.)
https://stackoverflow.com/a/60894618/8614565

Is Nested aggregate queries possible with Django queryset

I want to calculate the monthly based profit with the following models using django queryset methods. The tricky point is that I have a freightselloverride field in the order table. It overrides the sum of freightsell in the orderItem table. An order may contain multiple orderItems. That's why I have to calculate order based profit first and then calculate the monthly based profit. Because if there is any order level freightselloverride data I should take this into consideration.
Below I gave a try using annotate method but could not resolve how to reach this SQL. Does Django allow this kind of nested aggregate queries?
select sales_month
,sum(sumSellPrice-sumNetPrice-sumFreighNet+coalesce(FreightSellOverride,sumFreightSell)) as profit
from
(
select CAST(DATE_FORMAT(b.CreateDate, '%Y-%m-01 00:00:00') AS DATETIME) AS `sales_month`,
a.order_id,b.FreightSellOverride
,sum(SellPrice) as sumSellPrice,sum(NetPrice) as sumNetPrice
,sum(FreightNet) as sumFreighNet,sum(FreightSell) as sumFreightSell
from OrderItem a
inner join Order b
on a.order_id=b.id
group by 1,2,3
) c
group by sales_month
I tried this
result = (OrderItem.objects
.annotate(sales_month=TruncMonth('order__CreateDate'))
.values('sales_month','order','order__FreightSellOverride')
.annotate(sumSellPrice=Sum('SellPrice'),sumNetPrice=Sum('NetPrice'),sumFreighNet=Sum('FreightNet'),sumFreightSell=Sum('FreightSell'))
.values('sales_month')
.annotate(profit=Sum(F('sumSellPrice')-F('sumNetPrice')-F('sumFreighNet')+Coalesce('order__FreightSellOverride','sumFreightSell')))
)
but get this error
Exception Type: FieldError
Exception Value:
Cannot compute Sum('<CombinedExpression: F(sumSellPrice) - F(sumNetPrice) - F(sumFreighNet) + Coalesce(F(ProjectId__FreightSellOverride), F(sumFreightSell))>'): '<CombinedExpression: F(sumSellPrice) - F(sumNetPrice) - F(sumFreighNet) + Coalesce(F(ProjectId__FreightSellOverride), F(sumFreightSell))>' is an aggregate
from django.db import models
from django.db.models import F, Count, Sum
from django.db.models.functions import TruncMonth, Coalesce
class Order(models.Model):
CreateDate = models.DateTimeField(verbose_name="Create Date")
FreightSellOverride = models.FloatField()
class OrderItem(models.Model):
SellPrice = models.DecimalField(max_digits=10,decimal_places=2)
FreightSell = models.DecimalField(max_digits=10,decimal_places=2)
NetPrice = models.DecimalField(max_digits=10,decimal_places=2)
FreightNet = models.DecimalField(max_digits=10,decimal_places=2)
order = models.ForeignKey(Order,on_delete=models.DO_NOTHING,related_name="Item")

Django: How can I add an aggregated field to a queryset based on data from the row and data from another Model?

I have a Django App with the following models:
CURRENCY_CHOICES = (('USD', 'US Dollars'), ('EUR', 'Euro'))
class ExchangeRate(models.Model):
currency = models.CharField(max_length=3, default='USD', choices=CURRENCY_CHOICES)
rate = models.FloatField()
exchange_date = models.DateField()
class Donation(models.Model):
donation_date = models.DateField()
donor = models.CharField(max_length=250)
amount = models.FloatField()
currency = models.CharField(max_length=3, default='USD', choices=CURRENCY_CHOICES)
I also have a form I use to filter donations based on some criteria:
class DonationFilterForm(forms.Form)
min_amount = models.FloatField(required=False)
max_amount = models.FloatField(required=False)
The min_amount and max_amount fields will always represent values in US Dollars.
I need to be able to filter a queryset based on min_amount and max_amount, but for that all the amounts must be in USD. To convert the donation amount to USD I need to multiply by the ExchangeRate of the donation currency and date.
The only way I found of doing this so far is by iterating the dict(queryset) and adding a new value called usd_amount, but that may offer very poor performance in the future.
Reading Django documentation, it seems the same thing can be done using aggregation, but so far I haven't been able to create the right logic that would give me same result.

I knew I had to use annotate to solve this, but I didn't know exactly how because it involved getting data from an unrelated Model.
Upon further investigation I found what I needed in the Django Documentation. I needed to use the Subquery and the OuterRef expressions to get the values from the outer queryset so I could filter the inner queryset.
The final solution looks like this:
# Prepare the filter with dynamic fields using OuterRef
rates = ExchangeRate.objects.filter(exchange_date=OuterRef('date'), currency='EUR')
# Get the exchange rate for every donation made in Euros
qs = Donation.objects.filter(currency='EUR').annotate(exchange_rate=Subquery(rates.values('rate')[:1]))
# Get the equivalent amount in USD
qs = qs.annotate(usd_amount=F('amount') * F('exchange_rate'))
So, finally, I could filter the resulting queryset like so:
final_qs = qs.filter(usd_amount__gte=min_amount, usd_amount__lte=max_amount)

Sum averages over date ranges in Django

I'm trying to construct a query in Django that sums averages that were taken (i.e. averaged) over a range of times.
Here is the relevant Django model:
class Data(models.Model):
class Meta:
verbose_name_plural = "Data"
site = models.ForeignKey(Site)
created_on = models.DateTimeField(auto_created=True)
reported_on = models.DateTimeField(null=True, blank=True)
baseline_power_kw = models.FloatField('Baseline Power (kw)', blank=True, null=True)
measured_power_kw = models.FloatField('Measured Power (kw)', blank=True, null=True)
In my query, I'm trying to average sites' data over a range of times, and then sum those averages for each range of time. Here is the query I have so far, which I believe just gets the average of all sites' data over a range of times.
t_data = Data.objects.filter(site__in=sites) \
.filter(created_on__range=(start, end)) \
.extra(select={'date_slice': "trunc(extract(epoch from created_on) / '60' )"}) \
.values('date_slice') \
.annotate(avg_baseline_power_kw=Avg('baseline_power_kw'),
avg_measured_power_kw=Avg('measured_power_kw'),
time=Min('created_on')) \
.order_by('-created_on')
Do you know how I can proceed? I am using Django with Postgres.
Thanks!

If you add 'site' to your .values() clause, like this:
.values('date_slice', 'site')
and remove the order_by, which will cause the 'created_on' field to get added to the generated SQL GROUP BY, you should get averages for your two measurements for each slice+site. You can then sum those values to get the totals across all sites.

django product price tracker: getting the amount and date of the all time max and min price

I'm trying to build a django app where I can track product prices over time. The app fetches new prices routinely, graphs them and shows the recent history of price changes.
I'm checking the price once a day and saving that price plus the date timestamp to my models.
models.py
Class Product(models.Model):
title = models.CharField(max_length=255)
Class Price(models.Model):
product = models.ForeignKey(Product, on_delete=models.CASCADE)
date_seen = models.DateTimeField(auto_now_add=True)
price = models.IntegerField(blank=True, null=True)
Along with the current price of a product I'd also like to show the max and min over all the price data I've collected. I want to get the value and also the date it was at that value. So far I can get the value but I can't get the corresponding date. I'm using this:
def price_hla(self):
return Product.objects.filter(price__product=self).aggregate(high_price=Max('price__price'), low_price=Min('price__price'), avg_price=Avg('price___price'))
Any advice? Thanks in advance!
EDIT: Based on responses I have the following. My problem is I'm getting the MAX price and MAX date independent of each other. I want the MAX price with that max price's date in the same response.
def price_hla(self):
return
Product.objects.filter(price__product=self)[:1].annotate(Max('price__price'), Max('price__date_seen'))`

Try this:
Product.objects.filter(price__product=self).annotate(
high_price=Max('price__price'),
).filter(price=F('max_price'))
Which should give you the max price and date in the resulting objects.
I can't think of a way to simultaneously find the minimum price/date in the same query though. I also have a feeling that this is going to be very slow if you have a large number of items.

Figured this out and I'm getting what I want. If anyone reads this I'd love feedback about if this is best practice or if I'm going to be overloading my database.
Because I needed both the actual price and the date the price was max I needed to return the whole Price Object. So I wrote some QuerySets on my DetailView by overwriting the default get_context_data method.
views.py
class ProductDetailView(DetailView):
model = Product
def get_context_data(self, **kwargs):
context = super(ProductDetailView, self).get_context_data(**kwargs)
context['high'] = Price.objects.filter(product_id=self.get_object()).order_by('price').last()
context['low'] = Price.objects.filter(product_id=self.get_object()).order_by('-price').last()
context['avg'] = Price.objects.filter(product_id=self.get_object()).aggregate(avg_price=Avg('price'))
Then I pulled it in to my templates using high.price and high.date_seen, etc.