Develop Reference

Django - Can I add a calculated field that only exists for a particular sub-set or occurences of my model?

Imagine that you have a model with some date-time fields that can be categorized depending on the date. You make an annotation for the model with different cases that assign a different 'status' depending on the calculation for the date-time fields:
#Models.py
class Status(models.TextChoices):
status_1 = 'status_1'
status_2 = 'status_2'
status_3 = 'status_3'
special_status = 'special_status'
class MyModel(models.Model):
important_date_1 = models.DateField(null=True)
important_date_2 = models.DateField(null=True)
calculated_status = models.CharField(max_length=32, choices=Status.choices, default=None, null=True, blank=False,)
objects = MyModelCustomManager()
And the manager with which to do the calculation as annotations:
# managers.py
class MyModelCustomManager(models.Manager):
def get_queryset(self):
queryset = super().get_queryset().annotate(**{
'status': Case(
When(**{'important_date_1' is foo, 'then':
Value(Status.status_1)}),
When(**{'important_date_2' is fii, 'then':
Value(Status.status_2)}),
When(**{'important_date_1' is foo AND 'importante_date_2' is whatever, 'then':
Value(Status.status_3)}),
# And so on and so on
)
}
)
return queryset
Now, here's where it gets tricky. Only one of these sub-sets of occurrences on the model requires an ADDITIONAL CALCULATED FIELD that literally only exists for it, that looks something like this:
special_calculated_field = F('important_date_1') - F('importante_date_2') #Only for special_status
So, basically I want to make a calculated field with the condition that the model instance must belong to this specific status. I don't want to make it an annotation, because other instances of the model would always have this value set to Null or empty if it were a field or annotation and I feel like it would be a waste of a row in the database.
Is there way, for example to do this kind of query:
>>> my_model_instance = MyModel.objects.filter(status='special_status')
>>> my_model_instance.special_calculated_field
Thanks a lot in advance if anyone can chime in with some help.

Django Left join how

I fairly new to Django and stuck with creating a left join in Django. I tried so many, but none of them seems to be working:
The query I want to translate to Django is:
select ssc.id
,mgz.Title
,tli.id
,tli.Time
from Subscription ssc
join Person prs
on ssc.PersonID = prs.id
and prs.id = 3
join Magazine mgz
on mgz.id = ssc.MagazineID
and mgz.from <= date.today()
and mgz.until > date.today()
left join TimeLogedIn tli
on tli.SubscriptionID = ssc.id
and tli.DateOnline = date.today()
The model I'm using looks like this:
class Magazine(models.Model):
Title = models.CharField(max_length=100L)
from = models.Datefield()
until = models.Datefield()
Persons = models.ManyToManyField(Person, through='Subscription')
class Person(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
Magazines = models.ManyToManyField(Magazine, through='Subscription')
class Subscription(models.Model):
MagazineID = models.ForeignKey(Magazine,on_delete=models.CASCADE)
PersonID = models.ForeignKey(Person,on_delete=models.CASCADE)
class TimeLogedIn(models.Model):
SubscriptionID = models.ForeignKey('Subscription', on_delete=models.CASCADE)
DateOnline = models.DateField()
Time = models.DecimalField(max_digits=5, decimal_places=2)
Like I said, tried so many but no succes and now I don't know how to do this in Django ORM , is it even possible? I created already a raw-query and this is working ok, but how to create this in Django ORM?

You can use field lookups lte and gt to filter your objects and then values() method.
You can also querying in the opposite direction and use Q objects for null values:
from django.db.models import Q
Subscription.objects.filter(
PersonID_id=3,
MagazineID__from__lte=date.today(),
MagazineID__until__gt=date.today()
).filter(
Q(TimeLogedIn__DateOnline=date.today()) | Q(TimeLogedIn__DateOnline__isnull=True)
).values("id", "MagazineID__Title", "TimeLogedIn__id", "TimeLogedIn__Time")
OR from TimeLogedIn:
TimeLogedIn.objects.filter(DateOnline=date.today()).filter(
SubscriptionID__MagazineID__from__lte=date.today(),
SubscriptionID__MagazineID__util__gt=date.today()
).values(
"SubscriptionID_id", "SubscriptionID__MagazineID__Title", "id", "Time"
)
Querysets also have the query attribute that contains the sql query to be executed, you can see it like following:
print(TimeLogedIn.objects.filter(...).values(...).query)
Note: Behind the scenes, Django appends "_id" to the field name to create its database column name. Therefore it should be
subscription, instead of SubscriptionID.
You can also use prefetch_related() and select_related() to prevent multiple database hits:
SubscriptionID.objects.filter(...).prefetch_related("TimeLogedIn_set")
SubscriptionID.objects.filter(...).select_related("PersonID")

Django: Search results with django-tables2 and django-filter

I'd like to retrieve a model's objects via a search form but add another column for search score. I'm unsure how to achieve this using django-tables2 and django-filter.
In the future, I'd like the user to be able to use django-filter to help filter the search result. I can access the form variables from PeopleSearchListView but perhaps it's a better approach to integrate a django form for form handling?
My thought so far is to handle to the get request in get_queryset() and then modify the queryset before it's sent to PeopleTable, but adding another column to the queryset does not seem like a standard approach.
tables.py
class PeopleTable(tables.Table):
score = tables.Column()
class Meta:
model = People
template_name = 'app/bootstrap4.html'
exclude = ('id',)
sequence = ('score', '...')
views.py
class PeopleFilter(django_filters.FilterSet):
class Meta:
model = People
exclude = ('id',)
class PeopleSearchListView(SingleTableMixin, FilterView):
table_class = PeopleTable
model = People
template_name = 'app/people.html'
filterset_class = PeopleFilter
def get_queryset(self):
p = self.request.GET.get('check_this')
qs = People.objects.all()
####
# Run code to score users against "check_this".
# The scoring code I'm using is complex, so below is a simpler
# example.
# Modify queryset using output of scoring code?
####
for person in qs:
if person.first_name == 'Phil' and q == 'Hey!':
score = 1
else:
score = 0
return qs
urls.py
urlpatterns = [
...
path('search/', PeopleSearchListView.as_view(), name='search_test'),
... ]
models.py
class People(models.model):
first_name = models.CharField(max_length=200)
last_name = models.CharField(max_length=200)
Edit:
The scoring algorithm is a bit more complex than the above example. It requires a full pass over all of the rows in the People table to generate a score matrix, before finally comparing each scored row with the search query. It's not a one-off score. For example:
def get_queryset(self):
all = []
for person in qs:
all.append(person.name)
# Do something complex with all,
# e.g., measure cosine distance between every person,
# and finally compare to the get request
scores = measure_cosine(all, self.request.GET.get('check_this'))
# We now have the scores for each person.

So you can add extra columns when you initialise the table.
I've got a couple of tables which do this based on events in the system;
def __init__(self, *args, **kwargs):
"""
Override the init method in order to add dynamic columns as
we need to declare one column per existent event on the system.
"""
extra_columns = []
events = Event.objects.filter(
enabled=True,
).values(
'pk', 'title', 'city'
)
for event in events:
extra_columns.append((
event['city'],
MyColumn(event_pk=event['pk'])
))
if extra_columns:
kwargs.update({
'extra_columns': extra_columns
})
super().__init__(*args, **kwargs)
So you could add your score column similar to this when a score has been provided. Perhaps passing your scores into the table from the view so you can identify they're present and add the column, then use the data when rendering the column.
extra_columns doesn't appear to be in the tables2 docs, but you can find the code here; https://github.com/jieter/django-tables2/blob/master/django_tables2/tables.py#L251

When you define a new column for django-tables2 which is not included in table data or queryset, you should provide a render method to calculate it's value.
You don't have to override get_queryset if a complex filtering, preprocess or join required.
In your table class:
class PeopleTable(tables.Table):
score = tables.Column(accessor="first_name")
class Meta:
model = People
def render_score(self, record):
return 1 if record["first_name"] == "Phil" and q == "Hey!" else 0
In your view you can override and provide complex data as well as special filtering or aggregates with get_context_data:
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context["filter"] = self.filter
aggs = {
"score": Function("..."),
"other": Sum("..."),
}
_data = (
People.objects.filter(**params)
.values(*values)
.annotate(**aggs)
.order_by(*values)
.distinct()
)
df = pandas.DataFrame(_data)
df = df....
chart_data = df.to_json()
data = df.to_dict()...
self.table = PeopleTable(data)
context["table"] = self.table
context['chart_data']=chart_data
return context

Perform lookup and update within a single Django query

I have two models: MetaModel and RelatedModel. I want to include the result of a RelatedModel lookup within a MetaModel query, and I'd like to do this within a single DB call.
I've tried to define a 'subquery' QuerySet for use in the main query, but that hasn't worked - it's still making two queries to complete the operation.
Note: I can't use a traditional ForeignKey relationship because the profile_id field is not unique. Uniqueness is a combination of profile_id and channel. This is an aggregation table and profile_id is not guaranteed to be unique across multiple third-party channels.
Any suggestions?
Models:
class Channel(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(
max_length=25,
)
class MetaModel(models.Model):
profile_id = fields.IntegerField()
channel = fields.ForeignKey(Channel))
metadata = fields.TextField()
class RelatedModel(models.Model):
related_id = fields.IntegerField()
profile_id = fields.IntegerField()
channel = fields.ForeignKey(Channel))
Dummy data
channel = Channel("Web site A")
channel.save()
sample_meta = MetaModel(profile_id=1234, channel=channel)
sample_related = RelatedModel(profile_id=1234, related_id=5678, channel=channel)
Query:
# Create a queryset to filter down to the single record we need the `profile_id` for
# I've limited to the only field we need via a `values` operation
related_qs = RelatedAccount.objects.filter(
related_id=5678,
channel=channel
).values_list("profile_id", flat=True)
# I'm doing an update_or_create as there is other data to store, not included for brevity
obj, created = MetaModel.objects.update_or_create(
profile_id=related_qs.first(), # <<< This var is the dynamic part of the query
channel=channel,
defaults={"metadata": "Metadata is added to a new or existing record."}
)

Regarding your note on uniqueness, you can use unique_together option in Django as described here in the documentation.
class MetaModel(models.Model):
profile_id = fields.ForeignKey(RelatedModel)
channel = fields.ForeignKey(Channel)
metadata = fields.TextField()
class Meta:
unique_together = ('profile_id', 'channel')
Then you can change your query accordingly and should solve your problem.

Can't sort table by model's property (accessor set)

I can't sort table by it's models property. I know that I should set accessor in the column so django-tables2 knows what field to process but it does not work.
This is the table:
class ScansTable(tables.Table):
site = tables.columns.Column(accessor='occurence.site', verbose_name='Site')
url = tables.columns.TemplateColumn("""{{ record.occurence.url|truncatechars:20 }}""",
accessor='occurence.url', verbose_name='Url')
price = tables.columns.TemplateColumn(u"""{{ record.price }} €""")
date = tables.columns.Column(accessor='date',order_by='date')
time = tables.columns.Column(accessor='time',order_by='time')
class Meta:
model = Scan
fields = ('date', 'time', 'site', 'url', 'valid', 'price')
attrs = {'id': 'cans_table',
'class': 'table',}
This is the Scan model:
class Scan(models.Model):
occurence = models.ForeignKey('Occurence', related_name='scans')
datetime = models.DateTimeField()
price = models.DecimalField(max_digits=20,decimal_places=2,null=True,blank=True,verbose_name='Price')
valid = models.BooleanField(default=True,verbose_name='Valid')
def __unicode__(self):
return u'{} {} {} {}'.format(self.occurence, self.datetime, self.price, u'OK' if self.valid else 'NOK')
#property
def date(self):
return self.datetime.date()
#property
def time(self):
return self.datetime.time()
The view:
def scans(request):
...
scans = Scan.objects.filter(occurence__product=product)
scans_table = ScansTable(scans)
RequestConfig(request).configure(scans_table)
scans_table.paginate(page=request.GET.get('page', 1), per_page=50)
return render(request,"dashboard_app/scans.html",context={'scans_table':scans_table})
The table is being properly renderd when I don't want to sort it. When I click on time (for example), it returns:
Cannot resolve keyword u'time' into field. Choices are: datetime,
groups, id, occurence, occurence_id, price, valid
Do you know where is the problem?

it's strange what the type product ?? you show the Occurence model and what value it in the view

It appears that defined properties/methods of the model are not available for sorting/filtering within the queryset. I don't fully understand why that is the case. A solution would be to NOT define date and time as properties on the Scan model, but instead annotate them to the queryset used to populate the data.
from django.db import models
def scans(request):
...
scans = Scan.objects.filter(occurence__product=product).annotate(
date=models.F('datetime__date'),
time=models.F('datetime__time')
)
...
See the documentation here on field lookups. Also you could use the tables specific columns for those fields - note that you don't need to define the accessors now the results are already in the queryset:
class ScansTable(tables.Table):
...
date = tables.DateColumn()
time = tables.TimeColumn()
...