Develop Reference

Django: Search results with django-tables2 and django-filter

I'd like to retrieve a model's objects via a search form but add another column for search score. I'm unsure how to achieve this using django-tables2 and django-filter.
In the future, I'd like the user to be able to use django-filter to help filter the search result. I can access the form variables from PeopleSearchListView but perhaps it's a better approach to integrate a django form for form handling?
My thought so far is to handle to the get request in get_queryset() and then modify the queryset before it's sent to PeopleTable, but adding another column to the queryset does not seem like a standard approach.
tables.py
class PeopleTable(tables.Table):
score = tables.Column()
class Meta:
model = People
template_name = 'app/bootstrap4.html'
exclude = ('id',)
sequence = ('score', '...')
views.py
class PeopleFilter(django_filters.FilterSet):
class Meta:
model = People
exclude = ('id',)
class PeopleSearchListView(SingleTableMixin, FilterView):
table_class = PeopleTable
model = People
template_name = 'app/people.html'
filterset_class = PeopleFilter
def get_queryset(self):
p = self.request.GET.get('check_this')
qs = People.objects.all()
####
# Run code to score users against "check_this".
# The scoring code I'm using is complex, so below is a simpler
# example.
# Modify queryset using output of scoring code?
####
for person in qs:
if person.first_name == 'Phil' and q == 'Hey!':
score = 1
else:
score = 0
return qs
urls.py
urlpatterns = [
...
path('search/', PeopleSearchListView.as_view(), name='search_test'),
... ]
models.py
class People(models.model):
first_name = models.CharField(max_length=200)
last_name = models.CharField(max_length=200)
Edit:
The scoring algorithm is a bit more complex than the above example. It requires a full pass over all of the rows in the People table to generate a score matrix, before finally comparing each scored row with the search query. It's not a one-off score. For example:
def get_queryset(self):
all = []
for person in qs:
all.append(person.name)
# Do something complex with all,
# e.g., measure cosine distance between every person,
# and finally compare to the get request
scores = measure_cosine(all, self.request.GET.get('check_this'))
# We now have the scores for each person.

So you can add extra columns when you initialise the table.
I've got a couple of tables which do this based on events in the system;
def __init__(self, *args, **kwargs):
"""
Override the init method in order to add dynamic columns as
we need to declare one column per existent event on the system.
"""
extra_columns = []
events = Event.objects.filter(
enabled=True,
).values(
'pk', 'title', 'city'
)
for event in events:
extra_columns.append((
event['city'],
MyColumn(event_pk=event['pk'])
))
if extra_columns:
kwargs.update({
'extra_columns': extra_columns
})
super().__init__(*args, **kwargs)
So you could add your score column similar to this when a score has been provided. Perhaps passing your scores into the table from the view so you can identify they're present and add the column, then use the data when rendering the column.
extra_columns doesn't appear to be in the tables2 docs, but you can find the code here; https://github.com/jieter/django-tables2/blob/master/django_tables2/tables.py#L251

When you define a new column for django-tables2 which is not included in table data or queryset, you should provide a render method to calculate it's value.
You don't have to override get_queryset if a complex filtering, preprocess or join required.
In your table class:
class PeopleTable(tables.Table):
score = tables.Column(accessor="first_name")
class Meta:
model = People
def render_score(self, record):
return 1 if record["first_name"] == "Phil" and q == "Hey!" else 0
In your view you can override and provide complex data as well as special filtering or aggregates with get_context_data:
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context["filter"] = self.filter
aggs = {
"score": Function("..."),
"other": Sum("..."),
}
_data = (
People.objects.filter(**params)
.values(*values)
.annotate(**aggs)
.order_by(*values)
.distinct()
)
df = pandas.DataFrame(_data)
df = df....
chart_data = df.to_json()
data = df.to_dict()...
self.table = PeopleTable(data)
context["table"] = self.table
context['chart_data']=chart_data
return context

How to stop SELECT before INSERT with Django REST Framework

I've used Django REST Framework to expose an API which is only used by another service to POST new data. It basically just takes json and inserts it in the DB. That's all.
It's quite a high volume data source (sometimes more than 100 records/second), so I need to tune it a bit.
So I was logging the (PostgreSQL) queries that are run, and I see that every POST gives 3 queries:
2019-10-01 11:09:03.320 CEST [23983] postgres#thedb LOG: statement: SET TIME ZONE 'UTC'
2019-10-01 11:09:03.322 CEST [23983] postgres#thedb LOG: statement: SELECT (1) AS "a" FROM "thetable" WHERE "thetable"."id" = 'a7f74e5c-7cad-4983-a909-49857481239b'::uuid LIMIT 1
2019-10-01 11:09:03.363 CEST [23983] postgres#thedb LOG: statement: INSERT INTO "thetable" ("id", "version", "timestamp", "sensor", [and 10 more fields...]) VALUES ('a7f74e5c-7cad-4983-a909-49857481239b'::uuid, '1', '2019-10-01T11:09:03.313690+02:00'::timestamptz, 'ABC123', [and 10 more fields...])
I tuned the DB for INSERTs to be fast, but SELECTs are slow. So I would like to remove the SELECT from the system. I added this line to the Serializer:
id = serializers.UUIDField(validators=[])
But it still does a SELECT. Does anybody know how I can prevent the SELECT from happening?
For complete info; the full Serializer now looks like this:
import logging
from rest_framework import serializers
from .models import TheData
log = logging.getLogger(__name__)
class TheDataSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = TheData
fields = [
'id',
'version',
'timestamp',
'sensor',
[and 10 more fields...]
]
class TheDataDetailSerializer(serializers.ModelSerializer):
id = serializers.UUIDField(validators=[])
class Meta:
model = TheData
fields = '__all__'
Edit
And as requested by frankie567, the ViewSet:
class TheDataViewSet(DetailSerializerMixin, viewsets.ModelViewSet):
serializer_class = serializers.TheDataSerializer
serializer_detail_class = serializers.TheDataDetailSerializer
queryset = TheData.objects.all().order_by('timestamp')
http_method_names = ['post', 'list', 'get']
filter_backends = [DjangoFilterBackend]
filter_class = TheDataFilter
pagination_class = TheDataPager
def get_serializer(self, *args, **kwargs):
""" The incoming data is in the `data` subfield. So I take it from there and put
those items in root to store it in the DB"""
request_body = kwargs.get("data")
if request_body:
new_request_body = request_body.get("data", {})
new_request_body["details"] = request_body.get("details", None)
request_body = new_request_body
kwargs["data"] = request_body
serializer_class = self.get_serializer_class()
kwargs['context'] = self.get_serializer_context()
return serializer_class(*args, **kwargs)

After some digging, I was able to see where this behaviour comes from. If you look at Django Rest Framework code:
if getattr(model_field, 'unique', False):
unique_error_message = model_field.error_messages.get('unique', None)
if unique_error_message:
unique_error_message = unique_error_message % {
'model_name': model_field.model._meta.verbose_name,
'field_label': model_field.verbose_name
}
validator = UniqueValidator(
queryset=model_field.model._default_manager,
message=unique_error_message)
validator_kwarg.append(validator)
We see that if unique is True (which is in your case, as I guess you defined your UUID field as primary key), DRF adds automatically a UniqueValidator. This validator performs a SELECT request to check if the value doesn't already exist.
It is appended to the ones you are defining in the validators parameter of the field, so that's why what you did has no effect.
So, how do we circumvent this?
First attempt
class TheDataDetailSerializer(serializers.ModelSerializer):
# ... your code
def get_fields(self):
fields = super().get_fields()
fields['id'].validators.pop()
return fields
Basically, we remove the validators of the id field after they have been generated. There are surely more clever ways to do this. It seems to me though that DRF may be too opinionated on this matter.
Second attempt
class TheDataDetailSerializer(serializers.ModelSerializer):
# ... your code
def build_standard_field(self, field_name, model_field):
field_class, field_kwargs = super().build_standard_field(field_name, model_field)
if field_name == 'id':
field_kwargs['validators'] = []
return field_class, field_kwargs
When generating the field arguments, set an empty validators list if we are generating the id field.

Accessing child from parent

I have the models defined below:
class PrimaryAsset(models.Model):
title = models.Charfield(max_length=200)
class Service(PrimaryAsset):
description = models.Charfield(max_length=200)
class Website(PrimaryAsset):
url = models.Charfield(max_length=200)
class AssetLinks(models.model):
high = models.ForeignKey(PrimaryAsset)
low = models.ForeignKey(PrimaryAsset)
AssetLinks.objects.filter(high=212)[0].low
When I do the filter above, how can I know which instance the objects is (website or service)? Also, is there a way to avoid an N+1 query using prefetch_related in a way that it gets all the child information as well?

You can even use select_related instead of prefetch_related. Something like this should do the trick:
asset = AssetLinks.objects.filter(high=212).select_related(
'high__service', 'high__website',
'low__service', 'low__website',
)[0]
#check for service/website
service = getattr(asset.high, 'service', None)
website = getattr(asset.high, 'website', None)

Django Rest Framework writable nested serializer with multiple nested objects

I'm trying to create a writable nested serializer. My parent model is Game and the nested models are Measurements. I am trying to post this data to my DRF application using AJAX. However, when try to post the data, the nested Measurements are empty OrderedDict().
Here are my models:
class Game(models.Model):
start_timestamp = models.DateTimeField(auto_now_add=False)
end_timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
class Measurement(models.Model):
game = models.ForeignKey(Game, on_delete=models.PROTECT, related_name='measurements')
measurement_type = models.CharField(max_length=56)
measurement = models.CharField(max_length=56)
timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
Here are my serializers:
class MeasurementSerializer(serializers.ModelSerializer):
timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']), required=False)
class Meta:
model = Measurement
fields = ('measurement_type', 'measurement', 'timestamp')
class GameSerializer(serializers.ModelSerializer):
start_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
end_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
measurements = MeasurementSerializer(many=True)
class Meta:
model = Game
fields = ('id', 'start_timestamp', 'end_timestamp', 'measurements')
def create(self, validated_data):
measurements = validated_data.pop('measurements')
game = Game.objects.create(**validated_data)
for measurement in measurements:
Measurement.objects.create(game=game, **measurement)
return game
My view for Game is the following:
class GameList(generics.ListCreateAPIView):
queryset = Game.objects.all()
serializer_class = GameSerializer
I followed this tutorial for the structure.
I am trying to post to this API via AJAX, the code below:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {
"start_timestamp": "2016-02-16 14:51:43.000000",
"end_timestamp": "2016-02-16 14:53:43.000000",
"measurements":[
{'measurement_type':'type1', 'measurement':'71', 'timestamp':'2016-02-16 14:53:43.000000'},
{'measurement_type':'type1', 'measurement':'72', 'timestamp':'2016-02-16 14:54:43.000000'},
{'measurement_type':'type1', 'measurement':'73', 'timestamp':'2016-02-16 14:55:43.000000'},
]
},
type: 'POST'
})
.error(function(r){})
.success(function(data){})
});
On posting this data, I find in the create method within the GameSerializer that the validate_data.pop('measurements') contains a list of 3 ordered dictionaries (OrderedDict()) that are empty.
UPDATE: I've found that that the initial_data coming in via request.data is structured like so:
'emotion_measurements[0][measurement_type]' (4397175560) = {list} ['type1']
'emotion_measurements[0][measurement]' (4397285512) = {list} ['71']
'emotion_measurements[0][timestamp]' (4397285600) = {list} ['2016-02-16 14:53:43.000000']
'emotion_measurements[1][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[1][measurement]' (4397285864) = {list} ['72']
'emotion_measurements[1][timestamp]' (4397285952) = {list} ['2016-02-16 14:54:43.000000']
'emotion_measurements[2][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[2][measurement]' (4397285864) = {list} ['73']
'emotion_measurements[2][timestamp]' (4397285952) = {list} ['2016-02-16 14:55:43.000000']
Has anyone encountered this issue before? Thanks!
UPDATE #2
I was able to resolve this (although I believe it is more of a workaround than a solution) by adding the following to my MeasurementSerializer:
def to_internal_value(self, data):
formatted_data = json.dumps(data)
formatted_data = formatted_data.replace("[", "").replace("]","")
formatted_data = json.loads(formatted_data)
return formatted_data
The Measurement data coming in was a QueryDict when I believe I needed a Dict. There were also some extra brackets around the key and values so I had to remove those as well.
Still seeking a better answer than this!

The problem here is on the front-end side. By default the server interprets the data as application/x-www-form-urlencoded and in order for it to understand that you are sending it a json, you need to specify the contentType in your $.ajax request:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {...},
contentType: 'application/json; charset=UTF-8', // add this line
type: 'POST'
})
.error(function(r){})
.success(function(data){});
Now your validated_data.pop('measurements') in create() method of your GameSerializer should yield three objects with your measurements (but don't forget to redo your workaround from Update#2).

Django Tastypie throws a 'maximum recursion depth exceeded' when full=True on reverse relation.

I get a maximum recursion depth exceeded if a run the code below:
from tastypie import fields, utils
from tastypie.resources import ModelResource
from core.models import Project, Client
class ClientResource(ModelResource):
projects = fields.ToManyField(
'api.resources.ProjectResource', 'project_set', full=True
)
class Meta:
queryset = Client.objects.all()
resource_name = 'client'
class ProjectResource(ModelResource):
client = fields.ForeignKey(ClientResource, 'client', full=True)
class Meta:
queryset = Project.objects.all()
resource_name = 'project'
# curl http://localhost:8000/api/client/?format=json
# or
# curl http://localhost:8000/api/project/?format=json
If a set full=False on one of the relations it works. I do understand why this is happening but I need both relations to bring data, not just the "resource_uri". Is there a Tastypie way to do it? I managed to solve the problem creating a serialization method on my Project Model, but it is far from elegant. Thanks.

You would have to override full_dehydrate method on at least one resource to skip dehydrating related resource that is causing the recursion.
Alternatively you can define two types of resources that use the same model one with full=Trueand another with full=False.

Thanks #astevanovic pointing the right direction.
I found that overriding dehydrate method to process only some specified fields is a bit less tedious than overriding full_hydrate method to skip fields.
In the pursuit of reusability, I came up with the following code snippets. Hope it would be useful to some:
class BeeModelResource(ModelResource):
def dehydrate(self, bundle):
bundle = super(BeeModelResource, self).dehydrate(bundle)
bundle = self.dehydrate_partial(bundle)
return bundle
def dehydrate_partial(self, bundle):
for field_name, resource_field in self.fields.items():
if not isinstance(resource_field, RelatedField):
continue
if resource_field.full: # already dehydrated
continue
if not field_name in self._meta.partial_fields:
continue
if isinstance(resource_field, ToOneField):
fk_object = getattr(bundle.obj, resource_field.attribute)
fk_bundle = Bundle(obj=fk_object, request=bundle.request)
fk_resource = resource_field.get_related_resource(fk_object)
bundle.data[field_name] = fk_resource.dehydrate_selected(
fk_bundle, self._meta.partial_fields[field_name]).data
elif isinstance(resource_field, ToManyField):
data = []
fk_objects = getattr(bundle.obj, resource_field.attribute)
for fk_object in fk_objects.all():
fk_bundle = Bundle(obj=fk_object, request=bundle.request)
fk_resource = resource_field.get_related_resource(fk_object)
fk_bundle = fk_resource.dehydrate_selected_fields(
fk_bundle, self._meta.partial_fields[field_name])
data.append(fk_bundle.data)
bundle.data[field_name] = data
return bundle
def dehydrate_selected_fields(self, bundle, selected_field_names):
# Dehydrate each field.
for field_name, field_object in self.fields.items():
# A touch leaky but it makes URI resolution work.
# (borrowed from tastypie.resources.full_dehydrate)
if field_name in selected_field_names and not self.is_special_fields(field_name):
if getattr(field_object, 'dehydrated_type', None) == 'related':
field_object.api_name = self._meta.api_name
field_object.resource_name = self._meta.resource_name
bundle.data[field_name] = field_object.dehydrate(bundle)
bundle.data['resource_uri'] = self.get_resource_uri(bundle.obj)
bundle.data['id'] = bundle.obj.pk
return bundle
#staticmethod
def is_special_fields(field_name):
return field_name in ['resource_uri']
With #sigmus' example, the resources will need 3 modifications:
both resource will use BeeModuleResource as its super class (or, add dehydrate_partial to one resource and dehydrate_selected to the other.)
unset full=True on either of the resource
add partial_fields into the resource Meta the unset resource
```
class ClientResource(BeeModelResource): # make BeeModelResource a super class
projects = fields.ToManyField(
'api.resources.ProjectResource', 'project_set'
) # remove full=True
class Meta:
queryset = Client.objects.all()
resource_name = 'client'
partial_fields = {'projects': ['memo', 'title']} # add partial_fields
class ProjectResource(BeeModelResource): # make BeeModelResource a super class
client = fields.ForeignKey(ClientResource, 'client', full=True)
class Meta:
queryset = Project.objects.all()
resource_name = 'project'

Dead simple solution: set the use_in = 'list' kwarg on both relationship fields!
The docs: http://django-tastypie.readthedocs.org/en/latest/fields.html#use-in