Develop Reference

Best architecture for dynamically validating and saving field

I am looking for the good architecture for my problem. I am using django rest framework for building an API. I receive a list of dict which contains an id and a list of values. The list of values need to be validated according to the id.
Example of my code:
class AttributesSerializer(serializers.Serializer):
id = serializers.PrimaryKeyRelatedField(queryset=Attribute.objects.all(), source="attribute", required=True)
values = serializers.ListField()
def validate(self, validated_data):
attribute = validated_data["attribute"]
values = validated_data["values"]
# This function returns the corresponding field according to attribute
values_child_field = get_values_field(attribute)
self.fields["values"].child = values_child_fields
new_values = self.fields["values"].run_child_validation(values)
set_value(validated_data, "values", new_values)
return validated_data
class BaseObjectApiInputSerializer(serializers.Serializer):
category_id = serializers.PrimaryKeyRelatedField(
queryset=Category.objects.all()
)
attributes = AttributesSerializer(many=True)
I want to parse json like this:
{
"categorty_id": 42, # Category pk of the baseobject. which defines some constraints about attributes available
"attributes": [
{"id": 124, "values": ["value"]},
{"id": 321, "values": [42]},
{
"id": 18,
"values": [
{
"location": {"type": "Point", "geometry": {...}},
"address": "an address",
}
],
},
]
}
Currently, this code does not work. DRF seems to try to revalidate all values entries for each iteration with each child field. I do not understand why... I guess I could make it work without using this fields["values"] for making the validation and just retrieve the field and use it directly, but i need this field for making the save later.
Do you think my architecture is ok? What is the good way for parsing this type of data with DRF?
EDIT:
Structure of models are complex but a version simplified following:
class Attribute(models.Model):
class DataType(models.TextChoices):
TEXT = "TEXT", _("datatype_text")
INTEGER = "INTEGER", _("datatype_integer")
DATETIME = "DATETIME", _("datatype_datetime")
BOOL = "BOOL", _("datatype_bool")
# Some examples, but there are about 30 items with
# type very complicated like RecurrenceRule (RFC2445)
# or GeoJSON type
label = models.CharField()
category = models.ForeignKey(Category)
attribute_type = models.CharField(choices=DataType.choices)
class AttributeValue(models.Model):
attribute = models.ForeignKey(Attribute)
# a model which represents an object with list of attributes
baseobject = models.ForeignKey(BaseObject)
value = models.TextField()
AttributeValue is like a through table for manytomany relation between BaseObject model and Attribute model.
My JSON represents the list of attribute/values attached to a baseobject.
In fact I don't understand why DRf doesn't allow delegating registration in the child serializers of the parent serializer. This would allow much greater flexibility in code architecture and separation of responsibilities.
EDIT 2 :
My urls.py
router = routers.DefaultRouter()
router.register("baseobjects", BaseObjectViewSet, basename="baseobjects")
I am using the default router and url for DRF viewset.
The view looks like:
class BaseObjectViewSet(viewsets.ModelViewSet):
permission_classes = [IsAuthenticated]
authentication_classes = [TokenAuthentication]
def create(self, request, *args, **kwargs):
serializer = BaseObjectApiInputSerializer(
data=request.data
)
if not serializer.is_valid():
return Response(serializer.errors, status=HTTP_400_BAD_REQUEST)
baseobject: BaseObject = serializer.save()
return Response(
{"results": [{"id": baseobject.pk}]}, status=HTTP_200_OK
)

I think you should use ListField with JSONField as child argument for values field.
validators = {
TinyurlShortener.DataType.TEXT: serializers.CharField(),
TinyurlShortener.DataType.INTEGER: serializers.IntegerField(),
TinyurlShortener.DataType.DATETIME: serializers.DateTimeField(),
TinyurlShortener.DataType.BOOL: serializers.BooleanField(),
}
class AttributesSerializer(serializers.Serializer):
id = serializers.PrimaryKeyRelatedField(queryset=Attribute.objects.all(), source="attribute", required=True)
values = serializers.ListField(
child=serializers.JSONField()
)
def validate(self, attrs):
attribute = attrs.get('id')
field = validators[attribute.attribute_type]
for v in attrs['values']:
field.run_validation(json.loads(v.replace("'", '"')))
return super().validate(attrs)
class BaseObjectApiInputSerializer(serializers.Serializer):
category_id = serializers.PrimaryKeyRelatedField(
queryset=Category.objects.all()
)
attributes = AttributesSerializer(many=True)

Override JSON data in Django RestFramwork

Right now with the current Views functions I am getting the data given below:
{"item": "zxmnb",
"category": "zxc",
"price": "zxc",
"restaurant": 1}
Here is my views file:
class RestaurantMenuView(generics.RetrieveAPIView):
lookup_field = 'item'
serializer_class = MenuSerializer
def get_queryset(self):
return Menu.objects.all()
But the issue is I want the data to be in a format as:
{"restaurant": "name"
"item":"some item",
"category": "some category",
"price": "some price"
}
I want to mention that Restaurant is another model in my models class,Now I know that if I use restaurant I will only get the pk. But what I want is the JSON to be displayed like that.

You need to modify your MenuSerializer. Specifically, you need to change the restaurant field to be a CharField and also provide a source attribute. Something like the following:
class MenuSerializer(serializers.ModelSerializer):
restaurant = serializers.CharField(source='restaurant.name')
# ... other stuff in the serializer
Here, I am assuming that your Restaurant model has a name field.
You can read more about Serializer fields here: https://www.django-rest-framework.org/api-guide/fields/

Why don't you redefine the to_representation() function. Something like this:
class MenuSerializer(serializers.ModelSerializer):
def to_representation(self, obj):
restaurants = RestaurantSerializer(instance=obj,
context=self.context).data
data = []
for restaurant in restaurants:
data.append(
{
"restaurant": {
"item": obj.item,
"category": obj.category,
"price": obj.price,
"name": restaurant.name,
}
}
)
return data
Without looking at your models or why you want restaurant in there, I added the for loop in order to show you that you can pretty much access any of the data in your to_representation() and put it in any sort of format you want. I use this when I'm trying to render my JSON objects into XML in a specific way. Hope this helps.
Also check out the documentation:
https://www.django-rest-framework.org/api-guide/serializers/#overriding-serialization-and-deserialization-behavior
Another solution that you could consider is adding a Foreign Key on your Menu model back to the Restaurant, then you could define your serializer like this:
class RestaurantSerializer(serializers.ModelSerializer):
menu = MenuViewSerializer(read_only=True, many=True)
class Meta:
model = Restaurant
fields = [
"id",
"name",
"menu",
]
extra_kwargs = {
"menu": {"read_only": True},
}

Serialize OneToMany relation into array in Django

Suppose I have some django models:
class Restaurant(Model):
name = CharField(max_length=200)
class Food(Model):
restaurant = ForeignKey(Restaurant, on_delete=CASCADE)
name = CharField(max_length=200)
One Restaurant can have multiple foods.
And I want my json to look like this:
[
{
"name": "xxx",
"food": [{"name": "xxx"}, {"name": "xxx"}, ...]
}, ...
]
I can use something like:
restaurants = Restaurant.objects.all().values()
for restaurant in restaurants:
restaurant['food'] = Food.objects.filter(restaurant__id=restaurant['id']).values()
But this loses some optimization in SQL level, as this is using application level loop.
Is there a better way to do this?

check this
restaurant = Restaurant.objects.all()
result = []
for rest in restaurant:
data = {
'name': rest.name,
'food': rest.food_set.all().values('name')
}
result.appen(data)
Try it....
User django rest framework. using serialzers you can do it more effectively

#Nakamura You are correct, query inside the loop is not a good way of coding.
below is one way of solving
restaurants = Restaurant.objects.all().values()
foods = Food.objects.all()
for restaurant in restaurants:
restaurant['food'] = foods.filter(restaurant__id=restaurant['id']).values()
As you gave required JSON format i suggested this answer.

Convert nested dict/json to a django ORM model, without hard coding the data structure

I want to import data from json files into my django db. The json contains nested objects.
Current steps are:
Set up my django object models to match the json schema (done manually - see models.py file below)
Import json file into python dict using mydict = json.loads(file.read()) (done)
Convert dict to django models (done - but solution is not pretty)
Is there a way I can convert my nested dict into django models (i.e. step 3) without hard-coding the data structure into the logic?
Bonus points for automatically generating the django models (i.e. the models.py file) based on an example json file.
Thanks in advance!
How I'm currently doing it
Step 3 is easy if the dict does not contain any nested dicts - just construct a new object from the dict i.e. MyModel.objects.create(**mydict) or use django fixtures.
However, because my json/dict contains nested objects, I'm currently doing step 3 like this:
# read the json file into a python dict
d = json.loads(myfile.read())
# construct top-level object using the top-level dict
# (excluding nested lists of dicts called 'judges' and 'contestants')
c = Contest.objects.create(**{k:v for k,v in d.items() if k not in ('judges', 'contestants')})
# construct nested objects using the nested dicts
for judge in d['judges']:
c.judge_set.create(**judge)
for contestant in d['contestants']:
ct = c.contestant_set.create(**{k:v for k,v in contestant.items() if k not in ('singers', 'songs')})
# all contestants sing songs
for song in contestant['songs']:
ct.song_set.create(**song)
# not all contestants have a list of singers
if 'singers' in contestant:
for singer in contestant['singers']:
ct.singer_set.create(**singer)
This works, but requires the data structure to be hard coded into the logic:
Need to hard code the names of the nested dicts to exclude when calling create() (if you try to pass a nested dict to create() it throws a TypeError). I thought about instead doing **{k:v for k,v in contestant.items() if not hasattr(v, 'pop')} to exclude lists and dicts, but I suspect that won't work 100% of the time.
Need to hard code logic to iteratively create the nested objects
Need to hard code logic to deal with nested objects that are not always present
Data structures
example json looks like this:
{
"assoc": "THE BRITISH ASSOCIATION OF BARBERSHOP SINGERS",
"contest": "QUARTET FINAL (NATIONAL STREAM)",
"location": "CHELTENHAM",
"year": "2007/08",
"date": "25/05/2008",
"type": "quartet final",
"filename": "BABS/2008QF.pdf"
"judges": [
{"cat": "m", "name": "Rod"},
{"cat": "m", "name": "Bob"},
{"cat": "p", "name": "Pat"},
{"cat": "p", "name": "Bob"},
{"cat": "s", "name": "Mark"},
{"cat": "s", "name": "Barry"},
{"cat": "a", "name": "Phil"}
],
"contestants": [
{
"prev_tot_score": "1393",
"tot_score": "2774",
"rank_m": "1",
"rank_s": "1",
"rank_p": "1",
"rank": "1", "name": "Monkey Magic",
"pc_score": "77.1",
"songs": [
{"title": "Undecided Medley","m": "234","s": "226","p": "241"},
{"title": "What Kind Of Fool Am I","m": "232","s": "230","p": "230"},
{"title": "Previous","m": "465","s": "462","p": "454"}
],
"singers": [
{"part": "tenor","name": "Alan"},
{"part": "lead","name": "Zac"},
{"part": "bari","name": "Joe"},
{"part": "bass","name": "Duncan"}
]
},
{
"prev_tot_score": "1342",
"tot_score": "2690",
"rank_m": "2",
"rank_s": "2",
"rank_p": "2",
"rank": "2", "name": "Evolution",
"pc_score": "74.7",
"songs": [
{"title": "It's Impossible","m": "224","s": "225","p": "218"},
{"title": "Come Fly With Me","m": "225","s": "222","p": "228"},
{"title": "Previous","m": "448","s": "453","p": "447"}
],
"singers": [
{"part": "tenor","name": "Tony"},
{"part": "lead","name": "Michael"},
{"part": "bari","name": "Geoff"},
{"part": "bass","name": "Stuart"}
]
},
],
}
My models.py file:
from django.db import models
# Create your models here.
class Contest(models.Model):
assoc = models.CharField(max_length=100)
contest = models.CharField(max_length=100)
date = models.DateField()
filename = models.CharField(max_length=100)
location = models.CharField(max_length=100)
type = models.CharField(max_length=20)
year = models.CharField(max_length=20)
class Judge(models.Model):
contest = models.ForeignKey(Contest, on_delete=models.CASCADE)
name = models.CharField(max_length=60)
cat = models.CharField('Category', max_length=2)
class Contestant(models.Model):
contest = models.ForeignKey(Contest, on_delete=models.CASCADE)
name = models.CharField(max_length=100)
tot_score = models.IntegerField('Total Score')
rank_m = models.IntegerField()
rank_s = models.IntegerField()
rank_p = models.IntegerField()
rank = models.IntegerField()
pc_score = models.DecimalField(max_digits=4, decimal_places=1)
# optional fields
director = models.CharField(max_length=100, blank=True, null=True)
size = models.IntegerField(blank=True, null=True)
prev_tot_score = models.IntegerField(blank=True, null=True)
class Song(models.Model):
contestant = models.ForeignKey(Contestant, on_delete=models.CASCADE)
title = models.CharField(max_length=100)
m = models.IntegerField('Music')
s = models.IntegerField('Singing')
p = models.IntegerField('Performance')
class Singer(models.Model):
contestant = models.ForeignKey(Contestant, on_delete=models.CASCADE)
name = models.CharField(max_length=100)
part = models.CharField('Category', max_length=5)

You could browse the json object recursively and use a key to class mapping to instantiate your models dynamically. Here's an idea (not a working solution!):
key_model = {
"contestants": Contestant,
"singers": Singer
}
def make_sub_model(parent, model, vals):
for v in vals:
child = create_model(model, v)
parent.add_child(child) # or whatever it is with Django Models
def create_model(model, obj):
# model should be the class and obj a dict
# take care of the top lvl object
to_process = [] # store nest models
parent = {} # store parent attributes
for k, v in obj.items():
if isinstance(v, list): # you probably want dict as well
to_process.append((k, v))
else:
parent[k] = v
parent_obj = model.create(**parent)
# now process the chidlrend
for k, v in to_process:
make_sub_model(parent_obj, key_model[k], v)
return parent_obj
But in the end, I would discourage this because you are using a Schema based storage (SQL) so your code should enforce that the input matches your schema (you can't handle anything different on the fly anyway). If you don't care about having a schema at all go for a No-SQL solution and you won't have this problem. Or a hybrid like PostgresSQL.

How can one customize Django Rest Framework serializers output?

I have a Django model that is like this:
class WindowsMacAddress(models.Model):
address = models.TextField(unique=True)
mapping = models.ForeignKey('imaging.WindowsMapping', related_name='macAddresses')
And two serializers, defined as:
class WindowsFlatMacAddressSerializer(serializers.Serializer):
address = serializers.Field()
class WindowsCompleteMappingSerializer(serializers.Serializer):
id = serializers.Field()
macAddresses = WindowsFlatMacAddressSerializer(many=True)
clientId = serializers.Field()
When accessing the serializer over a view, I get the following output:
[
{
"id": 1,
"macAddresses": [
{
"address": "aa:aa:aa:aa:aa:aa"
},
{
"address": "bb:bb:bb:bb:bb:bb"
}
],
"clientId": null
}
]
Almost good, except that I'd prefer to have:
[
{
"id": 1,
"macAddresses": [
"aa:aa:aa:aa:aa:aa",
"bb:bb:bb:bb:bb:bb"
],
"clientId": null
}
]
How can I achieve that ?

Create a custom serializer field and implement to_native so that it returns the list you want.
If you use the source="*" technique then something like this might work:
class CustomField(Field):
def to_native(self, obj):
return obj.macAddresses.all()
I hope that helps.
Update for djangorestframework>=3.9.1
According to documentation, now you need override either one or both of the to_representation() and to_internal_value() methods. Example
class CustomField(Field):
def to_representation(self, value)
return {'id': value.id, 'name': value.name}

Carlton's answer will work do the job just fine. There's also a couple of other approaches you could take.
You can also use SlugRelatedField, which represents the relationship, using a given field on the target.
So for example...
class WindowsCompleteMappingSerializer(serializers.Serializer):
id = serializers.Field()
macAddresses = serializers.SlugRelatedField(slug_field='address', many=True, read_only=True)
clientId = serializers.Field()
Alternatively, if the __str__ of the WindowsMacAddress simply displays the address, then you could simply use RelatedField, which is a basic read-only field that will give you a simple string representation of the relationship target.
# models.py
class WindowsMacAddress(models.Model):
address = models.TextField(unique=True)
mapping = models.ForeignKey('imaging.WindowsMapping', related_name='macAddresses')
def __str__(self):
return self.address
# serializers.py
class WindowsCompleteMappingSerializer(serializers.Serializer):
id = serializers.Field()
macAddresses = serializers.RelatedField(many=True)
clientId = serializers.Field()
Take a look through the documentation on serializer fields to get a better idea of the various ways you can represent relationships in your API.