Develop Reference

Best architecture for dynamically validating and saving field

I am looking for the good architecture for my problem. I am using django rest framework for building an API. I receive a list of dict which contains an id and a list of values. The list of values need to be validated according to the id.
Example of my code:
class AttributesSerializer(serializers.Serializer):
id = serializers.PrimaryKeyRelatedField(queryset=Attribute.objects.all(), source="attribute", required=True)
values = serializers.ListField()
def validate(self, validated_data):
attribute = validated_data["attribute"]
values = validated_data["values"]
# This function returns the corresponding field according to attribute
values_child_field = get_values_field(attribute)
self.fields["values"].child = values_child_fields
new_values = self.fields["values"].run_child_validation(values)
set_value(validated_data, "values", new_values)
return validated_data
class BaseObjectApiInputSerializer(serializers.Serializer):
category_id = serializers.PrimaryKeyRelatedField(
queryset=Category.objects.all()
)
attributes = AttributesSerializer(many=True)
I want to parse json like this:
{
"categorty_id": 42, # Category pk of the baseobject. which defines some constraints about attributes available
"attributes": [
{"id": 124, "values": ["value"]},
{"id": 321, "values": [42]},
{
"id": 18,
"values": [
{
"location": {"type": "Point", "geometry": {...}},
"address": "an address",
}
],
},
]
}
Currently, this code does not work. DRF seems to try to revalidate all values entries for each iteration with each child field. I do not understand why... I guess I could make it work without using this fields["values"] for making the validation and just retrieve the field and use it directly, but i need this field for making the save later.
Do you think my architecture is ok? What is the good way for parsing this type of data with DRF?
EDIT:
Structure of models are complex but a version simplified following:
class Attribute(models.Model):
class DataType(models.TextChoices):
TEXT = "TEXT", _("datatype_text")
INTEGER = "INTEGER", _("datatype_integer")
DATETIME = "DATETIME", _("datatype_datetime")
BOOL = "BOOL", _("datatype_bool")
# Some examples, but there are about 30 items with
# type very complicated like RecurrenceRule (RFC2445)
# or GeoJSON type
label = models.CharField()
category = models.ForeignKey(Category)
attribute_type = models.CharField(choices=DataType.choices)
class AttributeValue(models.Model):
attribute = models.ForeignKey(Attribute)
# a model which represents an object with list of attributes
baseobject = models.ForeignKey(BaseObject)
value = models.TextField()
AttributeValue is like a through table for manytomany relation between BaseObject model and Attribute model.
My JSON represents the list of attribute/values attached to a baseobject.
In fact I don't understand why DRf doesn't allow delegating registration in the child serializers of the parent serializer. This would allow much greater flexibility in code architecture and separation of responsibilities.
EDIT 2 :
My urls.py
router = routers.DefaultRouter()
router.register("baseobjects", BaseObjectViewSet, basename="baseobjects")
I am using the default router and url for DRF viewset.
The view looks like:
class BaseObjectViewSet(viewsets.ModelViewSet):
permission_classes = [IsAuthenticated]
authentication_classes = [TokenAuthentication]
def create(self, request, *args, **kwargs):
serializer = BaseObjectApiInputSerializer(
data=request.data
)
if not serializer.is_valid():
return Response(serializer.errors, status=HTTP_400_BAD_REQUEST)
baseobject: BaseObject = serializer.save()
return Response(
{"results": [{"id": baseobject.pk}]}, status=HTTP_200_OK
)

I think you should use ListField with JSONField as child argument for values field.
validators = {
TinyurlShortener.DataType.TEXT: serializers.CharField(),
TinyurlShortener.DataType.INTEGER: serializers.IntegerField(),
TinyurlShortener.DataType.DATETIME: serializers.DateTimeField(),
TinyurlShortener.DataType.BOOL: serializers.BooleanField(),
}
class AttributesSerializer(serializers.Serializer):
id = serializers.PrimaryKeyRelatedField(queryset=Attribute.objects.all(), source="attribute", required=True)
values = serializers.ListField(
child=serializers.JSONField()
)
def validate(self, attrs):
attribute = attrs.get('id')
field = validators[attribute.attribute_type]
for v in attrs['values']:
field.run_validation(json.loads(v.replace("'", '"')))
return super().validate(attrs)
class BaseObjectApiInputSerializer(serializers.Serializer):
category_id = serializers.PrimaryKeyRelatedField(
queryset=Category.objects.all()
)
attributes = AttributesSerializer(many=True)

Filter Embedded Document List in GraphQL

I'm building a GraphQL application in Python/Graphene using a MongoDB backend (through MongoEngine). Everything has been working well, but noticed that there's not a lot documentation for handling nested lists of embedded documents. I thought one power of GraphQL was the ability to project only the properties you want, but it doesn't appear to be the case fully.
Looking at this collection as an example:
[
{
"name": "John Doe",
"age": 37,
"preferences": [
{
"key": "colour",
"value": "Green"
},
{
"key": "smell",
"value": "onions cooking in butter"
},
...
]
},
...
]
If I want to find a particular object through GraphQL, I would look up through a query like
{
person(name: "John Doe"){edges{node{
name age preferences{edges{node{
key value
}}}
}}}
}
But this could bring back hundreds of nested documents. What I would like to do instead is to identify the requested nested documents as part of the projection request.
{
person(name: "John Doe"){edges{node{
name age preferences(key: "colour"){edges{node{
key value
}}}
}}}
}
My understanding reading the GraphQL spec is these sub-queries are not possible, but wanted to confirm with experts first. And if it is possible, how would I implement it to support these types of requests?
Update Maybe a schema example will provide some more insightful responses.
class PreferenceModel(mongoengine.EmbeddedDocument):
key = mongoengine.fields.StringField()
value = mongoengine.fields.StringField()
class Preference(graphene_mongo.MongoengineObjectType):
class Meta:
interfaces = (graphene.relay.Node, )
model = PreferenceModel
class PersonModel(mongoengine.Document):
meta = {'collection': 'persons'}
name = mongoengine.fields.StringField()
age = mongoengine.fields.IntField()
preferences = mongoengine.fields.EmbeddedDocumentListField(PreferenceModel)
class Person(graphene_mongo.MongoengineObjectType):
class Meta:
interfaces = (graphene.relay.Node, )
model = PersonModel
class Query(graphene.ObjectType):
person = graphene_mongo.MongoengineConnectionField(Person)
schema = graphene.Schema(query=Query, types=[Person])
app = starlette.graphql.GraphQLApp(schema=schema)
Using this above structure, what changes would be necessary to allow for queries/filters on nested objects?

I had a similiar issue but working graphene-django. I solved it using custom resolvers on the DjangoObjectType, like this:
import graphene
from graphene_django import DjangoObjectType
from .models import Question, Choice, SubChoice
class SubChoiceType(DjangoObjectType):
class Meta:
model = SubChoice
fields = "__all__"
class ChoiceType(DjangoObjectType):
sub_choices = graphene.List(SubChoiceType, search_sub_choices=graphene.String())
class Meta:
model = Choice
fields = ("id", "choice_text", "question")
def resolve_sub_choices(self, info, search_sub_choices=None):
if search_sub_choices:
return self.subchoice_set.filter(sub_choice_text__icontains=search_sub_choices)
return self.subchoice_set.all()
class QuestionType(DjangoObjectType):
choices = graphene.List(ChoiceType, search_choices=graphene.String())
class Meta:
model = Question
fields = ("id", "question_text")
def resolve_choices(self, info, search_choices=None):
if search_choices:
return self.choice_set.filter(choice_text__icontains=search_choices)
return self.choice_set.all()
class Query(graphene.ObjectType):
all_questions = graphene.List(QuestionType, search_text=graphene.String())
all_choices = graphene.List(ChoiceType, search_text=graphene.String())
all_sub_choices = graphene.List(SubChoiceType)
def resolve_all_questions(self, info, search_text=None):
qs = Question.objects.all()
if search_text:
qs = qs.filter(question_text__icontains=search_text)
return qs
def resolve_all_choices(self, info, search_text=None):
qs = Choice.objects.all()
if search_text:
qs = qs.filter(choice_text__icontains=search_text)
return qs
def resolve_all_sub_choices(self, info):
qs = SubChoice.objects.all()
return qs
schema = graphene.Schema(query=Query)
you can find the example here: https://github.com/allangz/graphene_subfilters/blob/main/mock_site/polls/schema.py
It may work for you

Save related Images Django REST Framework

I have this basic model layout:
class Listing(models.Model):
name = models.TextField()
class ListingImage(models.Model):
listing = models.ForeignKey(Listing, related_name='images', on_delete=models.CASCADE)
image = models.ImageField(upload_to=listing_image_path)
Im trying to write a serializer which lets me add an rest api endpoint for creating Listings including images.
My idea would be this:
class ListingImageSerializer(serializers.ModelSerializer):
class Meta:
model = ListingImage
fields = ('image',)
class ListingSerializer(serializers.ModelSerializer):
images = ListingImageSerializer(many=True)
class Meta:
model = Listing
fields = ('name', 'images')
def create(self, validated_data):
images_data = validated_data.pop('images')
listing = Listing.objects.create(**validated_data)
for image_data in images_data:
ListingImage.objects.create(listing=listing, **image_data)
return listing
My Problems are:
I'm not sure how and if I can send a list of images in a nested dictionary using a multipart POST request.
If I just post an images list and try to convert it from a list to a list of dictionaries before calling the serializer, I get weird OS errors when parsing the actual image.
for key, item in request.data.items():
if key.startswith('images'):
# images.append({'image': item})
request.data[key] = {'image': item}
My request code looks like this:
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
api_token = 'xxxx'
images_data = MultipartEncoder(
fields={
'name': 'test',
'images[0]': (open('lilo.png', 'rb'), 'image/png'),
'images[1]': (open('panda.jpg', 'rb'), 'image/jpeg')
}
)
response = requests.post('http://127.0.0.1:8000/api/listings/', data=images_data,
headers={
'Content-Type': images_data.content_type,
'Authorization': 'Token' + ' ' + api_token
})
I did find a very hacky solution which I will post in the answers but its not really robust and there needs to be a better way to do this.

So my solution is based off of this post and works quite well but seems very unrobust and hacky.
I change the images field from a relation serializer requiring a dictonary to a ListField. Doing this i need to override the list field method to actually create a List out of the RelatedModelManager when calling "to_repesentation".
This baiscally behaves like a list on input, but like a modelfield on read.
class ModelListField(serializers.ListField):
def to_representation(self, data):
"""
List of object instances -> List of dicts of primitive datatypes.
"""
return [self.child.to_representation(item) if item is not None else None for item in data.all()]
class ListingSerializer(serializers.ModelSerializer):
images = ModelListField(child=serializers.FileField(max_length=100000, allow_empty_file=False, use_url=False))
class Meta:
model = Listing
fields = ('name', 'images')
def create(self, validated_data):
images_data = validated_data.pop('images')
listing = Listing.objects.create(**validated_data)
for image_data in images_data:
ListingImage.objects.create(listing=listing, image=image_data)
return listing

Django Rest Framework receive primary key value in POST and return model object as nested serializer

I'm not completely sure that the title of my question is as specific as I wanted it to be, but this is the case:
I have a HyperlinkedModelSerializer that looks like this:
class ParentArrivalSerializer(serializers.HyperlinkedModelSerializer):
carpool = SchoolBuildingCarpoolSerializer()
class Meta:
model = ParentArrival
As you can see the carpool is defined as a nested serializer object and what I want is to be able to make a POST request to create a ParentArrival in this way (data as application/json):
{
...
"carpool": "http://localhost:8000/api/school-building-carpools/10/"
...
}
And receive the data in this way:
{
"carpool": {
"url": "http://localhost:8000/api/school-building-carpools/10/"
"name": "Name of the carpool",
...
}
}
Basically, I'm looking for a way to deal with nested serializers without having to send data as an object (but id or url in this case) in POST request, but receiving the object as nested in the serialized response.

I have been happy with my previous solution, but decided to look again and I think I have another solution that does exactly what you want.
Basically, you need to create your own custom field, and just overwrite the to_representation method:
class CarpoolField(serializers.PrimaryKeyRelatedField):
def to_representation(self, value):
pk = super(CarpoolField, self).to_representation(value)
try:
item = ParentArrival.objects.get(pk=pk)
serializer = CarpoolSerializer(item)
return serializer.data
except ParentArrival.DoesNotExist:
return None
def get_choices(self, cutoff=None):
queryset = self.get_queryset()
if queryset is None:
return {}
return OrderedDict([(item.id, str(item)) for item in queryset])
class ParentArrivalSerializer(serializers.HyperlinkedModelSerializer):
carpool = CarpoolField(queryset=Carpool.objects.all())
class Meta:
model = ParentArrival
This will allow you to post with
{
"carpool": 10
}
and get:
{
"carpool": {
"url": "http://localhost:8000/api/school-building-carpools/10/"
"name": "Name of the carpool",
...
}
}

It's simple.
As you know, Django appends "_id" to the field name in the ModelClass, and you can achieve it in the SerializerClass, and the original filed can also be achieved. All you have to do is like this
class ParentArrivalSerializer(serializers.HyperlinkedModelSerializer):
# ...
carpool_id = serializers.IntegerField(write_only=True)
carpool = SchoolBuildingCarpoolSerializer(read_only=True)
# ...
class Meta:
fields = ('carpool_id', 'carpool', ...)
And use carpool_id in POST request.

How about overriding the to_representation method?
class YourSerializer(serializers.ModelSerializer):
class Meta:
model = ModelClass
fields = ["id", "foreignkey"]
def to_representation(self, instance):
data = super(YourSerializer, self).to_representation(instance)
data['foreignkey'] = YourNestedSerializer(instance.foreignkey).data
return data

One way to do it is to keep 'carpool' as the default you get from DRF, and then add a read-only field for the nested object.
Something like this (I don't have time to test the code, so consider this pseudo-code. If you cannot get it to work, let me know, and will spend more time):
class ParentArrivalSerializer(serializers.HyperlinkedModelSerializer):
carpool_info = serializers.SerializerMethodField(read_only=True)
class Meta:
model = ParentArrival
fields = ('id', 'carpool', 'carpool_info',)
def get_carpool_info(self, obj):
carpool = obj.carpool
serializer = SchoolBuildingCarpoolSerializer(carpool)
return serializer.data
If your only nested object is carpool, I would also suggest switching to the regular ModelSerializer so carpool only shows the ID (10) and the nested object then can show the URL.
class ParentArrivalSerializer(serializers.ModelSerializer):
....
and then if it all works, you will be able to do a post with
{
"carpool": 10
}
and your get:
{
"carpool": 10
"carpool_info": {
"url": "http://localhost:8000/api/school-building-carpools/10/"
"name": "Name of the carpool",
...
}
}
I have never found another solution, so this is the trick I have used several times.

Django Rest Framework writable nested serializer with multiple nested objects

I'm trying to create a writable nested serializer. My parent model is Game and the nested models are Measurements. I am trying to post this data to my DRF application using AJAX. However, when try to post the data, the nested Measurements are empty OrderedDict().
Here are my models:
class Game(models.Model):
start_timestamp = models.DateTimeField(auto_now_add=False)
end_timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
class Measurement(models.Model):
game = models.ForeignKey(Game, on_delete=models.PROTECT, related_name='measurements')
measurement_type = models.CharField(max_length=56)
measurement = models.CharField(max_length=56)
timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
Here are my serializers:
class MeasurementSerializer(serializers.ModelSerializer):
timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']), required=False)
class Meta:
model = Measurement
fields = ('measurement_type', 'measurement', 'timestamp')
class GameSerializer(serializers.ModelSerializer):
start_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
end_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
measurements = MeasurementSerializer(many=True)
class Meta:
model = Game
fields = ('id', 'start_timestamp', 'end_timestamp', 'measurements')
def create(self, validated_data):
measurements = validated_data.pop('measurements')
game = Game.objects.create(**validated_data)
for measurement in measurements:
Measurement.objects.create(game=game, **measurement)
return game
My view for Game is the following:
class GameList(generics.ListCreateAPIView):
queryset = Game.objects.all()
serializer_class = GameSerializer
I followed this tutorial for the structure.
I am trying to post to this API via AJAX, the code below:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {
"start_timestamp": "2016-02-16 14:51:43.000000",
"end_timestamp": "2016-02-16 14:53:43.000000",
"measurements":[
{'measurement_type':'type1', 'measurement':'71', 'timestamp':'2016-02-16 14:53:43.000000'},
{'measurement_type':'type1', 'measurement':'72', 'timestamp':'2016-02-16 14:54:43.000000'},
{'measurement_type':'type1', 'measurement':'73', 'timestamp':'2016-02-16 14:55:43.000000'},
]
},
type: 'POST'
})
.error(function(r){})
.success(function(data){})
});
On posting this data, I find in the create method within the GameSerializer that the validate_data.pop('measurements') contains a list of 3 ordered dictionaries (OrderedDict()) that are empty.
UPDATE: I've found that that the initial_data coming in via request.data is structured like so:
'emotion_measurements[0][measurement_type]' (4397175560) = {list} ['type1']
'emotion_measurements[0][measurement]' (4397285512) = {list} ['71']
'emotion_measurements[0][timestamp]' (4397285600) = {list} ['2016-02-16 14:53:43.000000']
'emotion_measurements[1][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[1][measurement]' (4397285864) = {list} ['72']
'emotion_measurements[1][timestamp]' (4397285952) = {list} ['2016-02-16 14:54:43.000000']
'emotion_measurements[2][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[2][measurement]' (4397285864) = {list} ['73']
'emotion_measurements[2][timestamp]' (4397285952) = {list} ['2016-02-16 14:55:43.000000']
Has anyone encountered this issue before? Thanks!
UPDATE #2
I was able to resolve this (although I believe it is more of a workaround than a solution) by adding the following to my MeasurementSerializer:
def to_internal_value(self, data):
formatted_data = json.dumps(data)
formatted_data = formatted_data.replace("[", "").replace("]","")
formatted_data = json.loads(formatted_data)
return formatted_data
The Measurement data coming in was a QueryDict when I believe I needed a Dict. There were also some extra brackets around the key and values so I had to remove those as well.
Still seeking a better answer than this!

The problem here is on the front-end side. By default the server interprets the data as application/x-www-form-urlencoded and in order for it to understand that you are sending it a json, you need to specify the contentType in your $.ajax request:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {...},
contentType: 'application/json; charset=UTF-8', // add this line
type: 'POST'
})
.error(function(r){})
.success(function(data){});
Now your validated_data.pop('measurements') in create() method of your GameSerializer should yield three objects with your measurements (but don't forget to redo your workaround from Update#2).