I'm trying to create a writable nested serializer. My parent model is Game and the nested models are Measurements. I am trying to post this data to my DRF application using AJAX. However, when try to post the data, the nested Measurements are empty OrderedDict().
Here are my models:
class Game(models.Model):
start_timestamp = models.DateTimeField(auto_now_add=False)
end_timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
class Measurement(models.Model):
game = models.ForeignKey(Game, on_delete=models.PROTECT, related_name='measurements')
measurement_type = models.CharField(max_length=56)
measurement = models.CharField(max_length=56)
timestamp = models.DateTimeField(auto_now_add=False)
date_added = models.DateTimeField(auto_now_add=True)
Here are my serializers:
class MeasurementSerializer(serializers.ModelSerializer):
timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']), required=False)
class Meta:
model = Measurement
fields = ('measurement_type', 'measurement', 'timestamp')
class GameSerializer(serializers.ModelSerializer):
start_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
end_timestamp = serializers.DateTimeField(input_formats=(['%Y-%m-%d %H:%M:%S.%Z', 'iso-8601']))
measurements = MeasurementSerializer(many=True)
class Meta:
model = Game
fields = ('id', 'start_timestamp', 'end_timestamp', 'measurements')
def create(self, validated_data):
measurements = validated_data.pop('measurements')
game = Game.objects.create(**validated_data)
for measurement in measurements:
Measurement.objects.create(game=game, **measurement)
return game
My view for Game is the following:
class GameList(generics.ListCreateAPIView):
queryset = Game.objects.all()
serializer_class = GameSerializer
I followed this tutorial for the structure.
I am trying to post to this API via AJAX, the code below:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {
"start_timestamp": "2016-02-16 14:51:43.000000",
"end_timestamp": "2016-02-16 14:53:43.000000",
"measurements":[
{'measurement_type':'type1', 'measurement':'71', 'timestamp':'2016-02-16 14:53:43.000000'},
{'measurement_type':'type1', 'measurement':'72', 'timestamp':'2016-02-16 14:54:43.000000'},
{'measurement_type':'type1', 'measurement':'73', 'timestamp':'2016-02-16 14:55:43.000000'},
]
},
type: 'POST'
})
.error(function(r){})
.success(function(data){})
});
On posting this data, I find in the create method within the GameSerializer that the validate_data.pop('measurements') contains a list of 3 ordered dictionaries (OrderedDict()) that are empty.
UPDATE: I've found that that the initial_data coming in via request.data is structured like so:
'emotion_measurements[0][measurement_type]' (4397175560) = {list} ['type1']
'emotion_measurements[0][measurement]' (4397285512) = {list} ['71']
'emotion_measurements[0][timestamp]' (4397285600) = {list} ['2016-02-16 14:53:43.000000']
'emotion_measurements[1][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[1][measurement]' (4397285864) = {list} ['72']
'emotion_measurements[1][timestamp]' (4397285952) = {list} ['2016-02-16 14:54:43.000000']
'emotion_measurements[2][measurement_type]' (4397175040) = {list} ['type1']
'emotion_measurements[2][measurement]' (4397285864) = {list} ['73']
'emotion_measurements[2][timestamp]' (4397285952) = {list} ['2016-02-16 14:55:43.000000']
Has anyone encountered this issue before? Thanks!
UPDATE #2
I was able to resolve this (although I believe it is more of a workaround than a solution) by adding the following to my MeasurementSerializer:
def to_internal_value(self, data):
formatted_data = json.dumps(data)
formatted_data = formatted_data.replace("[", "").replace("]","")
formatted_data = json.loads(formatted_data)
return formatted_data
The Measurement data coming in was a QueryDict when I believe I needed a Dict. There were also some extra brackets around the key and values so I had to remove those as well.
Still seeking a better answer than this!
The problem here is on the front-end side. By default the server interprets the data as application/x-www-form-urlencoded and in order for it to understand that you are sending it a json, you need to specify the contentType in your $.ajax request:
$.ajax({
url: base_url + '/games/',
dataType: "json",
data: {...},
contentType: 'application/json; charset=UTF-8', // add this line
type: 'POST'
})
.error(function(r){})
.success(function(data){});
Now your validated_data.pop('measurements') in create() method of your GameSerializer should yield three objects with your measurements (but don't forget to redo your workaround from Update#2).
Related
While parsing my request data from front-end and converting into JSON format using a serializer. I am getting some unexpected errors.
while request parsing pattern using serializers given as mentioned below, it shows me the following error:(I found the below error using: contact_serializer.errors)
{'address': {u'non_field_errors': [u'Invalid data. Expected a dictionary, but got str.']}}
I do not think it will work like this. You have to remember here is that if you input the values like this, it will ultimately be stored in DB, and it is hard coded values. Even if you insist to do it like this, then use a list of dictionary like this:
request.data['phone_number'] = [{'number': '9999999999'}]
request.data['cont_email'] = [{'email':'tim#gmail.com'}]
And update the serializer like this:
class CrmContactSerializer(serializers.ModelSerializer):
phone_number = PhoneNumberSerializer(source = 'contact_number', many=True)
cont_email = ContactEmailSerializer(source = 'contact_email', many=True)
class Meta:
model = RestaurantContactAssociation
fields = ('id','phone_number','cont_email','contact')
def create(self, validated_data):
phone_number = validated_data.pop('contact_number')
cont_email = validated_data.pop('contact_email')
restaurant = super(CrmContactSerializer, self).create(validated_data)
phone_instance = PhoneNumber(**phone_number)
phone_instance.restaurant = restaurant
phone_instance.save()
email_instance = ContactEmail(**phone_number)
email_instance.restaurant = restaurant
email_instance.save()
return restaurant
Reason for many=True is that one restaurant can have multiple numbers or emails(as it has one to many relationship with respective models).
Now, if you think of proper way of implementing, you can make phone_number and cont_email read only fields, so that it will be used when only reading, not writing:
class CrmContactSerializer(serializers.ModelSerializer):
phone_number = PhoneNumberSerializer(source = 'contact_number', read_only=True)
cont_email = ContactEmailSerializer(source = 'contact_email', read_only=True)
class Meta:
model = RestaurantContactAssociation
fields = ('id','phone_number','cont_email','contact')
In that way, validation error can be handled for phone number and cont email.
I've used Django REST Framework to expose an API which is only used by another service to POST new data. It basically just takes json and inserts it in the DB. That's all.
It's quite a high volume data source (sometimes more than 100 records/second), so I need to tune it a bit.
So I was logging the (PostgreSQL) queries that are run, and I see that every POST gives 3 queries:
2019-10-01 11:09:03.320 CEST [23983] postgres#thedb LOG: statement: SET TIME ZONE 'UTC'
2019-10-01 11:09:03.322 CEST [23983] postgres#thedb LOG: statement: SELECT (1) AS "a" FROM "thetable" WHERE "thetable"."id" = 'a7f74e5c-7cad-4983-a909-49857481239b'::uuid LIMIT 1
2019-10-01 11:09:03.363 CEST [23983] postgres#thedb LOG: statement: INSERT INTO "thetable" ("id", "version", "timestamp", "sensor", [and 10 more fields...]) VALUES ('a7f74e5c-7cad-4983-a909-49857481239b'::uuid, '1', '2019-10-01T11:09:03.313690+02:00'::timestamptz, 'ABC123', [and 10 more fields...])
I tuned the DB for INSERTs to be fast, but SELECTs are slow. So I would like to remove the SELECT from the system. I added this line to the Serializer:
id = serializers.UUIDField(validators=[])
But it still does a SELECT. Does anybody know how I can prevent the SELECT from happening?
For complete info; the full Serializer now looks like this:
import logging
from rest_framework import serializers
from .models import TheData
log = logging.getLogger(__name__)
class TheDataSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = TheData
fields = [
'id',
'version',
'timestamp',
'sensor',
[and 10 more fields...]
]
class TheDataDetailSerializer(serializers.ModelSerializer):
id = serializers.UUIDField(validators=[])
class Meta:
model = TheData
fields = '__all__'
Edit
And as requested by frankie567, the ViewSet:
class TheDataViewSet(DetailSerializerMixin, viewsets.ModelViewSet):
serializer_class = serializers.TheDataSerializer
serializer_detail_class = serializers.TheDataDetailSerializer
queryset = TheData.objects.all().order_by('timestamp')
http_method_names = ['post', 'list', 'get']
filter_backends = [DjangoFilterBackend]
filter_class = TheDataFilter
pagination_class = TheDataPager
def get_serializer(self, *args, **kwargs):
""" The incoming data is in the `data` subfield. So I take it from there and put
those items in root to store it in the DB"""
request_body = kwargs.get("data")
if request_body:
new_request_body = request_body.get("data", {})
new_request_body["details"] = request_body.get("details", None)
request_body = new_request_body
kwargs["data"] = request_body
serializer_class = self.get_serializer_class()
kwargs['context'] = self.get_serializer_context()
return serializer_class(*args, **kwargs)
After some digging, I was able to see where this behaviour comes from. If you look at Django Rest Framework code:
if getattr(model_field, 'unique', False):
unique_error_message = model_field.error_messages.get('unique', None)
if unique_error_message:
unique_error_message = unique_error_message % {
'model_name': model_field.model._meta.verbose_name,
'field_label': model_field.verbose_name
}
validator = UniqueValidator(
queryset=model_field.model._default_manager,
message=unique_error_message)
validator_kwarg.append(validator)
We see that if unique is True (which is in your case, as I guess you defined your UUID field as primary key), DRF adds automatically a UniqueValidator. This validator performs a SELECT request to check if the value doesn't already exist.
It is appended to the ones you are defining in the validators parameter of the field, so that's why what you did has no effect.
So, how do we circumvent this?
First attempt
class TheDataDetailSerializer(serializers.ModelSerializer):
# ... your code
def get_fields(self):
fields = super().get_fields()
fields['id'].validators.pop()
return fields
Basically, we remove the validators of the id field after they have been generated. There are surely more clever ways to do this. It seems to me though that DRF may be too opinionated on this matter.
Second attempt
class TheDataDetailSerializer(serializers.ModelSerializer):
# ... your code
def build_standard_field(self, field_name, model_field):
field_class, field_kwargs = super().build_standard_field(field_name, model_field)
if field_name == 'id':
field_kwargs['validators'] = []
return field_class, field_kwargs
When generating the field arguments, set an empty validators list if we are generating the id field.
I have this basic model layout:
class Listing(models.Model):
name = models.TextField()
class ListingImage(models.Model):
listing = models.ForeignKey(Listing, related_name='images', on_delete=models.CASCADE)
image = models.ImageField(upload_to=listing_image_path)
Im trying to write a serializer which lets me add an rest api endpoint for creating Listings including images.
My idea would be this:
class ListingImageSerializer(serializers.ModelSerializer):
class Meta:
model = ListingImage
fields = ('image',)
class ListingSerializer(serializers.ModelSerializer):
images = ListingImageSerializer(many=True)
class Meta:
model = Listing
fields = ('name', 'images')
def create(self, validated_data):
images_data = validated_data.pop('images')
listing = Listing.objects.create(**validated_data)
for image_data in images_data:
ListingImage.objects.create(listing=listing, **image_data)
return listing
My Problems are:
I'm not sure how and if I can send a list of images in a nested dictionary using a multipart POST request.
If I just post an images list and try to convert it from a list to a list of dictionaries before calling the serializer, I get weird OS errors when parsing the actual image.
for key, item in request.data.items():
if key.startswith('images'):
# images.append({'image': item})
request.data[key] = {'image': item}
My request code looks like this:
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
api_token = 'xxxx'
images_data = MultipartEncoder(
fields={
'name': 'test',
'images[0]': (open('lilo.png', 'rb'), 'image/png'),
'images[1]': (open('panda.jpg', 'rb'), 'image/jpeg')
}
)
response = requests.post('http://127.0.0.1:8000/api/listings/', data=images_data,
headers={
'Content-Type': images_data.content_type,
'Authorization': 'Token' + ' ' + api_token
})
I did find a very hacky solution which I will post in the answers but its not really robust and there needs to be a better way to do this.
So my solution is based off of this post and works quite well but seems very unrobust and hacky.
I change the images field from a relation serializer requiring a dictonary to a ListField. Doing this i need to override the list field method to actually create a List out of the RelatedModelManager when calling "to_repesentation".
This baiscally behaves like a list on input, but like a modelfield on read.
class ModelListField(serializers.ListField):
def to_representation(self, data):
"""
List of object instances -> List of dicts of primitive datatypes.
"""
return [self.child.to_representation(item) if item is not None else None for item in data.all()]
class ListingSerializer(serializers.ModelSerializer):
images = ModelListField(child=serializers.FileField(max_length=100000, allow_empty_file=False, use_url=False))
class Meta:
model = Listing
fields = ('name', 'images')
def create(self, validated_data):
images_data = validated_data.pop('images')
listing = Listing.objects.create(**validated_data)
for image_data in images_data:
ListingImage.objects.create(listing=listing, image=image_data)
return listing
I've got an API endpoint called TrackMinResource, which returns the minimal data for a music track, including the track's main artist returned as an ArtistMinResource. Here are the definitions for both:
class TrackMinResource(ModelResource):
artist = fields.ForeignKey(ArtistMinResource, 'artist', full=True)
class Meta:
queryset = Track.objects.all()
resource_name = 'track-min'
fields = ['id', 'artist', 'track_name', 'label', 'release_year', 'release_name']
include_resource_uri = False
cache = SimpleCache(public=True)
def dehydrate(self, bundle):
bundle.data['full_artist_name'] = bundle.obj.full_artist_name()
if bundle.obj.image_url != settings.NO_TRACK_IMAGE:
bundle.data['image_url'] = bundle.obj.image_url
class ArtistMinResource(ModelResource):
class Meta:
queryset = Artist.objects.all()
resource_name = 'artist-min'
fields = ['id', 'artist_name']
cache = SimpleCache(public=True)
def get_resource_uri(self, bundle_or_obj):
return '/api/v1/artist/' + str(bundle_or_obj.obj.id) + '/'
The problem is, the artist field on Track (previously a ForeignKey) is now a model method called main_artist (I've changed the structure of the database somewhat, but I'd like the API to return the same data as it did before). Because of this, I get this error:
{"error": "The model '<Track: TrackName>' has an empty attribute 'artist' and doesn't allow a null value."}
If I take out full=True from the 'artist' field of TrackMinResource and add null=True instead, I get null values for the artist field in the returned data. If I then assign the artist in dehydrate like this:
bundle.data['artist'] = bundle.obj.main_artist()
...I just get the artist name in the returned JSON, rather than a dict representing an ArtistMinResource (along with the associated resource_uris, which I need).
Any idea how to get these ArtistMinResources into my TrackMinResource? I can access an ArtistMinResource that comes out fine using the URL endpoint and asking for it by ID. Is there a function for getting that result from within the dehydrate function for TrackMinResource?
You can use your ArtistMinResource in TrackMinResource's dehydrate like this (assuming that main_artist() returns the object that your ArtistMinResource represents):
artist_resource = ArtistMinResource()
artist_bundle = artist_resource.build_bundle(obj=bundle.obj.main_artist(), request=request)
artist_bundle = artist_resource.full_dehydrate(artist_bundle)
artist_json = artist_resource.serialize(request=request, data=artist_bundle, format='application/json')
artist_json should now contain your full artist representation. Also, I'm pretty sure you don't have to pass the format if you pass the request and it has a content-type header populated.
I've Patient, Doctor, Story Model. Each Story have a patient_id and a doctor_id. I want to retrieve a list of doctors the patient have visited ever.
class Patient(Person):
def visits(self):
doctor_visits = []
for v in self.stories.values('doctor').annotate(visits=Count('doctor')):
# replace the doctor id with doctor object
v['doctor'] = Doctor.objects.get(id=v['doctor'])
doctor_visits.append(v)
return doctor_visits
Here is my tastypie Resource
class PatientResource(ModelResource):
stories = fields.ToManyField('patients.api.StoryResource', 'stories', null=True)
visits = fields.ListField(attribute='visits', readonly=True)
class Meta:
queryset = Patient.objects.all()
excludes = ['id', 'login', 'password']
with the above tastypie results the following
{
address:"ADDRESS",
dob:"1985-12-04",
email:"EMAIL",
name:"Nogen",
resource_uri:"/patients/api/v1/patient/9/",
sex:"M",
stories:[
"/patients/api/v1/story/1/",
"/patients/api/v1/story/2/",
"/patients/api/v1/story/4/"
],
visits:[
{
doctor:"Dr. X",
visits:2
},
{
doctor:"Dr. Y",
visits:1
}
]
}
See Its caling the __unicode__ method of Doctor rather I expected this to be a link /patients/api/v1/doctor/<doctor_id>/ Do I need to construct the path manually or There is some other way around ?
I've tried using dehydrate possibly incorrectly
class PatientResource(ModelResource):
stories = fields.ToManyField('patients.api.StoryResource', 'stories', null=True)
visits = fields.ListField(attribute='visits', readonly=True)
class Meta:
queryset = Patient.objects.all()
excludes = ['id', 'login', 'password']
def dehydrate_visits(self, bundle):
for visit in bundle.data['visits']:
visit['doctor'] = DoctorResource(visit['doctor'])
return bundle
Which Results in maximum recursion depth exceeded while calling a Python object Exception
Not sure why you get maximum recursion depth but your method is wrong.
class PatientResource(ModelResource):
[...]
def dehydrate_visits(self, bundle):
# Make sure `bundle.data['visits'][0]['doctor'] isn't string.
# If it's already dehydrated string: try use `bundle.obj.visits` instead.
for visit in bundle.data['visits']:
visit['doctor'] = DoctorResource.get_resource_uri(visit['doctor'])
return bundle
I didn't test that. So fill free to comment if its incorrect.