I am trying to make a serializer with a nested "many to many" relationship. The goal is to get a serialized JSON object contain an array of serialized related objects. The models look like this (names changed, structure preserved)
from django.contrib.auth.models import User
PizzaTopping(models.Model):
name = models.CharField(max_length=255)
inventor = models.ForeignKey(User)
Pizza(models.Model):
name = models.CharField(max_length=255)
toppings = models.ManyToManyField(PizzaTopping)
The incoming JSON looks like this
{
"name": "My Pizza",
"toppings": [
{"name": "cheese", "inventor": "bob"},
{"name": "tomatoes", "inventor": "alice"}
]
}
My current serializer code looks like this
class ToppingRelatedField(RelatedField):
def get_queryset(self):
return Topping.objects.all()
def to_representation(self, instance):
return {'name': instance.name, 'inventor': instance.inventor.username}
def to_internal_value(self, data):
name = data.get('name', None)
inventor = data.get('inventor', None)
try:
user = User.objects.get(username=inventor)
except Setting.DoesNotExist:
raise serializers.ValidationError('bad inventor')
return Topping(name=name, inventor=user)
class PizzaSerializer(ModelSerializer):
toppings = ToppingRelatedField(many=True)
class Meta:
model = Pizza
fields = ('name', 'toppings')
It seems that since I defined the to_internal_value() for the custom field, it should create/update the many-to-many field automatically. But when I try to create pizzas, I get "Cannot add "": the value for field "pizzatopping" is None" ValueError. It looks like somewhere deep inside, Django decided that the many to many field should be called by the model name. How do I convince it otherwise?
Edit #1: It seems that this might be a genuine bug somewhere in Django or DRF. DRF seems to be doing the right thing, it detects that it is dealing with a ManyToMany field and tries to create toppings from the data using the custom field and add them to the pizza. Since it only has a pizza instance and a field name, it uses setattr(pizza, 'toppings', toppings) to do it. Django seems to be doing the right thing. The __set__ is defined and seems to figure out that it needs to use add() method in the manager. But somewhere along the way, the field name 'toppings' gets lost and replaced by the default. Which is "related model name in lower case".
Edit #2: I have found a solution. I will document it in an answer once I am allowed. It seems that the to_internal_value() method in the RelatedField subclass needs to return a saved instance of a Topping for the ManyToMany thing to work properly. The existing docs show the opposite, a this link (http://www.django-rest-framework.org/api-guide/fields/#custom-fields) the example clearly returns an unsaved instance.
Seems like there is an undocumented requirement. For write operations to work with a custom ManyToMany field, the custom field class to_internal_value() method needs to save the instance before returning it. The DRF docs omit this and the example of making a custom field (at http://www.django-rest-framework.org/api-guide/fields/#custom-fields) shows the method returning an unsaved instance. I am going to update the issue I opened with the DRF team.
I was also trying to return multiple fields as json but getting error unhashable type: 'dict. Finally, I found what's wrong with my approach here - https://github.com/encode/django-rest-framework/issues/5104
RelatedFields generally represent a related object as a single value
(eg, a slug, primary key, url, etc...). If you want to provide a
nested object representation, then you should use a nested serializer.
Related
I, as a newbie Django developer, am trying to build a RESTful API for a mobile app. I've took over an existing project and previous developers have used Django REST Framework. Super cool package, easy to work with so far. Except for one thing...
There is this problem when I want to create new resources, which happen to have nested serializers. I'm not great on explaining software issues with words, so here is the simplified version of my case:
class UserSerializer(serializers.ModelSerializer):
company = CompanySerializer()
# other props and functions are unrelated
class CompanySerializer(serializers.ModelSerializer):
# props and functions are unrelated
Now with this structure, GET /users and GET /users/{id} endpoints work great and I get the results I expect. But with POST /users and PATCH /users/{id} I get a response that says I need to provide an object for company prop and it should resemble a Company object with all the required props, so that it can create the company too. And I'm sure it tries to create a new company because I've tried sending { company: { id: 1 } } and it simply ignores the ID and requires a name to create a new one. This is obviously not what I want because I just want to create a user (who may or may not belong to a company), not both a user and a company.
I've tried switching that CompanySerializer to a serializers.PrimaryKeyRelatedField and it seems like it works on create endpoint but now I don't get the Company object on list and detail endpoints.
What am I missing here? I'm 99% sure that they did not intend this framework to work this way.
You need to override create() and update() methods on nested serializers to make them writable. Otherwise DRF is not sure what to do with nested objects. The simplest override would go something like this:
class UserSerializer(serializers.ModelSerializer):
company = CompanySerializer()
...
def create(self, validated_data):
return User.objects.create(**validated_data)
def update(self, instance, validated_data):
user = instance
user.__dict__.update(validated_data)
user.save()
return user
Note: haven't tested this variant of update() might need adjustments.
The trick is to use a different serializer class for retrieving vs updating - one with a PrimaryKeyRelatedField and one with a nested serializer. You can override get_serializer_class to do this. Assuming you are using a viewset:
class BaseUserSerializer(serializers.ModelSerializer):
class Meta:
model = User
fields = (...)
class WriteUserSerializer(BaseUserSerializer):
company = CompanySerializer()
class ReadUserSerializer(BaseUserSerializer):
company = PrimaryKeyRelatedField()
class UserViewSet(viewsets.ViewSet):
def get_serializer_class(self):
if self.action in ["create", "update", "partial_update", "destroy"]:
return WriteUserSerializer
else:
return ReadUserSerializer
I am using Django's dumpdata to save data and loaddata to reload it. I am also using natural keys. My model looks similar to this:
class LinkManager(models.Manager):
def get_by_natural_key(self, url):
return self.get(url=url)
class Link(models.Model):
objects = LinkManager()
title = models.CharField(max_length=200)
url = models.URLField()
def natural_key(self):
return (self.url, )
If I export and reimport the data, Django recognizes that the objects already exist and doesn't create duplicates. If I change the title, it correctly updates the objects. However, if I change the URL, it correctly treats it as a new object - although I forgot to mark url unique! How does it guess my intent?
How does django know that my url field is the natural key? There is no get_natural_fields function. Django could call natural_key on the class instead of an instance to get the fields, but that seems really brittle:
>>> [f.field_name for f in Link.natural_key(Link)]
['url']
The reason I want to know this is that I am writing my own special importer (to replace my use of loaddata), and I would like to take advantage of natural keys without hardcoding the natural key (or the "identifying" fields) for each model. Currently, I "identify" an object by it's unique fields - I do:
obj, created = Model.objects.update_or_create(**identifying, defaults=other)
but Django seems to be choosing it's "identifying" fields differently.
I think I've found it out. Django does not just call get_by_natural_key, it first calls natural_key. How does it do that, if it doesn't have an instance of the model?
It simply creates an instance, not backed by the database, from the constructor (d'oh!): Model(**data). See build_instance in django.core.serializers.base. Then it calls natural_key on the newly created object, and immediately get_by_natural_key to retrive the pk that belongs to the object, if present in the database. This way, Django does not need to know what fields the natural key depends on, it just needs to know how to get it from data. You can just call save() on the retrieved instance, if it is in the database it will have a pk and will update, if not it will create a new row.
Source of the build_instance function (Django 1.11.2):
def build_instance(Model, data, db):
"""
Build a model instance.
If the model instance doesn't have a primary key and the model supports
natural keys, try to retrieve it from the database.
"""
obj = Model(**data)
if (obj.pk is None and hasattr(Model, 'natural_key') and
hasattr(Model._default_manager, 'get_by_natural_key')):
natural_key = obj.natural_key()
try:
obj.pk = Model._default_manager.db_manager(db).get_by_natural_key(*natural_key).pk
except Model.DoesNotExist:
pass
return obj
So I have follwoing models:
class A(models.Model):
name = models.CharField()
age = models.SmallIntergerField()
class B(models.Model):
a = models.OneToOneField(A)
salary = model.IntergerField()
Now I want to create one rest end point for there two as they are one to one. So I want following as get
{
url: 'http://localhost/customs/1/',
name: 'abc',
age: 24,
salary: 10000
}
Similary, I want to create records and update as well. Please let me know how can I achieve this in django rest framework 3.
I just encountered the same problem, it would indeed be useful to make the response structure less tied to the underlying model structure. Here's my take :
Reading is easy
Serializer fields have a source parameter, which can take dotted names to traverse attributes.
class ABSerializer(serializers.ModelSerializer):
class Meta:
model = A
fields = ['name', 'age', 'salary']
salary = serializer.IntegerField(source='b.salary') # this is your related_name
Writing is ... not officially supported
Validated data will show a nested structure, and the standard create and update methods will choke trying to assign a data dict to a OneToOneField.
The good news is that you can work around it by overriding create and update methods. Here's an example with update :
class ABSerializer(serializers.ModelSerializer):
class Meta:
model = A
fields = ['name', 'age', 'salary']
related_fields = ['b']
salary = serializer.IntegerField(source='b.salary') # this is your related_name
def update(self, instance, validated_data):
# Handle related objects
for related_obj_name in self.Meta.related_fields:
# Validated data will show the nested structure
data = validated_data.pop(related_obj_name)
related_instance = getattr(instance, related_obj_name)
# Same as default update implementation
for attr_name, value in data.items():
setattr(related_instance, attr_name, value)
related_instance.save()
return super(ABSerializer,self).update(instance, validated_data)
Of course, this example is very simplistic, doesn't do any exception handling, and won't work with more deeply nested objects... but you get the idea.
Another option
You could also create a read-write flavor of SerializerMethodField, which would consider both a getter and a setter, however that would probably end up being far more verbose in the end.
Hope that helps !
I know this is an old post but I came across this and after some research and reading through the Django Rest Framework documentation
So a quick search I found that you could use the related_name parameter for reverse relationships as stated here:
reverse relationships are not automatically included by the
ModelSerializer and HyperlinkedModelSerializer classes. To include
a reverse relationship, you must explicitly add it to the fields list.
For example:
class AlbumSerializer(serializers.ModelSerializer):
class Meta:
fields = ['tracks', ...]
You'll normally want to ensure that you've set an appropriate
related_name argument on the relationship, that you can use as the
field name.
For example:
class Track(models.Model):
album = models.ForeignKey(Album, related_name='tracks',
on_delete=models.CASCADE)
...
If you have not set a related name for the reverse relationship,
you'll need to use the automatically generated related name in the
fields argument.
For example:
class AlbumSerializer(serializers.ModelSerializer):
class Meta:
fields = ['track_set', ...]
Also, see the Django documentation on reverse
relationships
for more details.
I have been working on developing some RESTful Services in Django to be used with both Flash and Android apps.
Developing the services interface has been quite simple, but I have been running into an issue with serializing objects that have foreign key and many to many relationships.
I have a model like this:
class Artifact( models.Model ):
name = models.CharField( max_length = 255 )
year_of_origin = models.IntegerField( max_length = 4, blank = True, null = True )
object_type = models.ForeignKey( ObjectType, blank = True, null = True )
individual = models.ForeignKey( Individual, blank = True, null = True )
notes = models.TextField( blank = True, null = True )
Then I would perform a query on this model like this, using select_related(), to be sure that foreign key relationships are followed:
artifact = Artifact.objects.select_related().get(pk=pk)
Once I have the object, I serialize it, and pass that back to my view:
serializers.serialize( "json", [ artifact ] )
This is what I get back, note that the foreign keys (object_type and individual) are just the id's to their related objects.
[
{
pk: 1
model: "artifacts.artifact"
fields: {
year_of_origin: 2010
name: "Dummy Title"
notes: ""
object_type: 1
individual: 1
}
}
]
This is great, but what I was hoping for when using select_related() was that it would automatically populate the foreign key fields with the related object, not just the object's id.
I am recent convert to Django, but put in a fair amount of time developing with CakePHP.
What I really like about the Cake ORM was that it would follow the relationships and create nested objects by default, with the ability to unbind the relationships when you were calling your query.
This made it very easy to abstract the services in a way that did not require any intervention on a case by case basis.
I see that Django does not do this by default, but is there a way to automatically serialize an object and all of it's related objects? Any tips or reading would be much appreciated.
I had a similar requirement although not for RESTful purposes. I was able to achieve what I needed by using a "full" serializing module, in my case Django Full Serializers. This is part of wadofstuff and is distributed under the new BSD license.
Wadofstuff makes this quite easy. For e.g. in your case you'd need to do the following:
First, install wadofstuff.
Second, add the following setting to your settings.py file:
SERIALIZATION_MODULES = {
'json': 'wadofstuff.django.serializers.json'
}
Third, make a slight change to the code used for serialization:
artifact = Artifact.objects.select_related().get(pk=pk)
serializers.serialize( "json", [ artifact ], indent = 4,
relations = ('object_type', 'individual',))
The key change is the relations keyword parameter. The only (minor) gotcha is to use the name of the fields forming the relation not the names of the related models.
Caveat
From the documentation:
The Wad of Stuff serializers are 100% compatible with the Django serializers when serializing a model. When deserializing a data stream the the Deserializer class currently only works with serialized data returned by the standard Django serializers.
(Emphasis added)
Hope this helps.
UPDATE:
Actually Manoj's solution is a bit outdated, Wad of Stuff's serializer has been left un-updated for some time and when I tried that, it seems that it does not support Django 1.6 anymore.
However, take a look at Django's official doc here. It does provide some way around using the built-in natural key. It seems that django's built-in serializer has a a little problem supporting using ImageField as part of the natural key. But that can be easily fixed by your self.
I'm aware this topic is years old, however, I'm sharing my solution for the people still searching for an answer (during my search, I ended up here).
Please note, I was looking for a simple function which would give me nested (foreign key) objects/dictionaries (which could contain nested (foreign key) objects/dictionaries as well) within my model/queryset which I could then convert to JSON.
In my models.py, I have a custom function (not within a model class):
Models.py
def django_sub_dict(obj):
allowed_fields = obj.allowed_fields() # pick the list containing the requested fields
sub_dict = {}
for field in obj._meta.fields: # go through all the fields of the model (obj)
if field.name in allowed_fields: # be sure to only pick fields requested
if field.is_relation: # will result in true if it's a foreign key
sub_dict[field.name] = django_sub_dict(
getattr(obj, field.name)) # call this function, with a new object, the model which is being referred to by the foreign key.
else: # not a foreign key? Just include the value (e.g., float, integer, string)
sub_dict[field.name] = getattr(obj, field.name)
return sub_dict # returns the dict generated
This function loops through all the fields in a models.Model object, if the models.Model is provided. I call the function within a model as follows (for completeness sake, including one entire model):
the same Models.py
class sheet_categories(models.Model):
id = models.AutoField(primary_key=True, unique=True)
create_date = models.DateField(auto_now_add=True)
last_change = models.DateField(auto_now=True)
name = models.CharField(max_length=128)
sheet_type = models.ForeignKey(
sheet_types, models.SET_NULL, blank=False, null=True)
balance_sheet_sort = models.IntegerField(unique=True)
def allowed_fields(self):
return [
'name',
'sheet_type',
'balance_sheet_sort',
]
def natural_key(self):
return django_sub_dict(self) # call the custom function (which is included in this models.py)
Note:
The nested JSON objects will only contain fields which are included in the allowed_fields of a model. Thus not including sensitive information.
To ultimately generate a JSON, I have the following view in my views.py.
views.py
class BalanceSheetData(ListView): # I believe this doesn't have to **be** a ListView.
model = models.sheet_categories
def get_queryset(self):
return super().get_queryset().filter() # the filter is for future purposes. For now, not relevant
def get(self, request, *args, **kwargs):
context = {
'queryset': serializers.serialize("json",
self.get_queryset(),
use_natural_foreign_keys=True, # this or the one below makes django include the natural_key() within a model. Not sure.
use_natural_primary_keys=True, # this or the one above makes django include the natural_key() within a model. Not sure.
),
}
return JsonResponse(context)
This ultimately provided me with all the nested details I required in a JSON response. Although I do not share the JSON response, as this one is barely readable.
Feel free to comment.
You can find more information on this ticket:
Allow In-depth serialization by specifying depth to follow relationship
https://code.djangoproject.com/ticket/4656
Adding a newer answer to this older question: I created and recently published django-serializable-model as an easily extensible way to serialize models, managers, and querysets. When your models extend SerializableModel, they receive an overridable .serialize method that has built-in support for all relations.
Using your example, once all of the involved models extend SerializableModel:
joins = ['object_type', 'individual']
artifact = Artifact.objects.select_related(*joins).get(pk=pk)
artifact.serialize(*joins)
Calling .serialize with the relations as arguments will have the library recurse over the related objects, calling .serialize on them as well. This returns a dictionary that looks like:
{
'id': 1,
'year_of_origin': 2010,
'name': 'Dummy Title',
'notes': '',
'object_type_id': 1,
'individual_id': 1,
'object_type': { ... nested object here ... },
'individual': { ... nested object here ... }
}
You can then call json.dumps on this dictionary to transform it to JSON.
By default, extending SerializableModel will also set the model's manager to SerializableManager (you can extend it yourself if you're using a custom manager) which uses SerializableQuerySet. This means you can call .serialize on a manager or queryset as well:
artifacts = Artifact.objects.select_related(*joins).all()
artifacts.serialize(*joins)
This simply calls .serialize on each model object in the queryset, returning a list of dictionaries in the same format as above.
django-serializable-model also allows you to easily override the default behavior on a per model basis, giving you the ability to do things like: add allowlists or denylists applied to each model's .serialize, always serialize certain joins (so you don't have to add them as arguments all the time), and more!
I've been working through an ordered ManyToManyField widget, and have the front-end aspect of it working nicely:
Unfortunately, I'm having a great deal of trouble getting the backend working. The obvious way to hook up the backend is to use a through table keyed off a model with ForeignKeys to both sides of the relationship and overwrite the save method. This would work great, except that due to idiosyncrasies of the content, it is an absolute requirement that this widget be placed in a fieldset (using the ModelAdmin fieldsets property), which is apparently not possible.
I'm out of ideas. Any suggestions?
Thanks!
In regard to how to set up the models, you're right in that a through table with an "order" column is the ideal way to represent it. You're also right in that Django will not let you refer to that relationship in a fieldset. The trick to cracking this problem is to remember that the field names you specify in the "fieldsets" or "fields" of a ModelAdmin do not actually refer to the fields of the Model, but to the fields of the ModelForm, which we are free to override to our heart's delight. With many2many fields, this gets tricky, but bear with me:
Let's say you're trying to represent contests and competitors that compete in them, with an ordered many2many between contests and competitors where the order represents the competitors' ranking in that contest. Your models.py would then look like this:
from django.db import models
class Contest(models.Model):
name = models.CharField(max_length=50)
# More fields here, if you like.
contestants = models.ManyToManyField('Contestant', through='ContestResults')
class Contestant(models.Model):
name = models.CharField(max_length=50)
class ContestResults(models.Model):
contest = models.ForeignKey(Contest)
contestant = models.ForeignKey(Contestant)
rank = models.IntegerField()
Hopefully, this is similar to what you're dealing with. Now, for the admin. I've written an example admin.py with plenty of comments to explain what's happening, but here's a summary to help you along:
Since I don't have the code to the ordered m2m widget you've written, I've used a placeholder dummy widget that simply inherits from TextInput. The input holds a comma-separated list (without spaces) of contestant IDs, and the order of their appearance in the string determines the value of their "rank" column in the ContestResults model.
What happens is that we override the default ModelForm for Contest with our own, and then define a "results" field inside it (we can't call the field "contestants", since there would be a name conflict with the m2m field in the model). We then override __init__(), which is called when the form is displayed in the admin, so we can fetch any ContestResults that may have already been defined for the Contest, and use them to populate the widget. We also override save(), so that we can in turn get the data from the widget and create the needed ContestResults.
Note that for the sake of simplicity this example omits things like validation of the data from the widget, so things will break if you try to type in anything unexpected in the text input. Also, the code for creating the ContestResults is quite simplistic, and could be greatly improved upon.
I should also add that I've actually ran this code and verified that it works.
from django import forms
from django.contrib import admin
from models import Contest, Contestant, ContestResults
# Generates a function that sequentially calls the two functions that were
# passed to it
def func_concat(old_func, new_func):
def function():
old_func()
new_func()
return function
# A dummy widget to be replaced with your own.
class OrderedManyToManyWidget(forms.widgets.TextInput):
pass
# A simple CharField that shows a comma-separated list of contestant IDs.
class ResultsField(forms.CharField):
widget = OrderedManyToManyWidget()
class ContestAdminForm(forms.models.ModelForm):
# Any fields declared here can be referred to in the "fieldsets" or
# "fields" of the ModelAdmin. It is crucial that our custom field does not
# use the same name as the m2m field field in the model ("contestants" in
# our example).
results = ResultsField()
# Be sure to specify your model here.
class Meta:
model = Contest
# Override init so we can populate the form field with the existing data.
def __init__(self, *args, **kwargs):
instance = kwargs.get('instance', None)
# See if we are editing an existing Contest. If not, there is nothing
# to be done.
if instance and instance.pk:
# Get a list of all the IDs of the contestants already specified
# for this contest.
contestants = ContestResults.objects.filter(contest=instance).order_by('rank').values_list('contestant_id', flat=True)
# Make them into a comma-separated string, and put them in our
# custom field.
self.base_fields['results'].initial = ','.join(map(str, contestants))
# Depending on how you've written your widget, you can pass things
# like a list of available contestants to it here, if necessary.
super(ContestAdminForm, self).__init__(*args, **kwargs)
def save(self, *args, **kwargs):
# This "commit" business complicates things somewhat. When true, it
# means that the model instance will actually be saved and all is
# good. When false, save() returns an unsaved instance of the model.
# When save() calls are made by the Django admin, commit is pretty
# much invariably false, though I'm not sure why. This is a problem
# because when creating a new Contest instance, it needs to have been
# saved in the DB and have a PK, before we can create ContestResults.
# Fortunately, all models have a built-in method called save_m2m()
# which will always be executed after save(), and we can append our
# ContestResults-creating code to the existing same_m2m() method.
commit = kwargs.get('commit', True)
# Save the Contest and get an instance of the saved model
instance = super(ContestAdminForm, self).save(*args, **kwargs)
# This is known as a lexical closure, which means that if we store
# this function and execute it later on, it will execute in the same
# context (i.e. it will have access to the current instance and self).
def save_m2m():
# This is really naive code and should be improved upon,
# especially in terms of validation, but the basic gist is to make
# the needed ContestResults. For now, we'll just delete any
# existing ContestResults for this Contest and create them anew.
ContestResults.objects.filter(contest=instance).delete()
# Make a list of (rank, contestant ID) tuples from the comma-
# -separated list of contestant IDs we get from the results field.
formdata = enumerate(map(int, self.cleaned_data['results'].split(',')), 1)
for rank, contestant in formdata:
ContestResults.objects.create(contest=instance, contestant_id=contestant, rank=rank)
if commit:
# If we're committing (fat chance), simply run the closure.
save_m2m()
else:
# Using a function concatenator, ensure our save_m2m closure is
# called after the existing save_m2m function (which will be
# called later on if commit is False).
self.save_m2m = func_concat(self.save_m2m, save_m2m)
# Return the instance like a good save() method.
return instance
class ContestAdmin(admin.ModelAdmin):
# The precious fieldsets.
fieldsets = (
('Basic Info', {
'fields': ('name', 'results',)
}),)
# Here's where we override our form
form = ContestAdminForm
admin.site.register(Contest, ContestAdmin)
In case you're wondering, I had ran into this problem myself on a project I've been working on, so most of this code comes from that project. I hope you find it useful.