Develop Reference

Proper custom writable (and readable) ModelSerializer field

I'm working on a project that manages school staff. The relationship between the two is poorly implemented in my opinion so I am trying to refactor it. However, to minimize impact, since this is not a priority I'm trying to do this without breaking the front-end, which should be refactored at a later date. The current Staff model looks like this:
class Staff(models.Model):
principal_at = models.ForeignKey(School, null=True)
teacher_at = models.ForeignKey(School, null=True)
# I'm only showing these two roles for the sake of simplicity but there are more.
I'm refactoring it to this:
class Staff(models.Model):
ROLE_CHOICES = (
"principal",
"teacher",
# ...
)
school = models.ForeignKey(School)
role = models.CharField(choices=ROLE_CHOICES)
# ...
#property
def principal_at(self):
return self.school if self.role == "principal" else None
# ...
My StaffSerializer looks like this:
class StaffSerializer(serializers.ModelSerializer):
class Meta:
model = Staff
fields = (
"principal_at",
"teacher_at",
# ...
)
# There is also some custom validation to ensure the Staff can only have
# one role.
If I just keep my it unchanged, the principal_at field becomes a ReadOnlyField and it doesn't get saved, which makes sense since my model attribute is now a property and not an actual writeable field. I changed, therefore, my serializer to this:
class StaffSerializer(serializers.ModelSerializer):
kwargs = {"required": False, "queryset": School.objects.all()}
principal_at = serializers.PrimaryKeyRelatedField(**kwargs)
teacher_at = serializers.PrimaryKeyRelatedField(**kwargs)
class Meta:
model = Staff
fields = (
"principal_at",
"teacher_at",
# ...
)
def validate(self, data):
principal_at = data.pop("principal_at", None)
teacher_at = data.pop("teacher_at", None)
# ...
if principal_at:
data.update(school_id=principal_at, role="principal")
elif teacher_at:
data.update(school_id=teacher_at, role="teacher")
# ...
return data
Now the model is saved correctly (I checked with ipdb) but in the process of building the response I get this:
TypeError: Object of type School is not JSON serializable
After this point I honestly don't remember what I've tried, but I've tried many things and nothing works. One thing I do remember trying and that blew my mind was creating a custom field and overriding to_representation:
class RelatedSchoolSerializer(serializers.PrimaryKeyRelatedField):
def to_representation(self, value):
import ipdb; ipdb.set_trace()
return value.pk
class StaffSerializer(serializers.ModelSerializer):
kwargs = {"required": False, "queryset": School.objects.all()}
principal_at = serializers.PrimaryKeyRelatedField(**kwargs)
teacher_at = serializers.PrimaryKeyRelatedField(**kwargs)
# ...
This kept failing with the same error, however, prompting me to add a breakpoint and check what the f**k is going on. Inside the debugger, to my absolute dismay, I got the following:
ipdb> value
<rest_framework.relations.PKOnlyObject object at 0x7f9563c87f90>
ipdb> value.pk
<School: School 0>
ipdb> value.pk.pk
1
ipdb>
If I return value.pk.pk instead of value.pk, everything works. But even though my test suite is passing, I'm afraid that such a weird hack may have mysterious unintended consequences down the road. What on earth is going on? What is the proper way of doing this?

Django Tastypie : How to initiate a variable every time an api call is made?

I have two models :
class Category(models.Model):
name = models.CharField(max_length=50)
and
class SubCategory(models.Model):
sex = models.CharField(choices=SEX, blank=True, max_length=5)
name = models.TextField(blank=True)
category = models.ForeignKey(Category)
def __unicode__(self):
return u'%s' % (self.name)
I'm creating an api using tastypie that returns me "SubCategory" objects in a JSON. I want to add a custom field "start_counter_of_category" in every result set which contains the counter of the subcategory where the category id changed (when ordered on "category_id" field)
The algorithm is fairly straightforward, something like this in the "dehydrate" function:
API_SKU_VARS = {
'COUNTER' : 1,
'FIRST_ELEMENT' : 1,
'PREV_CATEGORY' : 1
}
class SubCategoryResource(ModelResource):
start_counter_of_category = fields.IntegerField(readonly=True)
category = fields.ForeignKey(CategoryResource,'category')
class Meta:
queryset = SubCategory.objects.all()
resource_name = 'subcategory'
filtering = {
'id' : ALL,
'name' : ALL,
'category': ALL_WITH_RELATIONS,
}
ordering = ['id','name','category']
serializer = Serializer(formats=['json'])
def dehydrate(self, bundle):
if API_SKU_VARS['PREV_CATEGORY'] != bundle.data['category']: #if the category of the current bundle is not equal to the category of the previous bundle, we update the ['PREV_CATEGORY']
API_SKU_VARS['FIRST_ELEMENT']=API_SKU_VARS['COUNTER'] #update the ['FIRST_ELEMENT'] with the counter of the current bundle
API_SKU_VARS['PREV_CATEGORY'] = bundle.data['category']
API_SKU_VARS['COUNTER'] = API_SKU_VARS['COUNTER']+1 #for every bundle passed, we update the counter
bundle.data['start_counter_of_category']=API_SKU_VARS['FIRST_ELEMENT']
return bundle.data
serializer = Serializer(formats=['json'])
It works perfectly for the first run after I start the server. Predictably the issue of course is that the second time I make the api call, the variables retain the values they had in the previous run.
Any ideas how to re-initiate the variables every time the api call is made?
SOLUTION:
re-initialte the variables in
build_filters if the api called is a filtering API
get_detail if the api called is a detail API
example (in my case):
def build_filters(self, filters=None):
if filters is None:
filters = {}
orm_filters = super(SubCategoryResource, self).build_filters(filters) #get the required response using the function's behavior from the super class
self.API_SKU_VARS = {
'PREV_CATEGORY':1,
'COUNTER':1,
'FIRST_ELEMENT':1,
}
return orm_filters
(These functions are over-ridden if you want to apply any custom logic into the API response)
BETTER AND MOST OBVIOUS SOLUTION
re-instantiate the variables in the init function, something like this:
def __init__(self,api_name=None):
self.API_SKU_VARS = {.....}
super(SKUResource,self).__init__(api_name)

Yes, you can reinitialize (not "reinitiate") the variables just by running this code at the start of each call:
API_SKU_VARS['COUNTER'] = 1
API_SKU_VARS['PREV_CATEGORY'] = 1
API_SKU_VARS['FIRST_ELEMENT'] = 1
But this is a bad idea. Why is this variable global in the first place? The whole point of a global variable is that it's shared by all objects in the module, and lives for the lifetime of the module. If you want something that's local to a single API call and lives for the lifetime of that call, make it a member of something that has those characteristics. Initialize the member as part of initializing the appropriate object. Then there's no need to reinitialize it.
To see why this is a bad idea: What happens if two clients connect up concurrently and both make the same API call?

How does django one-to-one relationships map the name to the child object?

Apart from one example in the docs, I can't find any documentation on how exactly django chooses the name with which one can access the child object from the parent object. In their example, they do the following:
class Place(models.Model):
name = models.CharField(max_length=50)
address = models.CharField(max_length=80)
def __unicode__(self):
return u"%s the place" % self.name
class Restaurant(models.Model):
place = models.OneToOneField(Place, primary_key=True)
serves_hot_dogs = models.BooleanField()
serves_pizza = models.BooleanField()
def __unicode__(self):
return u"%s the restaurant" % self.place.name
# Create a couple of Places.
>>> p1 = Place(name='Demon Dogs', address='944 W. Fullerton')
>>> p1.save()
>>> p2 = Place(name='Ace Hardware', address='1013 N. Ashland')
>>> p2.save()
# Create a Restaurant. Pass the ID of the "parent" object as this object's ID.
>>> r = Restaurant(place=p1, serves_hot_dogs=True, serves_pizza=False)
>>> r.save()
# A Restaurant can access its place.
>>> r.place
<Place: Demon Dogs the place>
# A Place can access its restaurant, if available.
>>> p1.restaurant
So in their example, they simply call p1.restaurant without explicitly defining that name. Django assumes the name starts with lowercase. What happens if the object name has more than one word, like FancyRestaurant?
Side note: I'm trying to extend the User object in this way. Might that be the problem?

If you define a custom related_name then it will use that, otherwise it will lowercase the entire model name (in your example .fancyrestaurant). See the else block in django.db.models.related code:
def get_accessor_name(self):
# This method encapsulates the logic that decides what name to give an
# accessor descriptor that retrieves related many-to-one or
# many-to-many objects. It uses the lower-cased object_name + "_set",
# but this can be overridden with the "related_name" option.
if self.field.rel.multiple:
# If this is a symmetrical m2m relation on self, there is no reverse accessor.
if getattr(self.field.rel, 'symmetrical', False) and self.model == self.parent_model:
return None
return self.field.rel.related_name or (self.opts.object_name.lower() + '_set')
else:
return self.field.rel.related_name or (self.opts.object_name.lower())
And here's how the OneToOneField calls it:
class OneToOneField(ForeignKey):
... snip ...
def contribute_to_related_class(self, cls, related):
setattr(cls, related.get_accessor_name(),
SingleRelatedObjectDescriptor(related))
The opts.object_name (referenced in the django.db.models.related.get_accessor_name) defaults to cls.__name__.
As for
Side note: I'm trying to extend the
User object in this way. Might that be
the problem?
No it won't, the User model is just a regular django model. Just watch out for related_name collisions.

How would you inherit from and override the django model classes to create a listOfStringsField?

I want to create a new type of field for django models that is basically a ListOfStrings. So in your model code you would have the following:
models.py:
from django.db import models
class ListOfStringsField(???):
???
class myDjangoModelClass():
myName = models.CharField(max_length=64)
myFriends = ListOfStringsField() #
other.py:
myclass = myDjangoModelClass()
myclass.myName = "bob"
myclass.myFriends = ["me", "myself", "and I"]
myclass.save()
id = myclass.id
loadedmyclass = myDjangoModelClass.objects.filter(id__exact=id)
myFriendsList = loadedclass.myFriends
# myFriendsList is a list and should equal ["me", "myself", "and I"]
How would you go about writing this field type, with the following stipulations?
We don't want to do create a field which just crams all the strings together and separates them with a token in one field like this. It is a good solution in some cases, but we want to keep the string data normalized so tools other than django can query the data.
The field should automatically create any secondary tables needed to store the string data.
The secondary table should ideally have only one copy of each unique string. This is optional, but would be nice to have.
Looking in the Django code it looks like I would want to do something similar to what ForeignKey is doing, but the documentation is sparse.
This leads to the following questions:
Can this be done?
Has it been done (and if so where)?
Is there any documentation on Django about how to extend and override their model classes, specifically their relationship classes? I have not seen a lot of documentation on that aspect of their code, but there is this.
This is comes from this question.

There's some very good documentation on creating custom fields here.
However, I think you're overthinking this. It sounds like you actually just want a standard foreign key, but with the additional ability to retrieve all the elements as a single list. So the easiest thing would be to just use a ForeignKey, and define a get_myfield_as_list method on the model:
class Friends(model.Model):
name = models.CharField(max_length=100)
my_items = models.ForeignKey(MyModel)
class MyModel(models.Model):
...
def get_my_friends_as_list(self):
return ', '.join(self.friends_set.values_list('name', flat=True))
Now calling get_my_friends_as_list() on an instance of MyModel will return you a list of strings, as required.

What you have described sounds to me really similar to the tags.
So, why not using django tagging?
It works like a charm, you can install it independently from your application and its API is quite easy to use.

I also think you're going about this the wrong way. Trying to make a Django field create an ancillary database table is almost certainly the wrong approach. It would be very difficult to do, and would likely confuse third party developers if you are trying to make your solution generally useful.
If you're trying to store a denormalized blob of data in a single column, I'd take an approach similar to the one you linked to, serializing the Python data structure and storing it in a TextField. If you want tools other than Django to be able to operate on the data then you can serialize to JSON (or some other format that has wide language support):
from django.db import models
from django.utils import simplejson
class JSONDataField(models.TextField):
__metaclass__ = models.SubfieldBase
def to_python(self, value):
if value is None:
return None
if not isinstance(value, basestring):
return value
return simplejson.loads(value)
def get_db_prep_save(self, value):
if value is None:
return None
return simplejson.dumps(value)
If you just want a django Manager-like descriptor that lets you operate on a list of strings associated with a model then you can manually create a join table and use a descriptor to manage the relationship. It's not exactly what you need, but this code should get you started.

Thanks for all those that answered. Even if I didn't use your answer directly the examples and links got me going in the right direction.
I am not sure if this is production ready, but it appears to be working in all my tests so far.
class ListValueDescriptor(object):
def __init__(self, lvd_parent, lvd_model_name, lvd_value_type, lvd_unique, **kwargs):
"""
This descriptor object acts like a django field, but it will accept
a list of values, instead a single value.
For example:
# define our model
class Person(models.Model):
name = models.CharField(max_length=120)
friends = ListValueDescriptor("Person", "Friend", "CharField", True, max_length=120)
# Later in the code we can do this
p = Person("John")
p.save() # we have to have an id
p.friends = ["Jerry", "Jimmy", "Jamail"]
...
p = Person.objects.get(name="John")
friends = p.friends
# and now friends is a list.
lvd_parent - The name of our parent class
lvd_model_name - The name of our new model
lvd_value_type - The value type of the value in our new model
This has to be the name of one of the valid django
model field types such as 'CharField', 'FloatField',
or a valid custom field name.
lvd_unique - Set this to true if you want the values in the list to
be unique in the table they are stored in. For
example if you are storing a list of strings and
the strings are always "foo", "bar", and "baz", your
data table would only have those three strings listed in
it in the database.
kwargs - These are passed to the value field.
"""
self.related_set_name = lvd_model_name.lower() + "_set"
self.model_name = lvd_model_name
self.parent = lvd_parent
self.unique = lvd_unique
# only set this to true if they have not already set it.
# this helps speed up the searchs when unique is true.
kwargs['db_index'] = kwargs.get('db_index', True)
filter = ["lvd_parent", "lvd_model_name", "lvd_value_type", "lvd_unique"]
evalStr = """class %s (models.Model):\n""" % (self.model_name)
evalStr += """ value = models.%s(""" % (lvd_value_type)
evalStr += self._params_from_kwargs(filter, **kwargs)
evalStr += ")\n"
if self.unique:
evalStr += """ parent = models.ManyToManyField('%s')\n""" % (self.parent)
else:
evalStr += """ parent = models.ForeignKey('%s')\n""" % (self.parent)
evalStr += "\n"
evalStr += """self.innerClass = %s\n""" % (self.model_name)
print evalStr
exec (evalStr) # build the inner class
def __get__(self, instance, owner):
value_set = instance.__getattribute__(self.related_set_name)
l = []
for x in value_set.all():
l.append(x.value)
return l
def __set__(self, instance, values):
value_set = instance.__getattribute__(self.related_set_name)
for x in values:
value_set.add(self._get_or_create_value(x))
def __delete__(self, instance):
pass # I should probably try and do something here.
def _get_or_create_value(self, x):
if self.unique:
# Try and find an existing value
try:
return self.innerClass.objects.get(value=x)
except django.core.exceptions.ObjectDoesNotExist:
pass
v = self.innerClass(value=x)
v.save() # we have to save to create the id.
return v
def _params_from_kwargs(self, filter, **kwargs):
"""Given a dictionary of arguments, build a string which
represents it as a parameter list, and filter out any
keywords in filter."""
params = ""
for key in kwargs:
if key not in filter:
value = kwargs[key]
params += "%s=%s, " % (key, value.__repr__())
return params[:-2] # chop off the last ', '
class Person(models.Model):
name = models.CharField(max_length=120)
friends = ListValueDescriptor("Person", "Friend", "CharField", True, max_length=120)
Ultimately I think this would still be better if it were pushed deeper into the django code and worked more like the ManyToManyField or the ForeignKey.

I think what you want is a custom model field.

Django serialization of inherited model

I have a problem with serialization of Django inherited models. For example
class Animal(models.Model):
color = models.CharField(max_length=50)
class Dog(Animal):
name = models.CharField(max_length=50)
...
# now I want to serialize Dog model with Animal inherited fields obviously included
print serializers.serialize('xml', Dog.objects.all())
and only Dog model has been serialized.
I can do smth like
all_objects = list(Animal.objects.all()) + list(Dog.objects.all())
print serializers.serialize('xml', all_objects)
But it looks ugly and because my models are very big so I have to use SAX parser and with such output it's difficult to parse.
Any idea how to serialize django models with parent class?
**EDIT: ** It use to work ok before this patch has been applied. And the explanation why the patch exist "Model saving was too aggressive about creating new parent class instances during deserialization. Raw save on a model now skips saving of the parent class. " I think there should be an option to be able to serialize "local fields only" by default and second option - "all" - to serialize all inherited fields.

You found your answer in the documentation of the patch.
all_objects = list(Animal.objects.all()) + list(Dog.objects.all())
print serializers.serialize('xml', all_objects)
However, if you change Animal to be an abstract base class it will work:
class Animal(models.Model):
color = models.CharField(max_length=50)
class Meta:
abstract = True
class Dog(Animal):
name = models.CharField(max_length=50)
This works as of Django 1.0. See http://docs.djangoproject.com/en/dev/topics/db/models/.

You'll need a custom serializer to support inherited fields, as Django's serializer will only serialize local fields.
I ended up writing my own when dealing with this issue, feel free to copy it: https://github.com/zmathew/django-backbone/blob/master/backbone/serializers.py
In order to use it on its own, you need to do:
serializer = AllFieldsSerializer()
serializer.serialize(queryset, fields=fields)
print serializer.getvalue()

I had the same problem, and i wrote a 'small' queryset serializer which navigates up the inheritance tree and returns all the fields serialized.
It's far from perfect... but works for me :)
a = QuerySetSerializer(MyModel, myqueryset)
a.serialize()
And the snippet:
from __future__ import unicode_literals
import json
import inspect
from django.core import serializers
from django.db.models.base import Model as DjangoBaseModel
class QuerySetSerializer(object):
def __init__(self, model, initial_queryset):
"""
#param model: The model of your queryset
#param initial_queryset: The queryset to serialize
"""
self.model = model
self.initial_queryset = initial_queryset
self.inheritance_tree = self._discover_inheritance_tree()
def serialize(self):
list_of_querysets = self._join_inheritance_tree_objects()
merged_querysets = self._zip_queryset_list(list_of_querysets)
result = []
for related_objects in merged_querysets:
result.append(self._serialize_related_objects(related_objects))
return json.dumps(result)
def _serialize_related_objects(self, related_objects):
"""
In this method, we serialize each instance using the django's serializer function as shown in :
See https://docs.djangoproject.com/en/1.10/topics/serialization/#inherited-models
However, it returns a list with mixed objects... Here we join those related objects into one single dict
"""
serialized_objects = []
for related_object in related_objects:
serialized_object = self._serialize_object(related_object)
fields = serialized_object['fields']
fields['pk'] = serialized_object['pk']
serialized_objects.append(fields)
merged_related_objects = {k: v for d in serialized_objects for k, v in d.items()}
return merged_related_objects
def _serialize_object(self, obj):
data = serializers.serialize('json', [obj, ])
struct = json.loads(data)
return struct[0]
def _discover_inheritance_tree(self):
# We need to find the inheritance tree which excludes abstract classes,
# so we can then join them when serializing the instance
return [x for x in inspect.getmro(self.model) if x is not object and x is not DjangoBaseModel and not x._meta.abstract]
def _join_inheritance_tree_objects(self):
"""
Here we join the required querysets from the non abstract inherited models, which we need so we are able to
serialize them.
Lets say that MyUser inherits from Customer and customer inherits from django's User model
This will return [list(MyUser.objects.filter(...), list(Customer.objects.filter(...), list(User.objects.filter(...)
"""
initial_ids = self._get_initial_ids()
inheritance__querysets = [list(x.objects.filter(id__in=initial_ids).order_by("id")) for x in self.inheritance_tree]
return inheritance__querysets
def _zip_queryset_list(self, list_of_querysets):
"""
At this stage, we have something like:
(
[MyUser1, MyUser2, MyUser3],
[Customer1, Customer2, Customer3],
[User1, User2, User3]
)
And to make it easier to work with, we 'zip' the list of lists so it looks like:
(
[MyUser1, Customer1, User1],
[MyUser2, Customer2, User2],
[MyUser3, Customer3, User3],
)
"""
return zip(*list_of_querysets)
def _get_initial_ids(self):
"""
Returns a list of ids of the initial queryset
"""
return self.initial_queryset.order_by("id").values_list("id", flat=True)

You can define a custom Serializer:
class DogSerializer(serializers.ModelSerializer):
class Meta:
model = Dog
fields = ('color','name')
Use it like:
serializer = DogSerializer(Dog.objects.all(), many=True)
print serializer.data enter code here

Did you look at select_related() ?
as in
serializers.serialize('xml', Dog.objects.select_related().all())