I want to prefetch_related to two level of M2M values,
Here is my models.py
class A(models.Model):
name = models.CharField(max_length=40)
b = models.ManyToManyField('B')
class B(models.Model):
name = models.CharField(max_length=40)
c = models.ManyToManyField('C')
class C(models.Model):
name = models.CharField(max_length=40)
d = models.ManyToManyField('D')
And my ORM is
a_obj = A.objects.all().prefetch_related('a__b__c')
And I am trying to access the values like below,
Method A:
for each_obj in a_obj:
print(each_obj.a__b__c)
Method B:
for each_obj in a_obj:
print(each_obj.a.all())
Method A throws an error saying No such value a__b__b for A found
Method B doesn't throw any error, but the number of queries increases to the length of a_obj.
Is there a way to access a__b__c in a single query?
You load both the related B and C models with .prefetch_related(…) [Django-doc]:
a_objs = A.objects.prefetch_related('b__c')
But here .prefetch_related(…) does not change how the items look, it simply loads items. You thus can access these with:
for a in a_objs:
for b in a.b.all():
for c in b.c.all():
print(f'{a} {b} {c}')
You this still access the items in the same way, but here Django will already load the objects in advance to prevent extra queries.
I have two Django models and connected via Foreignkey element and in the second model I need to use the firs model's attribute - example (pseudocode):
Class Category(models.Model):
c_attribute = "Blue"
Class Object(models.Model):
o_category = models.ForeignKey(Category)
o_color = o_category.c_attribute
The key here is the last line - I got error saying that ForeignKey object has no attribute c_attribute.
Thanks
Because o_category is a Key to a Category, not a Category itself!
you can check by type(o_category) to check is not Category!
so you have access to related Cateogry of a Object in other parts of application when connected to database.for example in shell you can write:
c = Category()
c.save()
o = Object(o_category = c, ...) #create Object with desired params
... #some changes to o
o.save()
o.o_category.c_attribute #this will work! :)
You can use to_field='', but that might give you an error as well.
Class Object(models.Model):
o_category = models.ForeignKey(Category)
o_color = models.ForeignKey(Category, to_field="c_attribute")
The best thing is do create a function in your Object model, that would get you the categories c_attribute like so:
def get_c_attribute(self):
return self.o_category.c_attribute
UPDATE 2022: The original ticked #24272 which I opened 8 years ago about this issue is now closed in favor of #33651, which once implemented will give us a new syntax to do this type of prefetches.
============== END OF UPDATE ==============
What's all about?
Django has a GenericRelation class, which adds a “reverse” generic relationship to enable an additional API.
It turns out we can use this reverse-generic-relation for filtering or ordering, but we can't use it inside prefetch_related.
I was wondering if this is a bug, or its not supposed to work, or its something that can be implemented in the feature.
Let me show you with some examples what I mean.
Lets say we have two main models: Movies and Books.
Movies have a Director
Books have an Author
And we want to assign tags to our Movies and Books, but instead of using MovieTag and BookTag models, we want to use a single TaggedItem class with a GFK to Movie or Book.
Here is the model structure:
from django.db import models
from django.contrib.contenttypes.fields import GenericForeignKey, GenericRelation
from django.contrib.contenttypes.models import ContentType
class TaggedItem(models.Model):
tag = models.SlugField()
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = GenericForeignKey('content_type', 'object_id')
def __unicode__(self):
return self.tag
class Director(models.Model):
name = models.CharField(max_length=100)
def __unicode__(self):
return self.name
class Movie(models.Model):
name = models.CharField(max_length=100)
director = models.ForeignKey(Director)
tags = GenericRelation(TaggedItem, related_query_name='movies')
def __unicode__(self):
return self.name
class Author(models.Model):
name = models.CharField(max_length=100)
def __unicode__(self):
return self.name
class Book(models.Model):
name = models.CharField(max_length=100)
author = models.ForeignKey(Author)
tags = GenericRelation(TaggedItem, related_query_name='books')
def __unicode__(self):
return self.name
And some initial data:
>>> from tags.models import Book, Movie, Author, Director, TaggedItem
>>> a = Author.objects.create(name='E L James')
>>> b1 = Book.objects.create(name='Fifty Shades of Grey', author=a)
>>> b2 = Book.objects.create(name='Fifty Shades Darker', author=a)
>>> b3 = Book.objects.create(name='Fifty Shades Freed', author=a)
>>> d = Director.objects.create(name='James Gunn')
>>> m1 = Movie.objects.create(name='Guardians of the Galaxy', director=d)
>>> t1 = TaggedItem.objects.create(content_object=b1, tag='roman')
>>> t2 = TaggedItem.objects.create(content_object=b2, tag='roman')
>>> t3 = TaggedItem.objects.create(content_object=b3, tag='roman')
>>> t4 = TaggedItem.objects.create(content_object=m1, tag='action movie')
So as the docs show we can do stuff like this.
>>> b1.tags.all()
[<TaggedItem: roman>]
>>> m1.tags.all()
[<TaggedItem: action movie>]
>>> TaggedItem.objects.filter(books__author__name='E L James')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>]
>>> TaggedItem.objects.filter(movies__director__name='James Gunn')
[<TaggedItem: action movie>]
>>> Book.objects.all().prefetch_related('tags')
[<Book: Fifty Shades of Grey>, <Book: Fifty Shades Darker>, <Book: Fifty Shades Freed>]
>>> Book.objects.filter(tags__tag='roman')
[<Book: Fifty Shades of Grey>, <Book: Fifty Shades Darker>, <Book: Fifty Shades Freed>]
But, if we try to prefetch some related data of TaggedItem via this reverse generic relation, we are going to get an AttributeError.
>>> TaggedItem.objects.all().prefetch_related('books')
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
Some of you may ask, why I just don't use content_object instead of books here? The reason is, because this only work when we want to:
prefetch only one level deep from querysets containing different type of content_object.
>>> TaggedItem.objects.all().prefetch_related('content_object')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: action movie>]
prefetch many levels but from querysets containing only one type of content_object.
>>> TaggedItem.objects.filter(books__author__name='E L James').prefetch_related('content_object__author')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>]
But, if we want both 1) and 2) (to prefetch many levels from queryset containing different types of content_objects, we can't use content_object.
>>> TaggedItem.objects.all().prefetch_related('content_object__author')
Traceback (most recent call last):
...
AttributeError: 'Movie' object has no attribute 'author_id'
Django thinks that all content_objects are Books, and thus they have an Author.
Now imagine the situation where we want to prefetch not only the books with their author, but also the movies with their director. Here are few attempts.
The silly way:
>>> TaggedItem.objects.all().prefetch_related(
... 'content_object__author',
... 'content_object__director',
... )
Traceback (most recent call last):
...
AttributeError: 'Movie' object has no attribute 'author_id'
Maybe with custom Prefetch object?
>>> TaggedItem.objects.all().prefetch_related(
... Prefetch('content_object', queryset=Book.objects.all().select_related('author')),
... Prefetch('content_object', queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
...
ValueError: Custom queryset can't be used for this lookup.
Some solutions of this problem are shown here. But that's a lot of massage over the data which I want to avoid.
I really like the API coming from the reversed generic relations, it would be very nice to be able to do prefetchs like that:
>>> TaggedItem.objects.all().prefetch_related(
... 'books__author',
... 'movies__director',
... )
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
Or like that:
>>> TaggedItem.objects.all().prefetch_related(
... Prefetch('books', queryset=Book.objects.all().select_related('author')),
... Prefetch('movies', queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
But as you can see, we aways get that AttributeError.
I'm using Django 1.7.3 and Python 2.7.6. And i'm curious why Django is throwing that error? Why is Django searching for an object_id in the Book model?
Why I think this may be a bug?
Usually when we ask prefetch_related to resolve something it can't, we see:
>>> TaggedItem.objects.all().prefetch_related('some_field')
Traceback (most recent call last):
...
AttributeError: Cannot find 'some_field' on TaggedItem object, 'some_field' is an invalid parameter to prefetch_related()
But here, it is different. Django actually tries to resolve the relation... and fails. Is this a bug which should be reported? I have never reported anything to Django so that's why I'm asking here first. I'm unable to trace the error and decide for myself if this is a bug, or a feature which could be implemented.
If you want to retrieve Book instances and prefetch the related tags use Book.objects.prefetch_related('tags'). No need to use the reverse relation here.
You can also have a look at the related tests in the Django source code.
Also the Django documentation states that prefetch_related() is supposed to work with GenericForeignKey and GenericRelation:
prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python. This allows it to prefetch many-to-many and many-to-one objects, which cannot be done using select_related, in addition to the foreign key and one-to-one relationships that are supported by select_related. It also supports prefetching of GenericRelation and GenericForeignKey.
UPDATE: To prefetch the content_object for a TaggedItem you can use TaggedItem.objects.all().prefetch_related('content_object'), if you want to limit the result to only tagged Book objects you could additionally filter for the ContentType (not sure if prefetch_related works with the related_query_name). If you also want to get the Author together with the book you need to use select_related() not prefetch_related() as this is a ForeignKey relationship, you can combine this in a custom prefetch_related() query:
from django.contrib.contenttypes.models import ContentType
from django.db.models import Prefetch
book_ct = ContentType.objects.get_for_model(Book)
TaggedItem.objects.filter(content_type=book_ct).prefetch_related(
Prefetch(
'content_object',
queryset=Book.objects.all().select_related('author')
)
)
prefetch_related_objects to the rescue.
Starting from Django 1.10 (Note: it still presents in the previous versions, but was not part of the public API.), we can use prefetch_related_objects to divide and conquer our problem.
prefetch_related is an operation, where Django fetches related data after the queryset has been evaluated (doing a second query after the main one has been evaluated). And in order to work, it expects the items in the queryset to be homogeneous (the same type). The main reason the reverse generic generation does not work right now is that we have objects from different content types, and the code is not yet smart enough to separate the flow for different content types.
Now using prefetch_related_objects we do fetches only on a subset of our queryset where all the items will be homogeneous. Here is an example:
from django.db import models
from django.db.models.query import prefetch_related_objects
from django.core.paginator import Paginator
from django.contrib.contenttypes.models import ContentType
from tags.models import TaggedItem, Book, Movie
tagged_items = TaggedItem.objects.all()
paginator = Paginator(tagged_items, 25)
page = paginator.get_page(1)
# prefetch books with their author
# do this only for items where
# tagged_item.content_object is a Book
book_ct = ContentType.objects.get_for_model(Book)
tags_with_books = [item for item in page.object_list if item.content_type_id == book_ct.id]
prefetch_related_objects(tags_with_books, "content_object__author")
# prefetch movies with their director
# do this only for items where
# tagged_item.content_object is a Movie
movie_ct = ContentType.objects.get_for_model(Movie)
tags_with_movies = [item for item in page.object_list if item.content_type_id == movie_ct.id]
prefetch_related_objects(tags_with_movies, "content_object__director")
# This will make 5 queries in total
# 1 for page items
# 1 for books
# 1 for book authors
# 1 for movies
# 1 for movie directors
# Iterating over items wont make other queries
for item in page.object_list:
# do something with item.content_object
# and item.content_object.author/director
print(
item,
item.content_object,
getattr(item.content_object, 'author', None),
getattr(item.content_object, 'director', None)
)
Building on Bernhard's answer, which has a code-snippet at the end that throws the below error in reality:
ValueError: Custom queryset can't be used for this lookup.
I've overridden the GenericForeignKey to actually allow the behavior, how bulletproof this implementation is, is unknown to me at this time but it seems to get what I need done, so I'm posting it here, hopefully it'll help out others. Please lookout for START CHANGES and END CHANGES tags to see my changes to the original django code.
from django.contrib.contenttypes.fields import GenericForeignKey as BaseGenericForeignKey
class CustomGenericForeignKey(BaseGenericForeignKey):
def get_prefetch_queryset(self, instances, queryset=None):
"""
Enable passing queryset to get_prefetch_queryset when using GenericForeignKeys but only works when a single
content type is being queried
"""
# START CHANGES
# if queryset is not None:
# raise ValueError("Custom queryset can't be used for this lookup.")
# END CHANGES
# For efficiency, group the instances by content type and then do one
# query per model
fk_dict = defaultdict(set)
# We need one instance for each group in order to get the right db:
instance_dict = {}
ct_attname = self.model._meta.get_field(self.ct_field).get_attname()
for instance in instances:
# We avoid looking for values if either ct_id or fkey value is None
ct_id = getattr(instance, ct_attname)
if ct_id is not None:
fk_val = getattr(instance, self.fk_field)
if fk_val is not None:
fk_dict[ct_id].add(fk_val)
instance_dict[ct_id] = instance
ret_val = []
for ct_id, fkeys in fk_dict.items():
instance = instance_dict[ct_id]
# START CHANGES
if queryset is not None:
assert len(fk_dict) == 1 # only a single content type is allowed, else undefined behavior
ret_val.extend(queryset.filter(pk__in=fkeys))
else:
ct = self.get_content_type(id=ct_id, using=instance._state.db)
ret_val.extend(ct.get_all_objects_for_this_type(pk__in=fkeys))
# END CHANGES
# For doing the join in Python, we have to match both the FK val and the
# content type, so we use a callable that returns a (fk, class) pair.
def gfk_key(obj):
ct_id = getattr(obj, ct_attname)
if ct_id is None:
return None
else:
model = self.get_content_type(id=ct_id,
using=obj._state.db).model_class()
return (model._meta.pk.get_prep_value(getattr(obj, self.fk_field)),
model)
return (
ret_val,
lambda obj: (obj.pk, obj.__class__),
gfk_key,
True,
self.name,
True,
)
this is the representation of my models:
class B(models.Model):
"""I'm a dummy model, so doesn't pay atention of what I do"""
name = models.CharField(max_length=250)
class A(models.Model):
name = models.CharField(max_length=250)
many_b = models.ManyToManyField(B)
Now, suppose I have a list of B objects. And a single A object that will be related to that Bs. Something like this:
a = A.objects.get(id=1)
list_of_b = [B<name='B1'>,B<name='B2'>,B<name='B3'>,]
The way I relate them now is this:
for b_object in list_of_b:
a.many_b.add(b_object)
Is there any way to add all the B objects in a single transaction? Maybe in a single method, like:
a.many_b.addList(b) #This doesn't exist
From the docs:
>>> john = Author.objects.create(name="John")
>>> paul = Author.objects.create(name="Paul")
>>> george = Author.objects.create(name="George")
>>> ringo = Author.objects.create(name="Ringo")
>>> entry.authors.add(john, paul, george, ringo)
So if you have a list, use argument expansion:
a.many_b.add(*list_of_b)
I guess what you want is a kind of bulk insert right?
As far as I know this is just available in the Django TRUNK not in 1.3!
check it out some tutorial:
http://www.caktusgroup.com/blog/2011/09/20/bulk-inserts-django/
Apart from one example in the docs, I can't find any documentation on how exactly django chooses the name with which one can access the child object from the parent object. In their example, they do the following:
class Place(models.Model):
name = models.CharField(max_length=50)
address = models.CharField(max_length=80)
def __unicode__(self):
return u"%s the place" % self.name
class Restaurant(models.Model):
place = models.OneToOneField(Place, primary_key=True)
serves_hot_dogs = models.BooleanField()
serves_pizza = models.BooleanField()
def __unicode__(self):
return u"%s the restaurant" % self.place.name
# Create a couple of Places.
>>> p1 = Place(name='Demon Dogs', address='944 W. Fullerton')
>>> p1.save()
>>> p2 = Place(name='Ace Hardware', address='1013 N. Ashland')
>>> p2.save()
# Create a Restaurant. Pass the ID of the "parent" object as this object's ID.
>>> r = Restaurant(place=p1, serves_hot_dogs=True, serves_pizza=False)
>>> r.save()
# A Restaurant can access its place.
>>> r.place
<Place: Demon Dogs the place>
# A Place can access its restaurant, if available.
>>> p1.restaurant
So in their example, they simply call p1.restaurant without explicitly defining that name. Django assumes the name starts with lowercase. What happens if the object name has more than one word, like FancyRestaurant?
Side note: I'm trying to extend the User object in this way. Might that be the problem?
If you define a custom related_name then it will use that, otherwise it will lowercase the entire model name (in your example .fancyrestaurant). See the else block in django.db.models.related code:
def get_accessor_name(self):
# This method encapsulates the logic that decides what name to give an
# accessor descriptor that retrieves related many-to-one or
# many-to-many objects. It uses the lower-cased object_name + "_set",
# but this can be overridden with the "related_name" option.
if self.field.rel.multiple:
# If this is a symmetrical m2m relation on self, there is no reverse accessor.
if getattr(self.field.rel, 'symmetrical', False) and self.model == self.parent_model:
return None
return self.field.rel.related_name or (self.opts.object_name.lower() + '_set')
else:
return self.field.rel.related_name or (self.opts.object_name.lower())
And here's how the OneToOneField calls it:
class OneToOneField(ForeignKey):
... snip ...
def contribute_to_related_class(self, cls, related):
setattr(cls, related.get_accessor_name(),
SingleRelatedObjectDescriptor(related))
The opts.object_name (referenced in the django.db.models.related.get_accessor_name) defaults to cls.__name__.
As for
Side note: I'm trying to extend the
User object in this way. Might that be
the problem?
No it won't, the User model is just a regular django model. Just watch out for related_name collisions.