Develop Reference

Wagtail Page model transitive search on custom fields [duplicate]

I'm using Wagtail, and I want to filter a selection of child pages by a Foreign Key. I've tried the following and I get the error django.core.exceptions.FieldError: Cannot resolve keyword 'use_case' into field when I try children = self.get_children().specific().filter(use_case__slug=slug):
class AiLabResourceMixin(models.Model):
parent_page_types = ['AiLabResourceIndexPage']
use_case = models.ForeignKey(AiLabUseCase, on_delete=models.PROTECT)
content_panels = ArticlePage.content_panels + [
FieldPanel('use_case', widget=forms.Select())
]
class Meta:
abstract = True
class AiLabCaseStudy(AiLabResourceMixin, ArticlePage):
pass
class AiLabBlogPost(AiLabResourceMixin, ArticlePage):
pass
class AiLabExternalLink(AiLabResourceMixin, ArticlePage):
pass
class AiLabResourceIndexPage(RoutablePageMixin, BasePage):
parent_page_types = ['AiLabHomePage']
subpage_types = ['AiLabCaseStudy', 'AiLabBlogPost', 'AiLabExternalLink']
max_count = 1
#route(r'^$')
def all_resources(self, request):
children = self.get_children().specific()
return render(request, 'ai_lab/ai_lab_resource_index_page.html', {
'page': self,
'children': children,
})
#route(r'^([a-z0-9]+(?:-[a-z0-9]+)*)/$')
def filter_by_use_case(self, request, slug):
children = self.get_children().specific().filter(use_case__slug=slug)
return render(request, 'ai_lab/ai_lab_resource_index_page.html', {
'page': self,
'children': children,
})
I've seen this answer, but this assumes I only have one type of page I want to filter. Using something like AiLabCaseStudy.objects.filter(use_case__slug=slug) works, but this only returns AiLabCaseStudys, not AiLabBlogPosts or AiLabExternalLinks.
Any ideas?

At the database level, there is no efficient way to run the filter against all page types at once. Since AiLabResourceMixin is defined as abstract = True, this class has no representation of its own within the database - instead, the use_case field is defined separately for each of AiLabCaseStudy, AiLabBlogPost and AiLabExternalLink. As a result, there's no way for Django or Wagtail to turn .filter(use_case__slug=slug) into a SQL query, since use_case refers to three different places in the database.
A couple of possible ways around this:
If your data model allows, restructure it to use multi-table inheritance - this looks fairly similar to your current definition, except without the abstract = True:
class AiLabResourcePage(ArticlePage):
use_case = models.ForeignKey(AiLabUseCase, on_delete=models.PROTECT)
class AiLabCaseStudy(AiLabResourcePage):
pass
class AiLabBlogPost(AiLabResourcePage):
pass
class AiLabExternalLink(AiLabResourcePage):
pass
AiLabResourcePage will then exist in its own right in the database, and you can query its use_case field with an expression like: AiLabResourcePage.objects.child_of(self).filter(use_case__slug=slug).specific(). There'll be a small performance impact here, since Django has to pull data from one additional table to construct these page objects.
Run a preliminary query on each specific page type to retrieve the matching page IDs, before running the final query with specific():
case_study_ids = list(AiLabCaseStudy.objects.child_of(self).filter(use_case__slug=slug).values_list('id', flat=True))
blog_post_ids = list(AiLabBlogPost.objects.child_of(self).filter(use_case__slug=slug).values_list('id', flat=True))
external_link_ids = list(AiLabExternalLink.objects.child_of(self).filter(use_case__slug=slug).values_list('id', flat=True))
children = Page.objects.filter(id__in=(case_study_ids + blog_post_ids + external_link_ids)).specific()

Try:
children = self.get_children().filter(use_case__slug=slug).specific()

django - prefetch only the newest record?

I am trying to prefetch only the latest record against the parent record.
my models are as such
class LinkTargets(models.Model):
device_circuit_subnet = models.ForeignKey(DeviceCircuitSubnets, verbose_name="Device", on_delete=models.PROTECT)
interface_index = models.CharField(max_length=100, verbose_name='Interface index (SNMP)', blank=True, null=True)
get_bgp = models.BooleanField(default=False, verbose_name="get BGP Data?")
dashboard = models.BooleanField(default=False, verbose_name="Display on monitoring dashboard?")
class LinkData(models.Model):
link_target = models.ForeignKey(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT)
interface_description = models.CharField(max_length=200, verbose_name='Interface Description', blank=True, null=True)
...
The below query fails with the error
AttributeError: 'LinkData' object has no attribute '_iterable_class'
Query:
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.all().order_by('-id')[0]
)
)
I thought about getting LinkData instead and doing a select related but ive no idea how to get only 1 record for each link_target_id
link_data = LinkData.objects.filter(link_target__dashboard=True) \
.select_related('link_target')..?
EDIT:
using rtindru's solution, the pre fetched seems to be empty. there is 6 records in there currently, atest 1 record for each of the 3 LinkTargets
>>> link_data[0]
<LinkTargets: LinkTargets object>
>>> link_data[0].linkdata_set.all()
<QuerySet []>
>>>

The reason is that Prefetch expects a Django Queryset as the queryset parameter and you are giving an instance of an object.
Change your query as follows:
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.filter(pk=LinkData.objects.latest('id').pk)
)
)
This does have the unfortunate effect of undoing the purpose of Prefetch to a large degree.
Update
This prefetches exactly one record globally; not the latest LinkData record per LinkTarget.
To prefetch the max LinkData for each LinkTarget you should start at LinkData: you can achieve this as follows:
LinkData.objects.filter(link_target__dashboard=True).values('link_target').annotate(max_id=Max('id'))
This will return a dictionary of {link_target: 12, max_id: 3223}
You can then use this to return the right set of objects; perhaps filter LinkData based on the values of max_id.
That will look something like this:
latest_link_data_pks = LinkData.objects.filter(link_target__dashboard=True).values('link_target').annotate(max_id=Max('id')).values_list('max_id', flat=True)
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.filter(pk__in=latest_link_data_pks)
)
)

The following works on PostgreSQL. I understand it won't help OP, but it might be useful to somebody else.
from django.db.models import Count, Prefetch
from .models import LinkTargets, LinkData
link_data_qs = LinkData.objects.order_by(
'link_target__id',
'-id',
).distinct(
'link_target__id',
)
qs = LinkTargets.objects.prefetch_related(
Prefetch(
'linkdata_set',
queryset=link_data_qs,
)
).all()

LinkData.objects.all().order_by('-id')[0] is not a queryset, it is an model object, hence your error.
You could try LinkData.objects.all().order_by('-id')[0:1] which is indeed a QuerySet, but it's not going to work. Given how prefetch_related works, the queryset argument must return a queryset that contains all the LinkData records you need (this is then further filtered, and the items in it joined up with the LinkTarget objects). This queryset only contains one item, so that's no good. (And Django will complain "Cannot filter a query once a slice has been taken" and raise an exception, as it should).
Let's back up. Essentially you are asking an aggregation/annotation question - for each LinkTarget, you want to know the most recent LinkData object, or the 'max' of an 'id' column. The easiest way is to just annotate with the id, and then do a separate query to get all the objects.
So, it would look like this (I've checked with a similar model in my project, so it should work, but the code below may have some typos):
linktargets = (LinkTargets.objects
.filter(dashboard=True)
.annotate(most_recent_linkdata_id=Max('linkdata_set__id'))
# Now, if we need them, lets collect and get the actual objects
linkdata_ids = [t.most_recent_linkdata_id for t in linktargets]
linkdata_objects = LinkData.objects.filter(id__in=linkdata_ids)
# And we can decorate the LinkTarget objects as well if we want:
linkdata_d = {l.id: l for l in linkdata_objects}
for t in linktargets:
if t.most_recent_linkdata_id is not None:
t.most_recent_linkdata = linkdata_d[t.most_recent_linkdata_id]
I have deliberately not made this into a prefetch that masks linkdata_set, because the result is that you have objects that lie to you - the linkdata_set attribute is now missing results. Do you really want to be bitten by that somewhere down the line? Best to make a new attribute that has just the thing you want.

Tricky, but it seems to work:
class ForeignKeyAsOneToOneField(models.OneToOneField):
def __init__(self, to, on_delete, to_field=None, **kwargs):
super().__init__(to, on_delete, to_field=to_field, **kwargs)
self._unique = False
class LinkData(models.Model):
# link_target = models.ForeignKey(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT)
link_target = ForeignKeyAsOneToOneField(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT, related_name='linkdata_helper')
interface_description = models.CharField(max_length=200, verbose_name='Interface Description', blank=True, null=True)
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_helper',
queryset=LinkData.objects.all().order_by('-id'),
'linkdata'
)
)
# Now you can access linkdata:
link_data[0].linkdata
Ofcourse with this approach you can't use linkdata_helper to get related objects.

This is not a direct answer to you question, but solves the same problem. It is possible annotate newest object with a subquery, which I think is more clear. You also don't have to do stuff like Max("id") to limit the prefetch query.
It makes use of django.db.models.functions.JSONObject (added in Django 3.2) to combine multiple fields:
MainModel.objects.annotate(
last_object=RelatedModel.objects.filter(mainmodel=OuterRef("pk"))
.order_by("-date_created")
.values(
data=JSONObject(
id="id", body="body", date_created="date_created"
)
)[:1]
)

Django filtering based on count of related model

I have the following working code:
houses_of_agency = House.objects.filter(agency_id=90)
area_list = AreaHouse.objects.filter(house__in=houses_of_agency).values('area')
area_ids = Area.objects.filter(area_id__in=area_list).values_list('area_id', flat=True)
That returns a queryset with a list of area_ids. I want to filter further so that I only get area_ids where there are more than 100 houses belonging to the agency.
I tried the following adjustment:
houses_of_agency = House.objects.filter(agency_id=90)
area_list = AreaHouse.objects.filter(house__in=houses_of_agency).annotate(num_houses=Count('house_id')).filter(num_houses__gte=100).values('area')
area_ids = Area.objects.filter(area_id__in=area_list).values_list('area_id', flat=True)
But it returns an empty queryset.
My models (simplified) look like this:
class House(TimeStampedModel):
house_pk = models.IntegerField()
agency = models.ForeignKey(Agency, on_delete=models.CASCADE)
class AreaHouse(TimeStampedModel):
area = models.ForeignKey(Area, on_delete=models.CASCADE)
house = models.ForeignKey(House, on_delete=models.CASCADE)
class Area(TimeStampedModel):
area_id = models.IntegerField(primary_key=True)
parent = models.ForeignKey('self', null=True)
name = models.CharField(null=True, max_length=30)
Edit: I'm using MySQL for the database backend.

You are querying for agency_id with just one underscore. I corrected your queries below. Also, in django it's more common to use pk instead of id however the behaviour is the same. Further, there's no need for three separate queries as you can combine everything into one.
Also note that your fields area_id and house_pk are unnecessary, django automatically creates primary key fields which area accessible via modelname__pk.
# note how i inlined your first query in the .filter() call
area_list = AreaHouse.objects \
.filter(house__agency__pk=90) \
.annotate(num_houses=Count('house')) \ # <- 'house'
.filter(num_houses__gte=100) \
.values('area')
# note the double underscore
area_ids = Area.objects.filter(area__in=area_list)\
.values_list('area__pk', flat=True)
you could simplify this even further if you don't need the intermediate results. here are both queries combined:
area_ids = AreaHouse.objects \
.filter(house__agency__pk=90) \
.annotate(num_houses=Count('house')) \
.filter(num_houses__gte=100) \
.values_list('area__pk', flat=True)
Finally, you seem to be manually defining a many-to-many relation in your model (through AreaHouse). There are better ways of doing this, please read the django docs.

Django ManyToManyField Error when saving in admin?

What is wrong with my code?
class Group(ImageModel):
title = models.CharField(verbose_name = "Title", max_length=7)
photos = models.ManyToManyField('Photo', related_name='+',
verbose_name=_('Photo'),
null=True, blank=True)
.....
pid = Photo.objects.get(image = str_path)
gid= Group.objects.get(id = self.id)
self.save_photos(gid, pid)
....
def save_photos(self, gid, pid):
group_photo = GroupPhotos(groupupload=gid.id,
photo=pid.id
)
group_photo.save()
and my GroupPhotos models is:
class GroupPhotos(models.Model):
groupupload = models.ForeignKey('Group')
photo = models.ForeignKey('Photo')
class Meta:
db_table = u'group_photos'
when i want to save it from admin panel i am getting value error sth like this:
Cannot assign "38": "GroupPhotos.groupupload" must be a "Group" instance.
with group_photo = GroupPhotos(groupupload=gid, photo=pid) defination it is working but there is no any changes in GroupPhotos table(group_photos). printing this print pid.id,' >>> ',gid.id i am getting true relation...
UPDATE:
I have been working since morning, but no progress... i have also tried this but nothing changed:
pid = Photo.objects.get(image = str_path)
ger = Group.objects.get(id = self.id)
ger.title = self.title
ger.save()
ger.photos.add(pid)

The error is here:
group_photo = GroupPhotos(groupupload=gid.id, photo=pid.id)
The arguments to groupupload and photo should be instances of Group and Photo respectively. Try the following:
group_photo = GroupPhotos(groupupload=gid, photo=pid)
In other words, when creating an object you need to pass arguments of the expected type and not an integer (which may be the primary key key of the desired object but it also might not, which is why you need to pass an object of the correct type).

i have solved my problem with adding through option to my manytomanyfield:
photos = models.ManyToManyField('Photo', related_name='+',
verbose_name=_('Photo'),
null=True, blank=True, through=GroupPhotos)
some info about ManyToManyField.through here:
Django will automatically generate a table to manage many-to-many
relationships. However, if you want to manually specify the
intermediary table, you can use the through option to specify the
Django model that represents the intermediate table that you want to
use.
The most common use for this option is when you want to associate extra data with a many-to-many relationship.

Google app engine python problem

I'm having a problem with the datastore trying to replicate a left join to find items from model a that don't have a matching relation in model b:
class Page(db.Model):
url = db.StringProperty(required=True)
class Item(db.Model):
page = db.ReferenceProperty(Page, required=True)
name = db.StringProperty(required=True)
I want to find any pages that don't have any associated items.

You cannot query for items using a "property is null" filter. However, you can add a boolean property to Page that signals if it has items or not:
class Page(db.Model):
url = db.StringProperty(required=True)
has_items = db.BooleanProperty(default=False)
Then override the "put" method of Item to flip the flag. But you might want to encapsulate this logic in the Page model (maybe Page.add_item(self, *args, **kwargs)):
class Item(db.Model):
page = db.ReferenceProperty(Page, required=True)
name = db.StringProperty(required=True)
def put(self):
if not self.page.has_items:
self.page.has_items = True
self.page.put()
return db.put(self)
Hence, the query for pages with no items would be:
pages_with_no_items = Page.all().filter("has_items =", False)

The datastore doesn't support joins, so you can't do this with a single query. You need to do a query for items in A, then for each, do another query to determine if it has any matching items in B.

Did you try it like :
Page.all().filter("item_set = ", None)
Should work.