Develop Reference

django-elasticsearch-dsl does not return all records that match the query condition

I'm writing a site on Django. I'm developing a site search system using django-elasticsearch-dsl. But I noticed a problem that not all records that meet the search condition are displayed.
For example, I know that I have 6 books in my database that contain the word 'Python', but when I use the command
books = BookDocument.search().query('match', title='Python')
django-elasticsearch-dsl gives me only 3 entries
books = BookDocument.search().query('match', title='Python')
for i in books:
print(i.title)
|||||||||||||
Укус Python
Програмуємо на Python
Django:Практика створення Web-сайтів на Python
And the remaining 3 are not displayed
Here is my BookDocument class:
#registry.register_document
class BookDocument(Document):
"""
Індексація таблиці БД Book для Elasticsearch-пошуку
Згідно із документацією django-elasticsearch-dsl
https://django-elasticsearch-dsl.readthedocs.io/en/latest/fields.html
"""
image = FileField()
category = ObjectField(properties={
'title': TextField(),
'pk': IntegerField()
})
authors = NestedField(properties={
'pk': IntegerField(),
'name': TextField()
})
class Index:
name = 'book'
settings = {
'number_of_shards': 1,
'number_of_replicas': 0
}
class Django:
model = Book
fields = [
'id',
'title',
'description',
]
related_models = [Category, Author]
def get_instances_from_related(self, related_instance):
if isinstance(related_instance, Category):
return related_instance.book_set.all()
if isinstance(related_instance, Author):
return related_instance.book_set.all()
Maybe someone has already encountered this and knows how to fix it

How to make Many-to-many mutations in Django-Graphene?

I'm working in Django 3.2 and graphene-django 2.15.
I'm still learning how to use Graphene by the way.
Note: I'm not allowed to share all the code so I rewrote it for the purpose of this question. Please notify me if you've found any error unrelated to the question.
I have an Team model which has a Many-to-Many relationship with the default Django Group model:
from django.db import models
from django.contrib.auth.models import Group
class Team(models.Model):
team_id = models.IntegerField(unique=True) # Custom ID not related to Django's pk
name = models.CharField(max_length=255)
groups = models.ManyToManyField(Group, blank=True)
Here is my schema:
import graphene
from django.contrib.auth.models import Group
from graphene_django import DjangoObjectType
from .models import Team
class TeamType(DjangoObjectType):
class Meta:
model = Team
class GroupType(DjangoObjectType):
class Meta:
model = Group
class GroupInput(graphene.InputObjectType):
id = graphene.ID(required=True)
class UpdateTeam(graphene.Mutation):
team = graphene.Field(TeamType)
class Arguments:
team_id = graphene.ID(required=True)
name = graphene.String(required=True)
groups_id = graphene.List(GroupInput, required=True)
def mutate(self, info, team_id, name, groups_id):
team = Team.objects.get(pk=team_id)
team.name = name
team.groups = Group.objects.filter(pk__in=groups_id)
team.save()
return UpdateTeam(team=team)
class TeamMutations(graphene.ObjectType):
update_team = UpdateTeam.Field()
class Mutation(TeamMutations, graphene.ObjectType):
pass
schema = graphene.schema(query=Query, mutation=Mutation)
When I perform this query:
mutation{
updateTeam(
teamId: 65961826547,
name: "My Team Name",
groupsId: [{id: 1}, {id: 2}]
) {
team {
team_id,
name,
groups {
name,
}
}
}
}
I get this error:
{
"errors": [
{
"message": "Field 'id' expected a number but got {'id': '1'}.",
"locations": [
{
"line": 2,
"column": 3
}
],
"path": [
"updateTeam"
]
}
],
"data": {
"updateTeam": null
}
}
I don't really understand how Many-to-may relationships are managed by Graphene.
Does someone has a solution and some explanations for me? Thanks a lot.

I found the solution.
At first, I thought there was something related to graphene, especially these InputObjectTypes, I didn't get correctly.
But the issue is actually very simple.
The GroupInput is expecting a single value, which it an ID.
class GroupInput(graphene.InputObjectType):
id = graphene.ID(required=True)
Because I put it in graphene.List(), I'm now expecting a list of ID's:
class UpdateTeam(graphene.Mutation):
...
class Arguments:
...
groups_id = graphene.List(GroupInput, required=True)
But in my API call, instead of giving an actual list, I gave a dict:
mutation{
updateTeam(
...
groupsId: [{id: 1}, {id: 2}]
) {
...
}
So this works:
mutation{
updateTeam(
...
groupsId: [1, 2]
) {
...
}
Note 1:
graphene.ID also accepts ids given as strings:
groupsId: ["1", "2"]
Note 2:
I actually removed my GroupInput and put the graphene.ID field directly in the graphene.List:
class UpdateTeam(graphene.Mutation):
...
class Arguments:
...
groups_id = graphene.List(graphene.ID, required=True)

How to add extra key-value to QuerySet lists?

I am working with Django,i need to retrieve data from multiple database, which has different database name but with same table column structure.
So I use model.using(database).all()to get queryset and merge them into one.
I want to add extra databasename to indicate the data's database name, this is my code.
model:
class Sections(models.Model):
apply_id = models.PositiveIntegerField()
pathology_id = models.CharField(max_length=128)
user_id = models.PositiveIntegerField()
updated_at = models.DateTimeField(blank=True, null=True)
get_queryset:
def get_queryset(self):
slideset = []
database_config = ['database1', 'database2', 'database3']
for i, x in database_config:
slides = Sections.objects.using(x).all()
#### I want to add extra databasename column in every query object.
for x1 in slides:
x1.databasename = x
######
slideset.append(slides)
# merge QuerySet
query = functools.reduce(lambda a, b: a|b, slideset)
return query.order_by("updated_at").reverse()
the one return will be :
{
"apply_id": 1123,
"pathology_id": 1235,
"user_id": 1,
"updated_at": "202106011430",
# add extra databasename.
"databasename": "database1".
}
Because the column can't be modify, so I had to leave Sections model unchange, just add extra key-value to query, can someone help me on that?

thanks to #Abdul Aziz Barkat
use annotate
from django.db.models import CharField, Value
slides = Sections.objects.using(x).annotate(databasename=Value(databasename, output_field=CharField())

Django get total count and count by unique value in queryset

I have models Software and Domain described loosely as:
class Software(models.Model)
id = models.BigInteger(primary_key=True, db_index=True, null=False)
company = models.ForeignKey('Company')
domain = models.ForeignKey('Domain')
type = models.CharField(null=False)
vendor = models.CharField(null=False)
name = models.CharField(null=False)
class Domain(models.Model):
id = models.BigInteger(primary_key=True, db_index=True, null=False)
type = models.CharField()
importance = models.DecimalField(max_digits=11, decimal_places=10, null=False)
And I get a Software queryset with:
qs = Software.objects.filter(company=c).order_by('vendor')
The desired output should have an aggregated Domain importance with total count for each unique Software, i.e.
[
{
'type': 'type_1', \
'vendor': 'ajwr', | - unique together
'name': 'nginx', /
'domains': {
'total_count': 4,
'importance_counts': [0.1: 1, 0.5: 2, 0.9: 1] # sum of counts = total_count
},
},
{
...
},
]
I feel like the first step here should be to just group the type, vendor, name by Domain so each Software object has a list of Domains instead of just one but I'm not sure how to do that. Doing this in memory would make it a lot easier but it seems like it would be a lot slower than using querysets / SQL.

So I would do it like this:
from django.db.models import Sum
qs = Software.objects.filter(company=c).prefetch_related(
'domain'
).annotate(
total_count=Sum('domain__importance')
).order_by('vendor')
output = []
for obj in qs:
domains = obj.domain.all() # using prefetched domains, no db query
output.append({
# ...
'domains': {
'total_count': obj.total_count,
'importance_counts': [d.importance for d in domains]
}
})
And I belive it should be fast enough. Only if finding that it isn't I would try to improve. Remember "Premature optimization is the root of all evil"

QuerySet object has no attribute

I got a problem when I select the distinct value from DB.
Here is my model:
class Shift(models.Model):
shiftid = models.CharField(max_length=15)
shiftdesc = models.CharField(blank = False, null= False, max_length=20)
dayname = models.CharField(blank = False, null= False, max_length=20)
class Meta:
unique_together = ('shiftid','dayname')
This is the resulting data structure:
shiftid shiftdesc dayname
shift1 desc1 1
shift1 desc2 1
shift1 desc1 1
I want it to be like this:
shiftid shiftdesc dayname
shift1 desc1 1
shift1 desc2 1
I am trying to select the records like this:
#action(methods=['get'], detail=False)
def shiftsum(self, request):
newest = self.get_queryset().order_by('shiftid','dayname').values('shiftid','dayname').distinct()
serializer = self.get_serializer_class()(newest)
return Response(serializer.data)
When I try like that I always get this error:
QuerySet object has no attribute 'shiftid'
Also, I would like to know how to select the distinct value? I am new in Django and appreciate every help.

Serializers don't handle lists of objects by default. You need to pass many=True to tell it to process each item and output a list (rest framework docs).
self.get_serializer_class()(newest, many=True)
This will give you a list of days, like you expect:
[
{ "shiftid": "shift1", "dayname": "1" }
]
Distinct
Your distinct is fine. An example distinct query would look just like yours:
User.objects.values('field').order_by('field').distinct()
Final Query
The issue with your final query is that you are only selecting 2 of the 3 fields that you want, excluding shiftdesc.
There isn't really a logical way to get that value, since by definition you start with N and end up with 1.
If you just want ANY value for it, say for debugging or display, you can use .annotate() like this:
query = (
Shift.objects.values('shiftid', 'dayname')
.annotate(shiftdesc=Max('shiftdesc'))
.annotate(ct=Count('*')) # get count of rows in group
.order_by('shiftid', 'dayname')
.distinct()
)
Look into annotations/aggregations, more advanced stuff can be done that may help out, and some database specific stuff that can be really useful.
Full Example
Here is a full example, using the default django User table. You have not provided enough information to further debug it.
import json
from rest_framework.serializers import *
User.objects.create(email='x1#e', first_name='Angela', last_name='Smith')
User.objects.create(email='x2#e', first_name='James', last_name='Smith')
User.objects.create(email='x3#e', first_name='James', last_name='Joyce')
query = User.objects.values('last_name') \
.order_by('last_name') \
.annotate(first_name=Max('first_name')) \
.annotate(ct=Count('email')).distinct()
class X(Serializer):
last_name = CharField()
first_name = CharField()
ct = IntegerField()
data = X(query, many=True).data
print(json.dumps(data, indent=4))
[
{
"last_name": "Joyce",
"first_name": "James",
"ct": 1
},
{
"last_name": "Smith",
"first_name": "James",
"ct": 2
}
]