Hi i have two models like this,
class Sample(models.Model):
name = models.CharField(max_length=256) ##
processid = models.IntegerField(default=0) #
class Process(models.Model):
sample = models.ForeignKey(Sample, blank=False, null=True, on_delete=models.SET_NULL, related_name="process_set")
end_at = models.DateTimeField(null=True, blank=True)
and I want to join Sample and Process model. Because Sample is related to process and I want to get process information with sample .
SELECT sample.id, sample.name, process.endstat
FROM sample
INNER JOIN process
ON sample.processid = process.id
AND process.endstat = 1;
(i'm using SQLite)
I used
sample_list = sample_list.filter(process_set__endstat=1))
but it returned
SELECT sample.id, sample.name
FROM sample
INNER JOIN process
ON (sample.id = process.sample_id)
AND process.endstat = 1)
This is NOT what I want.
How can i solve the problem?
This should work for you
Process.objects.filter(end_at=1).values('sample__id','sample__name','end_at')
.values() method returns selective table fields.
I'm assuming sample_list = Sample.objects.
When you are filtering a model, only the fields defined in the model are selected. In your example, id and processid. If you want to retrieve values from related models as a single record you need to use values or values_list. To get the desired query you have to do this
sample_list = sample_list.filter(process_set__endstat=1).values('id', 'name', 'process__endstat')
Btw, Django does JOIN on the foreign key field. So, you can't get ON sample.processid = process.id since processid is not a ForeignKey field.
Reference:
https://docs.djangoproject.com/en/4.0/ref/models/querysets/#values
I found JOIN not on foreign key field in django.
sample_list = sample_list.filter(processid__in=Process.objects.filter(endstat=1)
I used the medthod of
Django-queryset join without foreignkey
Related
I fairly new to Django and stuck with creating a left join in Django. I tried so many, but none of them seems to be working:
The query I want to translate to Django is:
select ssc.id
,mgz.Title
,tli.id
,tli.Time
from Subscription ssc
join Person prs
on ssc.PersonID = prs.id
and prs.id = 3
join Magazine mgz
on mgz.id = ssc.MagazineID
and mgz.from <= date.today()
and mgz.until > date.today()
left join TimeLogedIn tli
on tli.SubscriptionID = ssc.id
and tli.DateOnline = date.today()
The model I'm using looks like this:
class Magazine(models.Model):
Title = models.CharField(max_length=100L)
from = models.Datefield()
until = models.Datefield()
Persons = models.ManyToManyField(Person, through='Subscription')
class Person(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
Magazines = models.ManyToManyField(Magazine, through='Subscription')
class Subscription(models.Model):
MagazineID = models.ForeignKey(Magazine,on_delete=models.CASCADE)
PersonID = models.ForeignKey(Person,on_delete=models.CASCADE)
class TimeLogedIn(models.Model):
SubscriptionID = models.ForeignKey('Subscription', on_delete=models.CASCADE)
DateOnline = models.DateField()
Time = models.DecimalField(max_digits=5, decimal_places=2)
Like I said, tried so many but no succes and now I don't know how to do this in Django ORM , is it even possible? I created already a raw-query and this is working ok, but how to create this in Django ORM?
You can use field lookups lte and gt to filter your objects and then values() method.
You can also querying in the opposite direction and use Q objects for null values:
from django.db.models import Q
Subscription.objects.filter(
PersonID_id=3,
MagazineID__from__lte=date.today(),
MagazineID__until__gt=date.today()
).filter(
Q(TimeLogedIn__DateOnline=date.today()) | Q(TimeLogedIn__DateOnline__isnull=True)
).values("id", "MagazineID__Title", "TimeLogedIn__id", "TimeLogedIn__Time")
OR from TimeLogedIn:
TimeLogedIn.objects.filter(DateOnline=date.today()).filter(
SubscriptionID__MagazineID__from__lte=date.today(),
SubscriptionID__MagazineID__util__gt=date.today()
).values(
"SubscriptionID_id", "SubscriptionID__MagazineID__Title", "id", "Time"
)
Querysets also have the query attribute that contains the sql query to be executed, you can see it like following:
print(TimeLogedIn.objects.filter(...).values(...).query)
Note: Behind the scenes, Django appends "_id" to the field name to create its database column name. Therefore it should be
subscription, instead of SubscriptionID.
You can also use prefetch_related() and select_related() to prevent multiple database hits:
SubscriptionID.objects.filter(...).prefetch_related("TimeLogedIn_set")
SubscriptionID.objects.filter(...).select_related("PersonID")
I have a model defined as bellow
class Image(model.Models):
# Stages
STAGE_TRAIN = 'train'
STAGE_VAL = 'val'
STAGE_TEST = 'test'
STAGE_TRASH = 'trash'
STAGE_CHOICES = (
(STAGE_TRAIN, 'Train'),
(STAGE_VAL, 'Validation'),
(STAGE_TEST, 'Test'),
(STAGE_TRASH, 'Trash'),
)
stage = models.CharField(max_length=5, choices=STAGE_CHOICES, default=STAGE_TRAIN)
commit = models.ForeignKey(Commit, on_delete=models.CASCADE, related_name="images", related_query_name="image")
In my database I have 170k images and I try to have an endpoint that will count all the images by stage
Currently I have something like that
base_query = Image.objects.filter(commit=commit_uuid).only('id', 'stage')
count_query = base_query.aggregate(count_train=Count('id', filter=Q(stage='train')),
count_val=Count('id', filter=Q(stage='val')),
count_trash=Count('id', filter=Q(stage='trash')))
but it takes around 40sec and when I try to see the SQL request in my shell I have something that looks ok
{'sql': 'SELECT COUNT("image"."id") FILTER (WHERE "image"."stage" = \'train\') AS "count_train", COUNT("image"."id") FILTER (WHERE "image"."stage" = \'val\') AS "count_val", COUNT("image"."id") FILTER (WHERE "image"."stage" = \'trash\') AS "count_trash" FROM "image" WHERE "image"."commit_id" = \'333681ff-886a-42d0-b88a-5d38f1e9fe94\'::uuid', 'time': '42.140'}
an other strange thing is that if I change my aggregate function with
count_query = base_query.aggregate(count_train=Count('id', filter=Q(stage='train')&Q(commit=commit_uuid)),
count_val=Count('id', filter=Q(stage='val')&Q(commit=commit_uuid)),
count_trash=Count('id', filter=Q(stage='trash')&Q(commit=commit_uuid)))
When I do that the query is twice as fast (still 20sec) and when I display the SQL I see that the filter on the commit is done inside the FILTER
So I have two questions:
Can I do something different to improve the speed of the query or should I store the count somewhere and change the values each time I change an image ?
I was expecting the query to filter first on the commit id and then on the stage but I have the feeling that it's done the otherway around
1) You can add the fields indices either with index_together option
class Image(model.Models):
class Meta:
index_together = [['stage'], ['stage', 'commit']]
or the indexes option (cf https://docs.djangoproject.com/en/2.0/ref/models/options/#django.db.models.Options.indexes)
class Image(model.Models):
class Meta:
indexes = [models.Index(fields=['stage', 'commit'])]
2) You don't need the necessity to look up the id:
base_query = Image.objects.filter(commit=commit_uuid).only('stage')
# count images in stages
count = base_query.aggregate(train=Count(1, filter=Q(commit=commit_uuid) & Q(stage='train')),
val=Count(1, filter=Q(commit=commit_uuid) & Q(stage='val')),
trash=Count(1, filter=Q(commit=commit_uuid) & Q(stage='trash')))
I would try this in your model:
stage = models.CharField(max_length=5, choices=STAGE_CHOICES, default=STAGE_TRAIN, index=True)
By adding an index to stage, you should avoid full table scans.
I have two models: MetaModel and RelatedModel. I want to include the result of a RelatedModel lookup within a MetaModel query, and I'd like to do this within a single DB call.
I've tried to define a 'subquery' QuerySet for use in the main query, but that hasn't worked - it's still making two queries to complete the operation.
Note: I can't use a traditional ForeignKey relationship because the profile_id field is not unique. Uniqueness is a combination of profile_id and channel. This is an aggregation table and profile_id is not guaranteed to be unique across multiple third-party channels.
Any suggestions?
Models:
class Channel(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(
max_length=25,
)
class MetaModel(models.Model):
profile_id = fields.IntegerField()
channel = fields.ForeignKey(Channel))
metadata = fields.TextField()
class RelatedModel(models.Model):
related_id = fields.IntegerField()
profile_id = fields.IntegerField()
channel = fields.ForeignKey(Channel))
Dummy data
channel = Channel("Web site A")
channel.save()
sample_meta = MetaModel(profile_id=1234, channel=channel)
sample_related = RelatedModel(profile_id=1234, related_id=5678, channel=channel)
Query:
# Create a queryset to filter down to the single record we need the `profile_id` for
# I've limited to the only field we need via a `values` operation
related_qs = RelatedAccount.objects.filter(
related_id=5678,
channel=channel
).values_list("profile_id", flat=True)
# I'm doing an update_or_create as there is other data to store, not included for brevity
obj, created = MetaModel.objects.update_or_create(
profile_id=related_qs.first(), # <<< This var is the dynamic part of the query
channel=channel,
defaults={"metadata": "Metadata is added to a new or existing record."}
)
Regarding your note on uniqueness, you can use unique_together option in Django as described here in the documentation.
class MetaModel(models.Model):
profile_id = fields.ForeignKey(RelatedModel)
channel = fields.ForeignKey(Channel)
metadata = fields.TextField()
class Meta:
unique_together = ('profile_id', 'channel')
Then you can change your query accordingly and should solve your problem.
I have the following working code:
houses_of_agency = House.objects.filter(agency_id=90)
area_list = AreaHouse.objects.filter(house__in=houses_of_agency).values('area')
area_ids = Area.objects.filter(area_id__in=area_list).values_list('area_id', flat=True)
That returns a queryset with a list of area_ids. I want to filter further so that I only get area_ids where there are more than 100 houses belonging to the agency.
I tried the following adjustment:
houses_of_agency = House.objects.filter(agency_id=90)
area_list = AreaHouse.objects.filter(house__in=houses_of_agency).annotate(num_houses=Count('house_id')).filter(num_houses__gte=100).values('area')
area_ids = Area.objects.filter(area_id__in=area_list).values_list('area_id', flat=True)
But it returns an empty queryset.
My models (simplified) look like this:
class House(TimeStampedModel):
house_pk = models.IntegerField()
agency = models.ForeignKey(Agency, on_delete=models.CASCADE)
class AreaHouse(TimeStampedModel):
area = models.ForeignKey(Area, on_delete=models.CASCADE)
house = models.ForeignKey(House, on_delete=models.CASCADE)
class Area(TimeStampedModel):
area_id = models.IntegerField(primary_key=True)
parent = models.ForeignKey('self', null=True)
name = models.CharField(null=True, max_length=30)
Edit: I'm using MySQL for the database backend.
You are querying for agency_id with just one underscore. I corrected your queries below. Also, in django it's more common to use pk instead of id however the behaviour is the same. Further, there's no need for three separate queries as you can combine everything into one.
Also note that your fields area_id and house_pk are unnecessary, django automatically creates primary key fields which area accessible via modelname__pk.
# note how i inlined your first query in the .filter() call
area_list = AreaHouse.objects \
.filter(house__agency__pk=90) \
.annotate(num_houses=Count('house')) \ # <- 'house'
.filter(num_houses__gte=100) \
.values('area')
# note the double underscore
area_ids = Area.objects.filter(area__in=area_list)\
.values_list('area__pk', flat=True)
you could simplify this even further if you don't need the intermediate results. here are both queries combined:
area_ids = AreaHouse.objects \
.filter(house__agency__pk=90) \
.annotate(num_houses=Count('house')) \
.filter(num_houses__gte=100) \
.values_list('area__pk', flat=True)
Finally, you seem to be manually defining a many-to-many relation in your model (through AreaHouse). There are better ways of doing this, please read the django docs.
I am using Django framework, appengine database.
My code for model is:
class Group(models.Model):
name = models.CharField(max_length=200)
ispublic = models.BooleanField()
logo = models.CharField(max_length=200)
description = models.CharField(max_length=200)
groupwebsite = models.CharField(max_length=200)
owner = models.ForeignKey('profile')
class Group_members(models.Model):
profile = models.CharField(max_length=200)
group = models.ForeignKey('group')
I am querying on Group_members to remove group. My query is as follows:
groups = Group_members.objects.filter(Q(profile=profile.id),~Q(group__in=group_id)
INFO:
group_id = ['128','52']
group is a foreign key to group model
My problem is when I run this query, it throws Database error: Lookup type 'in' can't be negated.
I have also performed query using __in it worked fine but does not work for foreign key.
Thanks in advance
I think you trying to filter profile id and remove groups in group_id in single filter
groups = Group_members.objects.filter(Q(profile=profile.id),~Q(group__in=group_id)
instead try this:
1)first filter the profiles form group_member :
groups = Group_members.objects.filter(profile=profile.id)
2)remove the groups form Queryset by:
groupId = [x.group.id for x in groups if x.group.id not in group_id]
Hope this will give you perfect result
2 suggestions.
Use ~Q(group__ pk__in=group_id)
Instead of using filter and not in, use exclude and in