Develop Reference

How can i change this query to ORM?

Hi i have two models like this,
class Sample(models.Model):
name = models.CharField(max_length=256) ##
processid = models.IntegerField(default=0) #
class Process(models.Model):
sample = models.ForeignKey(Sample, blank=False, null=True, on_delete=models.SET_NULL, related_name="process_set")
end_at = models.DateTimeField(null=True, blank=True)
and I want to join Sample and Process model. Because Sample is related to process and I want to get process information with sample .
SELECT sample.id, sample.name, process.endstat
FROM sample
INNER JOIN process
ON sample.processid = process.id
AND process.endstat = 1;
(i'm using SQLite)
I used
sample_list = sample_list.filter(process_set__endstat=1))
but it returned
SELECT sample.id, sample.name
FROM sample
INNER JOIN process
ON (sample.id = process.sample_id)
AND process.endstat = 1)
This is NOT what I want.
How can i solve the problem?

This should work for you
Process.objects.filter(end_at=1).values('sample__id','sample__name','end_at')
.values() method returns selective table fields.

I'm assuming sample_list = Sample.objects.
When you are filtering a model, only the fields defined in the model are selected. In your example, id and processid. If you want to retrieve values from related models as a single record you need to use values or values_list. To get the desired query you have to do this
sample_list = sample_list.filter(process_set__endstat=1).values('id', 'name', 'process__endstat')
Btw, Django does JOIN on the foreign key field. So, you can't get ON sample.processid = process.id since processid is not a ForeignKey field.
Reference:
https://docs.djangoproject.com/en/4.0/ref/models/querysets/#values

I found JOIN not on foreign key field in django.
sample_list = sample_list.filter(processid__in=Process.objects.filter(endstat=1)
I used the medthod of
Django-queryset join without foreignkey

Setting field values for M2M object in djnago

I have two models in my application and I am trying to update a M2M field . Django documentation says that it can be done using set() method . But I guess by dealt set() accepts pk of the objects as arguments and what I want to do is set it using the actual values .
For e.g. I have a client in Clients models named "adam" and I want to set the value for its corresponding M2M field "items_onsale" to ["ac" , "book"]
Below is my code for the models :-
from django.db import models
class Client(models.Model):
SEX_CHOICES = [('M', 'Male'), ('F', 'Female')]
fname = models.CharField(max_length=100)
lname = models.CharField(max_length=100)
mailid = models.EmailField(max_length=100)
sex = models.CharField(max_length=1, choices=SEX_CHOICES, blank=True)
age = models.IntegerField()
items_onsale = models.ManyToManyField('Sinfo', blank=True)
def __str__(self): # for displaying the variable correctly.
return self.fname , self.lname , self.mailid , self.sex , self.age , self.items_onsale
class Sinfo(models.Model): # data of items put on sale by clients
iname = models.CharField(max_length=100)
idesc = models.TextField(max_length=300, null=True)
def __str__(self): # for displaying the variable correctly.
return self.iname
What I have tried till now is :-
c = Client.objects.get(pk=17)
list=["ac","book"]
c.items_onsale_set(list)
And I am getting below error :-
ValueError: Field 'id' expected a number but got 'book'
I know that there is a way to update it using values but not sure how . The django documentation does suggest using "through_defaults" option but haven't given any such example , so I am quite not sure how to use it :-
For many-to-many relationships set() accepts a list of either model instances or field values, normally primary keys, as the objs argument.
Use the through_defaults argument to specify values for the new intermediate model instance(s), if needed. You can use callables as values in the through_defaults dictionary and they will be evaluated once before creating any intermediate instance(s).
I am there is a better to achieve this , please let me know .

through_defaults option works if you have multiple foreign-keys in the same model when using the through option.
in your case, the items_onsale have a direct relationship with the client, use the add function instead of set.
c = Client.objects.get(pk=17)
list=["ac","book"]
c.items_onsale.set(list)
to
c = Client.objects.get(pk=17)
list= Sinfo.objects.filter(iname__in=["ac","book"])
c.items_onsale.add(*list)
add() accepts an arbitrary number of arguments, not a list of them.
To expand that list into arguments, use *

How to create multiple objects with different values at a time in django?

I need to create two models from a single template. Creating Product model is fine. The Product model has the ManyToOne relation with ProductVariant. But I got problem while creating ProductVariant model.
request.POST.getlist('names') this gives me the result like this ['name1','name2] and the same goes for all.
I want to create ProductVariant object with each values. How can I do this ? Also I think there is a problem while stroing a HStoreField. request.POST.getlist('attributes') gives the value like this ['a:b','x:z'] so I converted it into dictionary(but not sure it works).
UPDATE:
What I want is
attributes, names ... all will have the same number of items in the list.
For example if the name is ['a','b','c'] then weight will also have 3 values in the list [12,15,23] like this.
I want to create ProductVariant object 3 times since every list will have 3 items in the list. The first object will have field values from the list first item which is name=a,weight=12.. and for the second object values will be name=b, weight=15 like this.
How will it be possible? Or I should change the logic ? Any suggestions ?
models
class ProductVariant(models.Model):
name = models.CharField(max_length=255, blank=False, null=False)
product = models.ForeignKey(Product, on_delete=models.CASCADE)
attributes = HStoreField()
price = models.FloatField(blank=False, null=False, default=0.0)
views
product = product_form.save()
attributes = request.POST.getlist('attributes')
names = request.POST.getlist('name')
up = request.POST.getlist('price')
weight = request.POST.getlist('weight')
print(names, 'names')
# converting attributes into the dictionary for the HStore field
for attribute in attributes:
attributes_dict = {}
key, value = attribute.split(':')
attributes_dict[key] = value
ProductVariant.objects.create(name=name,...) # for each value I want to create this.

Answer for update:
names = ['a', 'b', 'c']
weights = [12, 15, 23]
params = zip(names, weights)
products = [ProductVariant(name=param[0], weight=param[1]) for param in params]
ProductVariant.objects.bulk_create(products)

I disagree with this approach, but if you really want to do it this way, ziping would be the way as #forkcs pointed out.
I would use Django to help me as much as possible, before i get there, please make this change. float != money
class ProductVariant(models.Model):
name = models.CharField(max_length=255, blank=False, null=False)
product = models.ForeignKey(Product, on_delete=models.CASCADE)
attributes = HStoreField()
price = models.DecimalField(blank=False, null=False, default=0, max_digits=6, decimal_places=2)
Once thats done, the form should look like this:
class ProductVariantForm(forms.ModelForm):
class Meta:
fields = ('name', 'product', 'attributes', 'price')
model = ProductVariant
ProductVariantFormSet = formset_factory(ProductVariantForm)
Note that I don't have to parse/clean/format attributes? Thats because Django did it for me ;)
And you can use it as follow IF you raname your fields and not use the same name multiple times: (instead of all your fields being called "attributes", you call them "form-X-attributes" where X is the number 0-infinity, example)
product = product_form.save()
formset = ProductVariantFormSet(data=request.POST)
if formset.is_valid():
instances = []
for form in formset:
if form.is_valid(): # this could probably be removed
instances.append(form.save())
For extra credit you can also do: (it shouldn't really matter)
product = product_form.save()
formset = ProductVariantFormSet(data=request.POST)
if formset.is_valid():
instances = []
for form in formset:
if form.is_valid(): # this could probably be removed
instances.append(form.save(save=False))
ProductVariant.objects.bulk_create(instances)
What do you gain? STANDARDS!!! AND compartmentalization! Everyone that knows Django knows what you did. All your clean logic will be placed in the right place (the form), and you'll be less error prone.
Ps. i wrote tests for you. https://gist.github.com/kingbuzzman/937a9d207bd937d1b2bb22249ae6bdb2#file-formset_example-py-L142
If you want more information on my approach, see the docs https://docs.djangoproject.com/en/3.1/topics/forms/formsets/

As for attributes, it could be reduced to one line like this:
attributes_dict = dict(map(lambda x: x.split(':'), attributes))
To create multiple objects you should either iterate and create one object at a time or use bulk_create:
for name in names:
ProductVariant.objects.create(name=name,...)
Or
ProductVariant.objects.bulk_create([ProductVariant(name=name) for name in names])

Best practice for this is using bulk_create method.
product_variants = [ProductVariant(name=name) for name in names]
ProductVariant.objects.bulk_create(product_variants)

Django aggregation taking a lot of time

I have a model defined as bellow
class Image(model.Models):
# Stages
STAGE_TRAIN = 'train'
STAGE_VAL = 'val'
STAGE_TEST = 'test'
STAGE_TRASH = 'trash'
STAGE_CHOICES = (
(STAGE_TRAIN, 'Train'),
(STAGE_VAL, 'Validation'),
(STAGE_TEST, 'Test'),
(STAGE_TRASH, 'Trash'),
)
stage = models.CharField(max_length=5, choices=STAGE_CHOICES, default=STAGE_TRAIN)
commit = models.ForeignKey(Commit, on_delete=models.CASCADE, related_name="images", related_query_name="image")
In my database I have 170k images and I try to have an endpoint that will count all the images by stage
Currently I have something like that
base_query = Image.objects.filter(commit=commit_uuid).only('id', 'stage')
count_query = base_query.aggregate(count_train=Count('id', filter=Q(stage='train')),
count_val=Count('id', filter=Q(stage='val')),
count_trash=Count('id', filter=Q(stage='trash')))
but it takes around 40sec and when I try to see the SQL request in my shell I have something that looks ok
{'sql': 'SELECT COUNT("image"."id") FILTER (WHERE "image"."stage" = \'train\') AS "count_train", COUNT("image"."id") FILTER (WHERE "image"."stage" = \'val\') AS "count_val", COUNT("image"."id") FILTER (WHERE "image"."stage" = \'trash\') AS "count_trash" FROM "image" WHERE "image"."commit_id" = \'333681ff-886a-42d0-b88a-5d38f1e9fe94\'::uuid', 'time': '42.140'}
an other strange thing is that if I change my aggregate function with
count_query = base_query.aggregate(count_train=Count('id', filter=Q(stage='train')&Q(commit=commit_uuid)),
count_val=Count('id', filter=Q(stage='val')&Q(commit=commit_uuid)),
count_trash=Count('id', filter=Q(stage='trash')&Q(commit=commit_uuid)))
When I do that the query is twice as fast (still 20sec) and when I display the SQL I see that the filter on the commit is done inside the FILTER
So I have two questions:
Can I do something different to improve the speed of the query or should I store the count somewhere and change the values each time I change an image ?
I was expecting the query to filter first on the commit id and then on the stage but I have the feeling that it's done the otherway around

1) You can add the fields indices either with index_together option
class Image(model.Models):
class Meta:
index_together = [['stage'], ['stage', 'commit']]
or the indexes option (cf https://docs.djangoproject.com/en/2.0/ref/models/options/#django.db.models.Options.indexes)
class Image(model.Models):
class Meta:
indexes = [models.Index(fields=['stage', 'commit'])]
2) You don't need the necessity to look up the id:
base_query = Image.objects.filter(commit=commit_uuid).only('stage')
# count images in stages
count = base_query.aggregate(train=Count(1, filter=Q(commit=commit_uuid) & Q(stage='train')),
val=Count(1, filter=Q(commit=commit_uuid) & Q(stage='val')),
trash=Count(1, filter=Q(commit=commit_uuid) & Q(stage='trash')))

I would try this in your model:
stage = models.CharField(max_length=5, choices=STAGE_CHOICES, default=STAGE_TRAIN, index=True)
By adding an index to stage, you should avoid full table scans.

django - prefetch only the newest record?

I am trying to prefetch only the latest record against the parent record.
my models are as such
class LinkTargets(models.Model):
device_circuit_subnet = models.ForeignKey(DeviceCircuitSubnets, verbose_name="Device", on_delete=models.PROTECT)
interface_index = models.CharField(max_length=100, verbose_name='Interface index (SNMP)', blank=True, null=True)
get_bgp = models.BooleanField(default=False, verbose_name="get BGP Data?")
dashboard = models.BooleanField(default=False, verbose_name="Display on monitoring dashboard?")
class LinkData(models.Model):
link_target = models.ForeignKey(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT)
interface_description = models.CharField(max_length=200, verbose_name='Interface Description', blank=True, null=True)
...
The below query fails with the error
AttributeError: 'LinkData' object has no attribute '_iterable_class'
Query:
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.all().order_by('-id')[0]
)
)
I thought about getting LinkData instead and doing a select related but ive no idea how to get only 1 record for each link_target_id
link_data = LinkData.objects.filter(link_target__dashboard=True) \
.select_related('link_target')..?
EDIT:
using rtindru's solution, the pre fetched seems to be empty. there is 6 records in there currently, atest 1 record for each of the 3 LinkTargets
>>> link_data[0]
<LinkTargets: LinkTargets object>
>>> link_data[0].linkdata_set.all()
<QuerySet []>
>>>

The reason is that Prefetch expects a Django Queryset as the queryset parameter and you are giving an instance of an object.
Change your query as follows:
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.filter(pk=LinkData.objects.latest('id').pk)
)
)
This does have the unfortunate effect of undoing the purpose of Prefetch to a large degree.
Update
This prefetches exactly one record globally; not the latest LinkData record per LinkTarget.
To prefetch the max LinkData for each LinkTarget you should start at LinkData: you can achieve this as follows:
LinkData.objects.filter(link_target__dashboard=True).values('link_target').annotate(max_id=Max('id'))
This will return a dictionary of {link_target: 12, max_id: 3223}
You can then use this to return the right set of objects; perhaps filter LinkData based on the values of max_id.
That will look something like this:
latest_link_data_pks = LinkData.objects.filter(link_target__dashboard=True).values('link_target').annotate(max_id=Max('id')).values_list('max_id', flat=True)
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_set',
queryset=LinkData.objects.filter(pk__in=latest_link_data_pks)
)
)

The following works on PostgreSQL. I understand it won't help OP, but it might be useful to somebody else.
from django.db.models import Count, Prefetch
from .models import LinkTargets, LinkData
link_data_qs = LinkData.objects.order_by(
'link_target__id',
'-id',
).distinct(
'link_target__id',
)
qs = LinkTargets.objects.prefetch_related(
Prefetch(
'linkdata_set',
queryset=link_data_qs,
)
).all()

LinkData.objects.all().order_by('-id')[0] is not a queryset, it is an model object, hence your error.
You could try LinkData.objects.all().order_by('-id')[0:1] which is indeed a QuerySet, but it's not going to work. Given how prefetch_related works, the queryset argument must return a queryset that contains all the LinkData records you need (this is then further filtered, and the items in it joined up with the LinkTarget objects). This queryset only contains one item, so that's no good. (And Django will complain "Cannot filter a query once a slice has been taken" and raise an exception, as it should).
Let's back up. Essentially you are asking an aggregation/annotation question - for each LinkTarget, you want to know the most recent LinkData object, or the 'max' of an 'id' column. The easiest way is to just annotate with the id, and then do a separate query to get all the objects.
So, it would look like this (I've checked with a similar model in my project, so it should work, but the code below may have some typos):
linktargets = (LinkTargets.objects
.filter(dashboard=True)
.annotate(most_recent_linkdata_id=Max('linkdata_set__id'))
# Now, if we need them, lets collect and get the actual objects
linkdata_ids = [t.most_recent_linkdata_id for t in linktargets]
linkdata_objects = LinkData.objects.filter(id__in=linkdata_ids)
# And we can decorate the LinkTarget objects as well if we want:
linkdata_d = {l.id: l for l in linkdata_objects}
for t in linktargets:
if t.most_recent_linkdata_id is not None:
t.most_recent_linkdata = linkdata_d[t.most_recent_linkdata_id]
I have deliberately not made this into a prefetch that masks linkdata_set, because the result is that you have objects that lie to you - the linkdata_set attribute is now missing results. Do you really want to be bitten by that somewhere down the line? Best to make a new attribute that has just the thing you want.

Tricky, but it seems to work:
class ForeignKeyAsOneToOneField(models.OneToOneField):
def __init__(self, to, on_delete, to_field=None, **kwargs):
super().__init__(to, on_delete, to_field=to_field, **kwargs)
self._unique = False
class LinkData(models.Model):
# link_target = models.ForeignKey(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT)
link_target = ForeignKeyAsOneToOneField(LinkTargets, verbose_name="Link Target", on_delete=models.PROTECT, related_name='linkdata_helper')
interface_description = models.CharField(max_length=200, verbose_name='Interface Description', blank=True, null=True)
link_data = LinkTargets.objects.filter(dashboard=True) \
.prefetch_related(
Prefetch(
'linkdata_helper',
queryset=LinkData.objects.all().order_by('-id'),
'linkdata'
)
)
# Now you can access linkdata:
link_data[0].linkdata
Ofcourse with this approach you can't use linkdata_helper to get related objects.

This is not a direct answer to you question, but solves the same problem. It is possible annotate newest object with a subquery, which I think is more clear. You also don't have to do stuff like Max("id") to limit the prefetch query.
It makes use of django.db.models.functions.JSONObject (added in Django 3.2) to combine multiple fields:
MainModel.objects.annotate(
last_object=RelatedModel.objects.filter(mainmodel=OuterRef("pk"))
.order_by("-date_created")
.values(
data=JSONObject(
id="id", body="body", date_created="date_created"
)
)[:1]
)