On search screens, users can sort the results by clicking on a column header. Unfortunately, this doesn't work for all columns. It works fine for regular fields like name and price that are stored on the table itself. It also works for many-to-one fields by joining to the referenced table and using the default sort order for that table.
What doesn't work is most functional fields and related fields. (Related fields are a type of functional field.) When you click on the column, it just ignores you. If you change the field definition to be stored in the database, then you can sort by it, but is that necessary? Is there any way to sort by a functional field without storing its values in the database?
Apparently there has been some discussion of this, and CampToCamp posted a merge proposal with a general solution. There's also some discussion in their blog.
I haven't tried their solution yet, but I did create a specific solution for one field by overriding the _generate_order_by() method. Whenever the user clicks on a column header, _generate_order_by() tries to generate an appropriate ORDER BY clause. I found that you can actually put a SQL subquery in the ORDER BY clause to reproduce the values for a functional field.
As an example, we added a functional field to display the first supplier's name for each product.
def _product_supplier_name(self, cr, uid, ids, name, arg, context=None):
res = {}
for product in self.browse(cr, uid, ids, context):
supplier_name = ""
if len(product.seller_ids) > 0:
supplier_name = product.seller_ids[0].name.name
res[product.id] = supplier_name
return res
In order to sort by that column, we overrode _generate_order_by() with some pretty funky SQL. For any other column, we delegate to the regular code.
def _generate_order_by(self, order_spec, query):
""" Calculate the order by clause to use in SQL based on a set of
model fields. """
if order_spec != 'default_partner_name':
return super(product_product, self)._generate_order_by(order_spec,
query)
return """
ORDER BY
(
select min(supp.name)
from product_supplierinfo supinf
join res_partner supp
on supinf.name = supp.id
where supinf.product_id = product_product.id
),
product_product.default_code
"""
The reason for storing the field is that you delegate sorting to sql, that gives you more performance than any other subsequent sorting, for sure.
Related
Let's say I have following models:
class Invoice(models.Model):
...
class Note(models.Model):
invoice = models.ForeignKey(Invoice, related_name='notes', on_delete=models.CASCADE)
text = models.TextField()
and I want to select Invoices that have some notes. I would write it using annotate/Exists like this:
Invoice.objects.annotate(
has_notes=Exists(Note.objects.filter(invoice_id=OuterRef('pk')))
).filter(has_notes=True)
This works well enough, filters only Invoices with notes. However, this method results in the field being present in the query result, which I don't need and means worse performance (SQL has to execute the subquery 2 times).
I realize I could write this using extra(where=) like this:
Invoice.objects.extra(where=['EXISTS(SELECT 1 FROM note WHERE invoice_id=invoice.id)'])
which would result in the ideal SQL, but in general it is discouraged to use extra / raw SQL.
Is there a better way to do this?
You can remove annotations from the SELECT clause using .values() query set method. The trouble with .values() is that you have to enumerate all names you want to keep instead of names you want to skip, and .values() returns dictionaries instead of model instances.
Django internaly keeps the track of removed annotations in
QuerySet.query.annotation_select_mask. So you can use it to tell Django, which annotations to skip even wihout .values():
class YourQuerySet(QuerySet):
def mask_annotations(self, *names):
if self.query.annotation_select_mask is None:
self.query.set_annotation_mask(set(self.query.annotations.keys()) - set(names))
else:
self.query.set_annotation_mask(self.query.annotation_select_mask - set(names))
return self
Then you can write:
invoices = (Invoice.objects
.annotate(has_notes=Exists(Note.objects.filter(invoice_id=OuterRef('pk'))))
.filter(has_notes=True)
.mask_annotations('has_notes')
)
to skip has_notes from the SELECT clause and still geting filtered invoice instances. The resulting SQL query will be something like:
SELECT invoice.id, invoice.foo FROM invoice
WHERE EXISTS(SELECT note.id, note.bar FROM notes WHERE note.invoice_id = invoice.id) = True
Just note that annotation_select_mask is internal Django API that can change in future versions without a warning.
Ok, I've just noticed in Django 3.0 docs, that they've updated how Exists works and can be used directly in filter:
Invoice.objects.filter(Exists(Note.objects.filter(invoice_id=OuterRef('pk'))))
This will ensure that the subquery will not be added to the SELECT columns, which may result in a better performance.
Changed in Django 3.0:
In previous versions of Django, it was necessary to first annotate and then filter against the annotation. This resulted in the annotated value always being present in the query result, and often resulted in a query that took more time to execute.
Still, if someone knows a better way for Django 1.11, I would appreciate it. We really need to upgrade :(
We can filter for Invoices that have, when we perform a LEFT OUTER JOIN, no NULL as Note, and make the query distinct (to avoid returning the same Invoice twice).
Invoice.objects.filter(notes__isnull=False).distinct()
This is best optimize code if you want to get data from another table which primary key reference stored in another table
Invoice.objects.filter(note__invoice_id=OuterRef('pk'),)
We should be able to clear the annotated field using the below method.
Invoice.objects.annotate(
has_notes=Exists(Note.objects.filter(invoice_id=OuterRef('pk')))
).filter(has_notes=True).query.annotations.clear()
I have a table called user_info. I want to get names of all the users. So the table has a field called name. So in sql I do something like
SELECT distinct(name) from user_info
But I am not able to figure out how to do the same in django. Usually if I already have certain value known, then I can do something like below.
user_info.objects.filter(name='Alex')
And then get the information for that particular user.
But in this case for the given table, I want to get all the name values using django ORM just like I do in sql.
Here is my django model
class user_info(models.Model):
name = models.CharField(max_length=255)
priority = models.CharField(max_length=1)
org = models.CharField(max_length=20)
How can I do this in django?
You can use values_list.
user_info.objects.values_list('name', flat=True).distinct()
Note, in Python classes are usually defined in InitialCaps: your model should be UserInfo.
You can use values_list() as given in Daniel's answer, which will provide you your data in a list containing the values in the field. Or you can also use, values() like this:
user_info.object.values('name')
which will return you a queryset containing a dictionary. values_list() and values() are used to select the columns in a table.
Adding on to the accepted answer, if the field is a foreign key the id values(numerical) are returned in the queryset. Hence if you are expecting other kinds of values defined in the model of which the foreign key is part then you have to modify the query like this:
`Post.objects.values_list('author__username')`
Post is a model class having author as a foreign key field which in turn has its username field:
Here, "author" field was appended with double undersocre followed by the field "name", otherwise primary key of the model will be returned in queryset. I assume this was #Carlo's doubt in accepted answer.
I am looking for way how to add new objects to existing queryset, or how to implement what I want by other way.
contact = watson.filter(contacts, searchline)
This line returns queryset, which I later use to iterate.
Then I want to do this to add more objects, which watson couldn't find
contact_in_iteration = Contact.objects.get(id = fild.f_for)
contact.append(contact_in_iteration)
And sorry for my poor english
Did this
contacts = Contact.objects.filter(crm_id=request.session['crm_id'])
query = Q(contacts,searchline)
contact = watson.filter(query)
and get 'filter() missing 1 required positional argument: 'search_text'' error
You can use | and Q lookups. See the docs.
I'm not sure I've fully understood your initial query, but I think that in your case you would want to do:
query = Q(contacts='Foo', searchline='Bar')
contact = watson.filter(query)
Then later:
contact = watson.filter(query | Q(id=field.f_for))
Strictly speaking it won't append to the queryset, but will return a new queryset. But that's okay, because that's what .filter() does anyway.
You should look at a queryset as a sql query that will be executed later. When constructing a queryset and save the result in a variable, you can later filter it even more, but you can not expand it. If you need a query that has more particular rules (like, you need an OR operation) you should state that when you are constructing the query. One way of doing that is indeed using the Q object.
But it looks like you are confused about what querysets really are and how they are used. First of all:
Contact.objects.get(id = fild.f_for)
will never return a queryset, but an instance, because you use get and thus ask for a single particular record. You need to use filter() if you want to get a quersyet. So if you had an existing queryset say active_contacts and you wanted to filter it down so you only get the contacts that have a first_name of 'John' you would do:
active_contacts = Contact.objects.filter(active=True)
active_contacts_named_John = active_contacts.filter(first_name='John')
Of course you could do this in one line too, but I'm assuming you do the first queryset construction somewhere else in your code.
Second remark:
If in your example watson is a queryset, your user of filter() is unclear. This doesn't really make sense:
contact = watson.filter(contacts, searchline)
As stated earlier, filtering a queryset returns another queryset. So you should use a plurar as your variable name e.g. contacts. Then the correct use of filter would be:
contacts = watson.filter(first_name=searchline)
I'm assuming searchline here is a variable that contains a user inputted search term. So maybe here you should name your variable searchterm or similar. This will return all contacts that are filtered by whatever watson is filtering out already and whose first_name matches searchline exactly. You could also use a more liberate method and filter out results that 'contains' the searching term, like so:
contacts = watson.filter(first_name__contains=searchline)
Hope this helps you get on the right path.
I'm trying to do ranking of a QuerySet efficiently (keeping it a QuerySet so I can keep the filter and order_by functions), but cannot seem to find any other way then to iterate through the QuerySet and tack on a rank. I dont want to add rank to my model if I don't have to.
I know how I can get the values I need through SQL query, but can't seem to translate that into Django:
SET #rank = 0, #prev_val = NULL;
SELECT rank, name, school, points FROM
(SELECT #rank := IF(#prev_val = points, #rank, #rank+1) AS rank, #prev_val := points, points, CONCAT(users.first_name, ' ', users.last_name) as name, school.name as school
FROM accounts_userprofile
JOIN schools_school school ON school_id = school.id
JOIN auth_user users ON user_id = users.id
ORDER BY points DESC) as profile
ORDER BY rank DESC
I found that if I did iterate through the QuerySet and tacked on 'rank' manually and then further filtered the results, my 'rank' would disappear - unless is turned it into a list (which made filtering and sorting a bit of pain). Is there any other way you can think of to add rank to my QuerySet? Is there any way I could do the above query and get a QuerySet with filter and order_by functions still intact? I'm currently using the jQuery DataTables with Django to generate a leaderboard with pagination (which is why I need to preserver filtering and order_by).
Thanks in advance! Sorry if I did not post my question correctly - any help would be much appreciated.
I haven't used it myself, but I'm pretty sure you can do that with the extra() method.
Looking at your raw SQL I don't see anything special in your logic. You are simply enumerating all SQL rows on the join ordered by points with a counter from 1, while collapsing the same point values to a same rank.
My suggestion would be to write a custom manager that uses raw() or extra() method. In your manager you would use python's enumerate on all model instances as a rank previously ordered by points. Of course you would have to keep current max value of points and override what enumerate returns to you if they have the same amount of points. Look here for an example of something similar.
Then you could do something like:
YourQuerySet.objects.with_rankings().all()
Lets say I got 2 models, Document and Person. Document got relationship to Person via "owner" property. Now:
session.query(Document)\
.options(joinedload('owner'))\
.filter(Person.is_deleted!=True)
Will double join table Person. One person table will be selected, and the doubled one will be filtered which is not exactly what I want cuz this way document rows will not be filtered.
What can I do to apply filter on joinloaded table/model ?
You are right, table Person will be used twice in the resulting SQL, but each of them serves different purpose:
one is to filter the the condition: filter(Person.is_deleted != True)
the other is to eager load the relationship: options(joinedload('owner'))
But the reason your query returns wrong results is because your filter condition is not complete. In order to make it produce the right results, you also need to JOIN the two models:
qry = (session.query(Document).
join(Document.owner). # THIS IS IMPORTANT
options(joinedload(Document.owner)).
filter(Person.is_deleted != True)
)
This will return correct rows, even though it will still have 2 references (JOINs) to Person table. The real solution to your query is that using contains_eager instead of joinedload:
qry = (session.query(Document).
join(Document.owner). # THIS IS STILL IMPORTANT
options(contains_eager(Document.owner)).
filter(Person.is_deleted != True)
)