I have a model (products) which is actually containing a number of many-to-many fields.
Ideally I am trying to reduce queries to the number of products themselves -- 25, but considering the depth of foreignkeys and m2ms, I understand this might not be possible. But still I'm shooting for best results.
Here is my view:
def index(request):
products = Product.objects.select_related()
paginator = Paginator(products, 25)
page = request.GET.get('page')
try:
products = paginator.page(page)
except PageNotAnInteger:
# If page is not an integer, deliver first page.
products = paginator.page(1)
except EmptyPage:
# If page is out of range (e.g. 9999), deliver last page of results.
products = paginator.page(paginator.num_pages)
print (len(connection.queries))
return render(request, 'editor-index.html', {
'products': products,
'connection':connection,
'csrf':csrf(request)
})
How can I reduce the queries in this? It would be best to prefetch all data related to the product. Is such a thing possible?
select_related only works for foreign keys. For many to many fields you can use prefetch_related. I think that with prefect related you have to specify the related models you want to fetch, and that you can't call it with no arguments like select_related.
Note that you shouldn't need 25 queries for 25 products. It should be possible to reduce it to one query for the products queryset, one additional query for each prefetch_related argument, and (because you are using a paginator) one for the total number of objects.
Related
I'm a beginner and I have been working on project using Django.
I am wondering if there's a good way to avoid repeating the same code.
Also if there are similar logic in some functions, how can I decide if the logic is organized or not.
for example,
def entry_list(request):
entry_list = Entry.objects.all()
#this part is repeated
page = request.GET.get('page', 1)
paginator = Paginator(entry_list, 10)
try:
entries = paginator.page(page)
except PageNotAnInteger:
entries = paginator.page(1)
except EmptyPage:
entries = paginator.page(paginator.num_pages)
return render(request, 'blog/entry_list.html', {'entries': entries})
The logic to paginate is repeated in some other functions as well.
Where should I put the repeated code and how can I decide if I should organize code?
Using function-based views
You could encapsulate it in another function (for example construct such function in a file named utils.py):
# in app/utils.py
def get_page_entries(entry_list, page, per_page=10):
paginator = Paginator(entry_list, per_page)
try:
return paginator.page(page)
except PageNotAnInteger:
return paginator.page(1)
except EmptyPage:
return paginator.page(paginator.num_pages)
You can then use it like:
# app/views.py
from app.utils import get_page_entries
def entry_list(request):
entry_list = Entry.objects.all()
entries= get_page_entries(entry_list, request.GET.get('page', 1))
return render(request, 'blog/entry_list.html', {'entries': entries})
You can provide an optional third parameter with the number of elements per page. If not provided, it will by default be 10.
Or we can encapsulate the request.GET.get(..) logic as well, like:
# in app/utils.py
def get_page_entries(entry_list, querydict, per_page=10, key='page', default_page=1):
page = querydict.get(key, default_page)
paginator = Paginator(entry_list, per_page)
try:
return paginator.page(page)
except PageNotAnInteger:
return paginator.page(default_page)
except EmptyPage:
return paginator.page(paginator.num_pages)
and thus call it with:
# app/views.py
from app.utils import get_page_entries
def entry_list(request):
return render(request, 'blog/entry_list.html', {
'entries': get_page_entries(Entry.objects.all(), request.GET)
})
Using class-based views
You however do not need to use function-based views. This use case is covered by the ListView class [Django-doc]:
class EntryListView(ListView):
model = Entry
template_name = 'blog/entry_list.html'
context_object_name = 'entries'
paginate_by = 10
and then in the urls.py, use EntryListView.as_view() instead of the function, so:
# app/urls.py
from django.urls import path
from app.views import EntryListView
urlpatterns = [
path('entries', EntryListView.as_view(), name='entry-list'),
]
Note only did we reduce the number of lines of code, this is also a more declarative way of developing: instead of specifying how we want to do something, we specify what we want to do. How it is done is more up to Django's implementation of a ListView. Furthermore by providing the settings as attributes of the class, we can easily develop more tooling that take these parameters into account.
I asked myself the same questions for several things including:
functions I might use many times in my views,
messages I would use many times in my views but also tests (in order to test behavior of my apps),
exceptions that could be used in many models or views,
choices when defining Charfields which value can be only specific values,
context values that could be used in many templates,
etc.
Here how I work for instance:
functions: a utils.py file as Willem said does the trick,
messages: I've got a customMessages.py file used for that, where I define simple messages (e.g.:SUCCESS_M_CREATE_OBJECT=_("You created this object successfully."), then used by calling customMessages.SUCCESS_M_CREATE_OBJECT in my models, tests or views) or more complexe ones with variable fields for instance, defined by a function and called with lambda in my tests,
context: by defining dedicated functions in my context_processors.py returning dictionaries of context vars, registering them in my settings.py and then I'm fine if want to call any in my templates,
etc.
Well, there is always a way to not repeat yourself for anything you do in Python and Django, and you are just right to ask yourself those questions at anytime in your development, because your futur self will thank you for that!
This question already has answers here:
Django lazy QuerySet and pagination
(2 answers)
Closed 7 years ago.
So I was reading about pagination, I have done it quite a few times writing this app but I was wondering how does pagination in django work at sql level.
from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
def listing(request):
contact_list = Contacts.objects.all()
paginator = Paginator(contact_list, 25) # Show 25 contacts per page
page = request.GET.get('page')
try:
contacts = paginator.page(page)
except PageNotAnInteger:
# If page is not an integer, deliver first page.
contacts = paginator.page(1)
except EmptyPage:
# If page is out of range (e.g. 9999), deliver last page of results.
contacts = paginator.page(paginator.num_pages)
return render_to_response('list.html', {"contacts": contacts})
Does Contacts.objects.all() get called evertime I am hitting the view ?
Or does Paginator maintains a state somehow ? What would the sql query look like for paginator = Paginator(contact_list, 25) Are there generators being used behind the scenes and things are lazily evaluated?
Sorry if this is a dumb question, there is a lot of abstraction in Django and I seem to miss all the action behind.
Thanks in advance
I am posting this duplicate link as answer, because title of your question is very readable and there is answer for all your question
Django lazy QuerySet and pagination
I need to make real pagination instead of paginating on all retreived data. The example in Django documentation site, is like;
def listing(request):
contact_list = Contacts.objects.all()
paginator = Paginator(contact_list, 25) # Show 25 contacts per page
page = request.GET.get('page')
try:
contacts = paginator.page(page)
except PageNotAnInteger:
# If page is not an integer, deliver first page.
contacts = paginator.page(1)
except EmptyPage:
# If page is out of range (e.g. 9999), deliver last page of results.
contacts = paginator.page(paginator.num_pages)
return render_to_response('list.html', {"contacts": contacts})
This code is paginating records on all retreived records. But there is a trouble. Trying to retreive all record takes many time if there are so many records. I need a solution to retrieve the records page by page from database.
Is there another solution to do this in Django?
You make a false assumption. Django does not retrieve all objects when paginating: it slices the queryset appropriately, which uses LIMIT and COUNT on the SQL.
A QuerySet is a lazy object. When you assign contact_list = Contacts.objects.all(), Django will NOT hit the database. Until you call the methods such as count(), filter(), first(),..., or contact_list[1:5], Django will really retrieve data from the database. SQL statements that Django generate will correlate to each method and these SQL statments will be sent to the DB.
E.g: contact_list[1:5] generate a SQL statement have LIMIT and OFFSET clauses.
In Paginator class, the QuerySet will passed to the its constructor
paginator = Paginator(contact_list, 25)
When you call paginator.page(page), the statement is called:
return self._get_page(self.object_list[bottom:top], number, self)
Look inside Paginator class (django/core/paginator.py), it fetches only required pages. There is only one problem on big tables: if you want to show total page numbers you must make count(*) on entire table which can took a long time in some databases (i.e. postgresql, mysql with innodb).
BTW, try to use generic views in django, ListView would be fine here.
Here we using get_page() to get page wise data(1 page contain 25 data).I would suggest for this like :
def listing(request):
contact_list = Contacts.objects.all()
paginator = Paginator(contact_list, 25) # Show 25 contacts per page
page = request.GET.get('page')
contacts = paginator.get_page(page)
return render_to_response('list.html', {"contacts": contacts})
I have a data structure that declares relationships like this (pseudocode):
class User:
...
class Rating:
rater = User
post = Post
class Post:
ratings = hasmany(Rating)
page_id = ...
I'm building a website using these models, and I'd I'm lazy, so I pass my template the current logged in User, and a bunch of Posts on the current page. My page needs to know what rating the logged in user gave to each Post, so I use SQLAlchemy's one-instance-per-session feature:
posts = session.query(Post).filter(Post.page_id==current_pageid)
ratings = session.query(Post, Rating)\
.filter(Rating.rater==user.id)\
.filter(Post.page_id==current_pageid)
for post in posts:
post.user_rating = None # default value
for post, rating in ratings:
post.user_rating = rating
Then I pass my template the posts list. Is this ugly awful practice? Can I do it some better way?
What you are doing is good enough, except that your query is lacking a WHERE clause between the Post and Rating:
# ...
.filter(Post.id==Rating.post_id)\
But you can also get the result in one query:
qry = (session.query(Post, Rating).
outerjoin(Rating, and_(Post.id==Rating.post_id, Rating.user_id==user.id)).
filter(Post.page_id==current_pageid)
)
res = qry.all() # you can return *res* already to a view, but to get to your results, do below as well:
for post, rating in res:
post.user_rating = rating
posts = [post for post, rating in res]
return posts
Note that in your case posts is not really a list, but a query, and if you iterate over it second time, you might lose the user_rating attribute. You should be cautious returning session-bound objects like query to a view. It is safer to return lists like in the same code above. To fix your code, just add .all() to the query:
posts = session.query(Post).filter(Post.page_id==current_pageid).all()
Yes, it's bad practice. And it even might (in theory) beat you at some moment, e.g. when you query from the same session without clearing it for some post SQLAlchemy will return you the same cached object with already filled rating for some user unrelated to the current context. In practice it will work find in most cases.
Why not just pass a list of (post, rating) pairs to template? Most modern template engines available for Python can iterate over the list of pairs.
BTW, you can fetch both posts and ratings with single query (rating object will be None for OUTER JOIN when it's missing):
session.query(Post, Rating).select_from(Post)\
.outerjoin(Rating, (Post.id==Rating.post_id) & (Rating.rater==…))\
.filter(Post.page_id==…)
I am trying to build a photo gallery with Django.
It is set up by category.
I have paginated the results of a category by n amount of images per page. I want to also use the paginator on the page that shows just the single image and have a prev/next button for the prev/next image in that category.
My thought was to get the current index for the image itself and have that be the link to the /category/CUR_IMG_ID_PAGINATION_LIST/ as the result of paginating the entire set would yield the same index as the current image index in the paginated results.
For instance if the image i want is image 45 out of 150 images total for a category, then when i paginate the 150 images the 45 will be the actual number of the page I want.
If there's an easier way to do this, let me know. Django 1.1
I think the way you're describing it would work ok because behind the scenes I believe what Django is doing is using an SQL LIMIT to simply let the database do the heavy lifting of sorting out what and how much data to return. Because the database is optimized for doing this type of thing it's probably a reasonable way to perform this.
The key will probably be to keep the query the same and as you've demonstrated you could use the same view to do that. The view could simply have a mode which is a fancy way of changing the pagination page count.
You could end up with urls like this...
# View all "landscape" items in gallery mode starting on page 3
http://www.example.com/gallery/landscape/multi/3
# View the 45th landscape item in singular mode
http://www.example.com/gallery/landscape/single/45
When the template is rendered, the paginator will offer the has_next and has_previous methods letting you know if you can use render a Next/Previous link.
Here's what I'm thinking for the view, or something along these lines (this is totally un-tested and written off the top of my head)...
url(r'gallery/(?P<category>.+)/(?P<mode>.+)/(?P<offset>\d+)$', 'whatever.views.media_gallery'),
def media_gallery(request, category, mode, offset):
"""
Render a media gallery.
category = media item category filter
mode = ( multi | single )
offset = The pagination offset in multi mode or the media ID in single mode
"""
if mode == 'multi':
per_page = 20 # or however many items per page
elif mode == 'single':
per_page = 1
else:
pass # handle this however
# Queryitems
raw_media_items = Media.objects.filter(category=category)
# Setup paginator
paginator = Paginator(raw_media_items, per_page)
try:
# in multi mode offset is the page offset
# in single mode offset is the media ID
page = int(offset)
except:
page = 1
try:
media_items = paginator.page(page)
except (EmptyPage, InvalidPage):
media_items = paginator.page(paginator.num_pages)
if len(paginated_items) == 1:
# Render single view
return render_to_response('gallery/gallery_view.html',
{ 'media_item':media_items[0], 'paginator':paginator },
context_instance=RequestContext(request) )
else:
# Render gallery view
return render_to_response('gallery/gallery_view.html',
{ 'media_items':media_items, 'paginator':paginator },
context_instance=RequestContext(request) )