I am trying to optimize a Django project (vers. 1.8.6) in which each page shows 100 companies and their data at once. I noticed that an unnecessary amount of SQL queries (especially with contact.get_order_count) are performed within the index.html snippet below:
index.html:
{% for company in company_list %}
<tr>
<td>{{ company.name }}</td>
<td>{{ company.get_order_count }}</td>
<td>{{ company.get_order_sum|floatformat:2 }}</td>
<td><input type="checkbox" name="select{{company.pk}}" id=""></td>
</tr>
{% for contact in company.contacts.all %}
<tr>
<td> </td>
<td>{{ contact.first_name }} {{ contact.last_name }}</td>
<td>Orders: {{ contact.get_order_count }}</td>
<td></td>
</tr>
{% endfor %}
{% endfor %}
The problem seems to lie in constant SQL queries to other tables using foreign keys. I looked up how to solve this and found out that prefetch_related() seems to be the solution. However, I keep getting a TemplateSyntaxError about being unable the parse the prefetch, no matter what parameter I use. What is the proper prefetch syntax, or is there any other way to optimize this that I missed?
I've included relevant snippets of model.py below in case it's relevant. I got prefetch_related to work in the defined methods, but it doesn't change the performance or query amount.
model.py:
class Company(models.Model):
name = models.CharField(max_length=150)
def get_order_count(self):
return self.orders.count()
def get_order_sum(self):
return self.orders.aggregate(Sum('total'))['total__sum']
class Contact(models.Model):
company = models.ForeignKey(
Company, related_name="contacts", on_delete=models.PROTECT)
first_name = models.CharField(max_length=150)
last_name = models.CharField(max_length=150, blank=True)
def get_order_count(self):
return self.orders.count()
class Order(models.Model):
company = models.ForeignKey(Company, related_name="orders")
contact = models.ForeignKey(Contact, related_name="orders")
total = models.DecimalField(max_digits=18, decimal_places=9)
def __str__(self):
return "%s" % self.order_number
EDIT:
The view is a ListView and defines the company_list as model = Company. I altered the view based on given suggestions:
class IndexView(ListView):
template_name = "mailer/index.html"
model = Company
contacts = Contact.objects.annotate(order_count=Count('orders'))
contact_list = Company.objects.all().prefetch_related(Prefetch('contacts', queryset=contacts))
paginate_by = 100
Calling the get_order_count and get_order_sum methods causes one query every time the method is called. You can avoid this by annotating the queryset.
from django.db.models import Count, Sum
contacts = Contact.objects.annotate(order_count=Count('orders'), order_sum=Sum('orders'))
You then need to use a Prefetch object to tell Django to use your annotated queryset.
contact_list = Company.objects.all().prefetch_related(Prefetch("contacts", queryset=contacts)
Note that you need to add the prefetch_related to your queryset in the view, it is not possible to call it in the template.
Since you are using ListView, you should be overriding the get_queryset method, and calling prefetch_related() there:
class IndexView(ListView):
template_name = "mailer/index.html"
model = Company
paginate_by = 100
def get_queryset(self):
# Annotate the contacts with the order counts and sums
contacts = Contact.objects.annotate(order_count=Count('orders')
queryset = super(IndexView, self).get_queryset()
# Annotate the companies with order_count and order_sum
queryset = queryset.annotate(order_count=Count('orders'), order_sum=Sum('orders'))
# Prefetch the related contacts. Use the annotated queryset from before
queryset = queryset.prefetch_related(Prefetch('contacts', queryset=contacts))
return queryset
Then in your template, you should use {{ contact.order_count }} instead of {{ contact.get_order_count }}, and {{ company.order_count }} instead of {{ company.get_order_count }}.
Try this in views.py
company_list = Company.objects.all().prefetch_related("order", "contacts")
Related
Pretty new to Django and models and I am trying to make a wishlist where people can add to the wishlist an then view what is in their wishlist by clicking on the link to wishlist. I created a separate model for the wishlist that has a foreignfield of the user and then a many to many field for the items they want to add to the wish list. For right now i am trying to view the wishlist that i created for the user using the admin view in Django.
The problem that i am having is that when I try to print their wishlist to the template the below comes up on the page.
<QuerySet [<listings: item: Nimbus, description:this is the nimbus something at a price of 300 and url optional(https://www.google.com/ with a category of )>, <listings: item: broom, description:this is a broom stick at a price of 223 and url optional(www.youtube.com with a category of broom)>, <listings: item: wand, description:this is a wand at a price of 3020 and url optional(www.twitter.com with a category of sales)>]>
What i ideally want is the query set to be split such that the two listings and their information would be on seperate lines on the the html page when i iterate through then the items based on the user. I know it must be from the string representation that i have set up on the model itself but don't know how to manage to accomplish this. I trie .all,.values,.values_list,.prefetch_related, and i still get the same outcome when i go to page and or not iterable
if anything what would be nice is to have access to the the item,description, and price and print that onto the page for every item in the wishlist.
is this possible to do or is my approach wrong and should the wishlist be added to one of the other forms but i think this should work somehow. Don't know if i am close or there is something i am missing in one of my files or need to create a new separate view.
code below:
models.py
class User(AbstractUser):
pass
class listings(models.Model):
item = models.CharField(max_length=100)
description = models.CharField(max_length=100)
price = models.IntegerField()
url = models.CharField(max_length=100,blank=True)
category = models.CharField(max_length=100,blank=True)
def __str__(self):
return f"item: {self.item}, description:{self.description} at a price of {self.price} and url optional({self.url} with a category of {self.category})"
class bids(models.Model):
desired = models.ForeignKey(listings,on_delete=models.CASCADE,related_name="desired",null=True)
bid = models.IntegerField()
def __str__(self):
return f"{self.desired} Bid: {self.bid}"
class comments(models.Model):
information = models.ForeignKey(listings,on_delete=models.CASCADE, related_name ="Review",null=True)
title = models.CharField(max_length=100)
comment = models.CharField(max_length=100)
def __str__(self):
return f"title {self.title} Comment: {self.comment}"
class wl(models.Model):
user = models.ForeignKey(User ,on_delete=models.CASCADE, related_name="users")
product = models.ManyToManyField(listings, related_name="item_wished")
def __str__(self):
return f"{self.product.all()}"
views.py
def watch_list(request):
user = wl.objects.filter(user_id = request.user.id).prefetch_related('product')
return render(request, 'auctions/wishlist.html',{
'wishes': user
})
html template
{% extends "auctions/layout.html" %}
{% block body %}
<h2>Active Listings</h2>
{% for i in wishes.all %}
<li> {{ i }} <li>
{% endfor %}
{% endblock %}
Your code seems fine. Try tweaking your template a bit with structure, maybe an HTML table:
{% extends "auctions/layout.html" %}
{% block body %}
<h2>Active Listings</h2>
<table>
{% for w in wishes %}
<tr>
<td>{{ w.pk }}</td>
<td>{{ w.product.item }}</td>
<td>{{ w.product.description }}</td>
<td>{{ w.product.price }}</td>
</tr>
{% endfor %}
</table>
{% endblock %}
The query would return all wl objects. With:
{% for w in wishes %}
You iterate over all wl objects. Each "w" has "product" field which is a Many to many relation - it contains multiple objects and needs to be iterated again:
{% for each_wl in wishes %}
{% for each_product in each_wl %}
{{ each_product.item }}
Also, name your variables/Classes to something more verbose.
I have a model that has a ManyToManyField of Users:
models.py
class Excuse(models.Model):
students = models.ManyToManyField(User)
reason = models.CharField(max_length=50)
views.py
def excuse_list(request, block_id=None):
queryset = Excuse.objects.all().prefetch_related('students')
context = {
"object_list": queryset,
}
return render(request, "excuses/excuse_list.html", context)
Template (excuse_list.html)
...
{% for excuse in object_list %}
<tr>
<td>{{ excuse.reason }}</td>
<td>
{% for student in excuse.students.all %}
{{student.get_full_name}};
{% endfor %}
</td>
</tr>
{% endfor %}
...
How can I sort the User set (excuse.students.all) alphabetically by a User field, for example: user.last_name?
you can do this just changing your views.py with this:
queryset = Excuse.objects.all().prefetch_related(
Prefetch('students', queryset=User.objects.order_by('lastname')))
remember to import User model!
You can create a method in your Excuse model, that will return a sorted students set:
def get_sorted_students(self):
return self.students.all().order_by('last_name')
And then in your template use excuse.get_sorted_students instead of excuse.students.all
Alternatively, you could use a custom template tag to render the list of students.
I use Django 1.8.4 with Python 3.4
I have a model for tournaments that defines a method which returns a string if a subscription is forbidden.
class Tournament(models.Model):
name = models.CharField(max_length=200, null=True, blank=True)
subscriptions = models.ManyToManyField('ap_users.Profile')
is_subscription_open = models.BooleanField(default=True)
# ...
def why_subscription_impossible(self, request):
if not request.user.profile.is_profile_complete():
return 'Your profile is not complete'
elif not self.is_subscription_open:
return 'Subscriptions are closed'
elif <another_condition>:
return 'Another error message'
return None
I want to display the list of tournaments, using a generic ListView, and I want to use the result of the method to modify the way it is displayed:
<table class="table">
<thead>
<td>Tournament</td>
<td>Subscription</td>
</thead>
{% for tournament in tournament_list %}
<tr>
<td>{{ tournament.name }}</td>
<td>
{% if tournament.why_subscription_impossible %}
{{ tournament.why_subscription_impossible }}
{% else %}
Subscribe
{% endif %}
</td>
</tr>
{% endfor %}
</table>
The view is a class based generic view inherited from generic.ListView.
class IndexView(generic.ListView):
template_name = 'ap_tournament/index.html'
def get_queryset(self):
return Tournament.objects.all()
The shown solution doesn't work, because I need to pass the current request, to get information about logged user. So I tried to add the result of the method to a context in the view
class IndexView(generic.ListView):
template_name = 'ap_tournament/index.html'
def get_queryset(self):
return Tournament.objects.all()
def get_context_data(self, **kwargs):
context = super(IndexView, self).get_context_data(**kwargs)
additional_ctx_info = []
for tournament in self.get_queryset():
additional_ctx_info.append({
'reason_to_not_subscribe': tournament.why_subscription_impossible(self.request)
})
context['subscr_info'] = additional_ctx_info
return context
Obviously, this doesn't work too. I don't know how to access to the subscr_info[n] with n the current index in the tournament_list. I know the forloop.counter0 to get the index, but I can't use it in the template (or I don't know how). I tried :
{{ subscr_info.forloop.counter0.reason_to_not_subscribe }}
{{ subscr_info.{{forloop.counter0}}.reason_to_not_subscribe }}
I also tried to annotate the QuerySet in get_queryset() view method and read about aggregate(), but I feel that works only with operations supported by the database (AVG, COUNT, MAX, etc.).
I also feels that using a filter or a template tag will not work in my case since I need to use the result of the method in a if tag.
Is there a better solution or a completely diffferent method to achieve what I want ?
In your view, you could also do:
tournaments = self.get_queryset()
for tournament in tournaments:
tournament.reason_to_not_subscribe = tournament.why_subscription_impossible(self.request)
Then add tournaments to the context.
You have to create tournament_list in your views.py file, and set it depending on whether the user is logged and has the corresponding permissions.
If you need to count something, you can create the following Counter class :
class Counter:
count = 0
def increment(self):
self.count += 1
return ''
def decrement(self):
self.count -= 1
return ''
You can then use it in your templates by calling {{ counter.increment }} and {{ counter.count }}. ({{ subscr_info.{{counter.count}}.reason_to_not_subscribe }} and don't forget to place {{ counter.increment }} in your loop.
However, a nicer workaround I used was to create a dictionnary containing both the main element and the additional information, i.e.
ctx_info = []
for tournament in self.get_queryset():
ctx_info.append({
'tournament': tournament
'reason_to_not_subscribe': tournament.why_subscription_impossible(self.request)
})
and then loop on ctx_info. It's cleaner, but I however do not know how this can be implemented within a ListView (which I never used)
By the way, your template contains {{ why_subscription_impossible }} instead of {{ tournament.why_subscription_impossible }}, I don't know if it was intended...
I've read in the docs and stackoverflow but i cant figure out why my foreign key following isnt working, django just says that there is no such attribute as vote_story_set.all().
I have a table named story and one named vote_story and the story table has several foreign keys like a foreign to vote_story to get the number of votes for a news story.
According to the docs and the detailed answer here: *_set attributes on Django Models i created an object like this:
all_stories = Story.objects.all()
votes = all_stories.vote_story_set.all()
but this doesnt work since django says that there is no such attribute as "vote_story_set". The database table for story has the 'votes' attribute as a foreign key to the table Votes_story. From the examples ive seen this should be working so i dont understand why it doesnt. There is no foreign keys in the Votes_story table, just a primay key and the attribute 'votes' containing the number of votes.
Update:
Models and template is shown below for Story and Vote_story as well as the relevant view.
Models:
class Story(models.Model):
story_id = models.IntegerField(primary_key=True)
date_added = models.DateTimeField(auto_now_add=True)
date_modified = models.DateTimeField(auto_now=True)
title = models.CharField(max_length=150)
description = models.CharField(blank=True, null=True, max_length=2000)
story_text = models.TextField()
picture = models.ImageField(blank=True, null=True, upload_to="story/images")
storyAccessLevelID = models.ForeignKey(StoryAccessLevel)
categoryID = models.ForeignKey(Category)
votes = models.ForeignKey(Vote_story, blank=True, null=True)
comments = models.ForeignKey(Comment, blank=True, null=True)
def __unicode__(self):
return self.title
class Vote_story(models.Model):
votes = models.IntegerField()
def __unicode__(self):
return self.votes
In the file Vote_story is defined above Story so it can find it.
In Vote_story i let django create the primary key automatically unlike Story.
There is currently one vote for one of the stories added.
Template code (the relevant portion):
<table class="table table-hover">
<thead>
<tr>
<th>Title</th>
<th>Description</th>
<th>Date added</th>
<th>Votes</th>
<th>Comments</th>
</tr>
</thead>
<tbody>
{% for story in all_stories %}
<tr>
<td>{{ story.title }}</td>
<td>{{ story.description }}</td>
<td>{{ story.date_added }}</td>
<td>{{ story.votes }}</td>
<td>{{ story.comments }}</td>
</tr>
{% endfor %}
</tbody>
</table>
The view is like this:
def list_all_stories(request):
""" Show all stories """
all_stories = Story.objects.all()
return render(request, "base/story/all_stories.html", {'all_stories': all_stories})
all_stories is a queryset, not an instance. vote_story_set is an attribute on each instance within the queryset, not the queryset itself.
Plus, you seem to be confused about the direction of relations. If your ForeignKey goes from Vote to VoteStory, you don't need the reverse relation, you need the forward one, which is just whatever you called the field. Again, though, this is an attribute of each instance, not the queryset.
Here is my issue. I am new to python/django (about 2 months in). I have 2 tables, Project and Status. I have a foreign key pointing from status to project, and I am looking to try to display the value of the foreign key (status) on my project template, instead of the address of the foreign key.
Here is my models.py
from django.db import models
from clients.models import Clients
from django.contrib.auth.models import User
from settings import STATUS_CHOICES
from django.db import models
from clients.models import Clients
from django.contrib.auth.models import User
from settings import STATUS_CHOICES
class Project(models.Model):
client = models.ForeignKey(Clients, related_name='projects')
created_by = models.ForeignKey(User, related_name='created_by')
#general information
proj_name = models.CharField(max_length=255, verbose_name='Project Name')
pre_quote = models.CharField(max_length=3,default='10-')
quote = models.IntegerField(max_length=10, verbose_name='Quote #', unique=True)
desc = models.TextField(verbose_name='Description')
starts_on = models.DateField(verbose_name='Start Date')
completed_on = models.DateField(verbose_name='Finished On')
def __unicode__(self):
return u'%s' % (self.proj_name)
class Status(models.Model):
project = models.ForeignKey(Project, related_name='status')
value = models.CharField(max_length=20, choices=STATUS_CHOICES, verbose_name='Status')
date_created= models.DateTimeField(auto_now=True)
def __unicode__(self):
return self.value
class Meta:
verbose_name = ('Status')
verbose_name_plural = ("Status")
My views.py
#login_required
def addProject(request):
if request.method == 'POST':
form = AddSingleProjectForm(request.POST)
if form.is_valid():
project = form.save(commit=False)
project.created_by = request.user
project.save()
project.status.create(
value = form.cleaned_data.get('status', None)
)
return HttpResponseRedirect('/project/')
else:
form = AddSingleProjectForm()
return render_to_response('project/addProject.html', {
'form': form, 'user':request.user}, context_instance=RequestContext(request))
And finally my template:
{% if project_list %}
<table id="plist">
<tr id="plist">
<th>Quote #</th>
<th>Customer</th>
<th>Date</th>
<th>Project Name</th>
<th>Status</th>
<th>Contact</th>
</tr id="plist">
{% for p in project_list %}
<tr id="plist">
<td>{{ p.pre_quote }}{{ p.quote }}</td>
<td>{{ p.client }}</td>
<td>{{ p.starts_on }}</td>
<td>{{ p.proj_name }}</td>
<td>{{ p.status_set.select_related }}</td>
<td>{{ p.created_by }}</td>
</tr>
{% endfor %}
</table>
{% else %}
<p>No projects available.</p>
{% endif %}
Any help would be much appreciated. Thank you!
I'm guessing you mean here:
<td>{{ p.status_set.select_related }}</td>
This doesn't do anything. select_related is an optimisation feature, it has nothing to do with actually getting or displaying the related content. If you want to do that, you will have to iterate through the result of p.status_set.all.
In your model, you have defined the related name for this ForeignKey as "status." Thus, you can now use "status" as this name instead of the "_set" business.
Since this is a ForeignKey (ManyToOne) field, you can't simply display the field as if there were only one value. Instead you need .all, which will return a queryset of all the statuses that point to the object in question. You can then iterate through these.
If instead, you know that each project will have only one status, you can use a OneToOne field instead of a ForeignKey.