I'm using Django to create a Scrum board.
In Scrum, Tasks may or may not be part of an Epic.
Here's a simplified version of my models.py file:
from django.db import models
class Epic(models.Model):
name = models.CharField(max_length=50)
class Task(models.Model):
TO_DO = "TODO"
IN_PROGRESS = "IP"
DONE = "DONE"
TASK_STATUS_CHOICES = [
(TO_DO, 'To Do'),
(IN_PROGRESS, 'In Progress'),
(DONE, 'Done'),
]
name = models.CharField(max_length=50)
status = models.CharField(
max_length = 4,
choices = TASK_STATUS_CHOICES,
)
epic = models.ForeignKey(Epic, on_delete=models.DO_NOTHING, null=True, blank=True)
My views.py page currently filters objects by status:
def board(request, **kwargs):
todos = Task.objects.filter(status='TODO')
ip = Task.objects.filter(status='IP')
done = Task.objects.filter(status='DONE')
context = {
'todos': todos,
'ip': ip,
'done': done,
}
return render(request, 'scrum_board/main.html', context)
My template then displays three columns of tasks by iterating through each QuerySet by using a for loop.
However, I'd like tasks from the same Epic to be displayed together and enveloped within a container (can be a simple HTML table, or a Bootstrap card).
How can I do this? One way that immediately comes to mind is to edit views.py so that I can have QuerySets to execute the following (pseudo-code) for each column (task status):
for epic in epics:
# draw a html table cell
for task in tasks:
# display task
for task in tasks_without_epics:
# display task
I'm just wondering if there are simpler methods? Especially since down the road I'll have to further sort according to task/epic priority, and so on.
How about using .order_by() Queryset method? Yours will issue DB queries for each iteration in the inner for loop because Django lazily evaluates Querysets. If the DB is not that large (I assumed so because it's a scrum board), fetching all data in a query may be more beneficial than fetching with multiple DB queries. There can be the null valued epic value for some tasks, but you can handle it using a query expression with F() and its method desc() or asc(), which have boolean arguments nulls_first and nulls_last.
This is how we query todos ordered by ids of epics in descending order:
from django.db.models import F
todos = Task.objects.filter(status='TODO').order_by(F('epic__pk').desc(nulls_last=True))
Then you can implement your view using loops with only one DB query for each column, putting all tasks of null-valued epic at the tail. Another advantage of this approach is you can easily implement sorting according to other fields by adding sort conditions on .order_by()
Related
I have a program that lets users upload data files and lookup tables (both which are ID'd to a specific company) and map them together. One page lets users choose which company they want to map data for by looking at which companies have both data files and lookup tables, which I use a queryset/model manager for. The problem is if I load a new data file and hierarchy the queryset doesn't pick them up until the server restarts. The queryset returns all the companies that have a data file and hierarchies at the time the server starts, but not anything that's added afterwards. I think this must be because the queryset is defined at startup, but I'm not sure. Is there a way to work around this?
forms.py
class CompanySelectionForm(forms.Form):
companies = RawData.objects.get_companyNames(source="inRDandH")
companiesTuple = makeTuple(companies)
print(companiesTuple)
company = forms.ChoiceField(widget=forms.Select(attrs={'class': 'form-select'}), choices=companiesTuple)
managers.py
class RawDataManager(models.Manager):
def get_queryset(self):
return RawDataQuerySet(self.model, using=self._db)
def get_companyNames(self, source):
return self.get_queryset().get_companyNames(source)
class RawDataQuerySet(models.QuerySet):
def get_companyNames(self, source):
if (source == 'inRDandH'):
distinct_companiesRD = self.filter(fileType=0).values_list('companyName', flat=True).distinct()
distinct_companiesH = self.filter(fileType=1).values_list('companyName', flat=True).distinct()
distinct_companies = set(distinct_companiesRD).intersection(set(distinct_companiesH))
else:
distinct_companies = self.values_list('companyName', flat=True).distinct()
return distinct_companies
The problem is that this code runs only once, when the code is initialised on server start, because it is part of your form class definition:
companies = RawData.objects.get_companyNames(source="inRDandH")
The solution is to make choices a callable, which is run every time the form is instantiated. define that field dynamically, in the __init__ method of the form:
def get_companies_tuple():
companies = RawData.objects.get_companyNames(source="inRDandH")
return makeTuple(companies)
class CompanySelectionForm(forms.Form):
company = forms.ChoiceField(
widget=forms.Select(attrs={'class': 'form-select'}),
choices=get_companies_tuple
)
This will now fetch the data from the database every time the form is initialised, rather than only once during startup.
I have a Django model Task containing permissions.
from django.db import models
from django.contrib.auth.models import Permission
# Other stuff
class TaskType(models.Model):
required_permissions = models.ManyToManyField(Permission)
# Other non relevant fields
class Task(models.Model):
begin = models.DateTimeField( #Options left out
end = models.DateTimeField( #Options left out
type = models.ForeignKey(TaskType, on_delete=models.CASCADE)
# Other non relevant fields
Now I have a view where a user is supposed to select a Task in a specific Time slot but should only choose from tasks where the user has the required permissions for and where the task in question wouldn't be blocked by another task in that time slot the user was already assigned to.
My current solution to this problem is to query objects from the objects from the database that could be a valid candidates and filter out the ones the user has no permission for in python. This obviously feels a bit silly since this should be done by the database query (as database management systems are quite sophisticated for such tasks). My current solution looks like this:
def is_task_valid(self, task):
for p in task.type.permissions.all():
if not user.has_perm(p.content_type.app_label + "." + p.codename):
return False:
# Also checks for all assigned tasks of user in that time slot and returns False
# if they overlap
return True
def get_context_data(self):
# obtains other data
for t in self.get_task_query_set()
if self.is_task_valid(t):
context["tasks"].append(t)
# ...
return context
Is there a better solution to this problem? Filtering for the permissions using the Django QuerySet API troubles me quite a bit.
I have a case when user needs to update one instance together with adding/editing the m2m related objects on this instance.
Here is my solution:
# models.py
class AdditionalAction(SoftDeletionModel):
ADDITIONAL_CHOICES = (
('to_bring', 'To bring'),
('to_prepare', 'To prepare'),
)
title = models.CharField(max_length=50)
type = models.CharField(choices=ADDITIONAL_CHOICES, max_length=30)
class Event(models.Model):
title= models.CharField(max_length=255)
actions = models.ManyToManyField(AdditionalAction, blank=True)
# serializers.py
class MySerializer(serializers.ModelSerializer):
def update(self, instance, validated_data):
actions_data = validated_data.pop('actions')
# Use atomic block to rollback if anything raised Exception
with transaction.atomic():
# update main object
updated_instance = super().update(instance, validated_data)
actions = []
# Loop over m2m relation data and
# create/update each action instance based on id present
for action_data in actions_data:
action_kwargs = {
'data': action_data
}
id = action_data.get('id', False)
if id:
action_kwargs['instance'] = AdditionalAction.objects.get(id=id)
actions_ser = ActionSerializerWrite(**action_kwargs)
actions_ser.is_valid(raise_exception=True)
actions.append(actions_ser.save())
updated_instance.actions.set(actions)
return updated_instance
Can anyone suggest better solution?
P.S. actions can be created or updated in this case, so i can't just use many=True on serializer cause it also needs instance to update.
Using for loop with save here will be a killer if you have a long list or actions triggered on save, etc. I'd try to avoid it.
You may be better off using ORMS update with where clause: https://docs.djangoproject.com/en/2.0/topics/db/queries/#updating-multiple-objects-at-once and even reading the updated objects from the database after the write.
For creating new actions you could use bulk_create:https://docs.djangoproject.com/en/2.0/ref/models/querysets/#bulk-create
There is also this one: https://github.com/aykut/django-bulk-update (disclaimer: I am not a contributor or author of the package).
You have to be aware of cons of this method - if you use any post/pre_ save signals those will not be triggered by the update.
In general, running multiple saves will kill the database, and you might end up with hard to diagnose deadlocks. In one of the projects I worked on moving from save() in the loop into update() decreased response time from 30 something seconds to < 10 where the longest operations left where sending emails.
Writing my first Django app that gets messages from other applications and stores reports about them.
It is performing very slow due to the following logic that I hope can be improved but I'm struggling to find a way to do it with out a loop.
Basically I'm just trying to go through all of the apps (there are about 500 unique ones) and get the latest report for each one. Here are my models and function:
class App(models.Model):
app_name = models.CharField(max_length=200)
host = models.CharField(max_length=50)
class Report(models.Model):
app = models.ForeignKey(App)
date = models.DateTimeField(auto_now_add=True)
status = models.CharField(max_length=20)
runtime = models.DecimalField(max_digits=13, decimal_places=2,blank=True,null=True)
end_time = models.DateTimeField(blank=True,null=True)
def get_latest_report():
""" Returns the latest report from each app """
lset = set()
## get distinct app values
for r in Report.objects.order_by().values_list('app_id').distinct():
## get latest report (by date) and push in to stack.
lreport = Report.objects.filter(app_id=r).latest('date')
lset.add(lreport.pk)
## Filter objects and return the latest runs
return Report.objects.filter(pk__in = lset)
If you're not afraid of executing a query for every app in your database you can try it this way:
def get_latest_report():
""" Returns the latest report from each app """
return [app.report_set.latest('date') for app in App.objects.all()]
This adds a query for every app in your database, but is really expressive and sometimes maintainability and readability are more important than performance.
If you are using PostgreSQL you can use distinct and order_by in combination, giving you the latest report for each app like so
Report.objects.order_by('-date').distinct('app')
If you are using a database that does not support the DISTINCT ON clause, MySQL for example, and you do not mind changing the default ordering of the Report model, you can use prefetch_related to reduce 500+ queries to 2 (however this method will use a lot more memory as it will load every report)
class Report(models.Model):
# Fields
class Meta:
ordering = ['-date']
def get_latest_report():
latest_reports = []
for app in App.objects.all().prefetch_related('report_set'):
try:
latest_reports.append(app.report_set.all()[0])
except IndexError:
pass
return latest_reports
I'm getting multiple objects with prefetched relations from my db:
datei_logs = DateiLog.objects.filter(user=request.user)
.order_by("-pk")
.prefetch_related('transfer_logs')
transfer_logs refers to this:
class TransferLog(models.Model):
datei_log = models.ForeignKey("DateiLog", related_name="transfer_logs")
status = models.CharField(
max_length=1,
choices=LOG_STATUS_CHOICES,
default='Good'
)
server_name = models.CharField(max_length=100, blank=True, default="(no server)")
server = models.ForeignKey('Server')
class Meta:
verbose_name_plural = "Transfer-Logs"
def __unicode__(self):
return self.server_name
Now I want to get all the TransferLogs that have a status of "Good". But I think if I do this:
datei_logs[0].transfer_logs.filter(...)
It queries the db again! Since this happens on a website with many log entries I end up with 900 Queries!
I use:
datei_logs[0].transfer_logs.count()
As well and it causes lots of queries to the db too!
What can I do to "just get everything" and then just query an object that holds all the information instead of the db?
Since you're on Django 1.7 you can use the new Prefetch() objects to specify the queryset you want to use for the related lookup.
queryset = TransferLog.objects.filter(status='Good')
datei_logs = DateiLog.objects.filter(user=request.user)
.order_by("-pk")
.prefetch_related(Prefetch('transfer_logs',
queryset=queryset,
to_attr='good_logs'))
Then you can access datei_logs[0].good_logs and check len(datei_logs[0].good_logs).
If you're interested in multiple statuses, you can just use multiple Prefetch objects. But if you're going to get all the logs anyway, you might as well stick to your original query and then split the logs up in Python, rather than calling filter().