django reduce query load time - python

I using nginx as server.
While loading the simple html page without any queries using Django , It takes only 34 milliseconds to load.
<html><body><h1> Test</h1></body></html>
If run some queries ( having 500 rows in query) it take 4 seconds to load the page.
views.py
def testing3(request):
context = User.objects.all()
return render_to_response('pages/index1.html', {'users': context,})
index1.html
<html>
<head> <title>Testing </title> </head>
<body>
Testing
<h1>Testing</h1>
{% for e in users %}
<table>
<tr> <td class="active">{{ e.username }}</td>
<td>{{ e.email }}</td> </tr>
</table>
{% endfor %}
</body>
</html>
How to reduce the loading time.
How to reduce this waiting time?

First : optimize your db queries.
If you only want to display a couple db fields (user.username and user.email in your example) and don't need any of the model's method, use a ValuesQuerySet instead:
def myview(request):
users = User.objects.values("username", "email")
return render_to_response('pages/index1.html', {'users': users,})
OTHO if you need related objects, use the "select_related" keyword arg in your query to avoid the "n+1 queries" problem.
Once this is done, if your query returns hundreds or more records, you want to paginate the queryset. This is specially efficient with querysets as it uses the SQL offset and limit clauses so pagination is in fact done at the db level.
At this point you should have a significant boost already at least at the ORM/db level. If the rendering itself (template level) still takes too much time, it will be time to consider using the cache.

Related

How to implement datatable server side from this code using django

This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 20 hours.
Joao Daniel is looking for a canonical answer:
Please I need based on my detailed question one which I can test and implement
Currently I have this code that shows me the normal data table but if I have millions of records it becomes very slow, I understand that serverside of datatable can be used but I don't know how I can implement it
list.html
<table class="table table-striped table-hover responsive" style="width:100%" id="buyer">
<thead>
<th>#id</th>
<th>Fecha</th>
<th>Cod Cliente</th>
<th>Cliente</th>
<th>Dirección</th>
<th>Comentario</th>
<th>Tipo Retiro</th>
<th>Total</th>
<th>Guias a Generar</th>
</thead>
<tbody>
{% for item in ResulOrdenes %}
<tr>
<td>{{item.DOCNUM}}</td>
<td>{{item.DOC_DATE|date:"Y-m-d"}}</td>
<td>{{ item.CARDCODE }}</td>
<td>{{ item.CARDNAME }}</td>
<td>{{ item.ADDRESS }}</td>
<td>{{ item.COMMENTS }}</td>
<td>{{ item.URETIRO }}</td>
<td>{{ item.DOCTOTAL }}</td>
<td>{{ item.NGUIAS }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
{% block content_wrapper %}
<p></p>
{% endblock content_wrapper %}
{% block js_page %}
<script>
$(document).ready(function() {
$('.table').DataTable({
dom: 'Bfrtip',
});
</script>
{% endblock %}
view.py
def Listado_guia(request):
template_name='app_listguia/lista.html'
conn = dbapi.connect(..)
cursorSol2 = conn.cursor()
sql_command = 'select a."DocNum" as "DOCNUM", a."DocDate" as "DOC_DATE", a."CardCode" as "CARDCODE", a."CardName" as "CARDNAME", a."DocTotal" as "DOCTOTAL", CEILING(a."DocTotal"/20000) as "NGUIAS", a."DocStatus" as "DOCSTATUS", a."Address" as "ADDRESS", a."Address2" as "ADDRESS2", a."Comments" as "COMMENTS", a."U_Retiro_mercaderia" as "URETIRO" '
sql_command = sql_command + 'from "'+ connections.databases[ConfigBaseHana]["NAME"] +'"."ODLN" as a '
cursorSol2.execute(sql_command)
RES_ORDENES = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
dnow = today.strftime("%Y-%m-%d")
contexto = { "ResulOrdenes":RES_ORDENES, "dnow":dnow }
return render(request,template_name,contexto)
urls.py
from django.urls import path
from django.contrib.auth import views as auth_views
from app_guia.views import *
urlpatterns = [
path('guiaList',Listado_guia,name='urlListGuia_list'),
]
Can you please give me some light on how to proceed? Thanks
The approach you are using is not suitable for millions of records. First of all, if this is your complete sql, you should stick with normal Django QuerySets and switch your view to use a (generic) class-based view. This gives you a proper paginator, which will reduce the load on the database, django and the resulting html significantly.
If you do not want to have pagination, but rather a continuous list, you need a javascript component to virtually render your table. This way, only a few hundred DOM nodes are present in the clients javascript, and will be populated from server, on the fly, while scrolling. The term to search for is "virtualized table", an examplatory candidate would be react-window.
On a sidenote: if the code shown is your "real" production code (and not a boiled-down demo version for this post), I highly recommend working through a django tutorial, e.g. here or here, and sticking to the default django way as documented in the djangoproject as much as possible. You will save yourself hours of programming and huge problems with future enhancements.
To improve the performance of your data table, you can use the server-side processing mode of the DataTables library.
Add the serverSide option to your DataTable initialization code and
set it to true:
$(document).ready(function() {
$('.table').DataTable({
serverSide: true,
processing: true,
ajax: '/guiaList',
dom: 'Bfrtip',
});
});
In your view function, you need to handle the AJAX request and return
the data in the expected format. You can use the JsonResponse class
from Django to return the data in JSON format
from django.http import JsonResponse
from django.core.paginator import Paginator
def Listado_guia(request):
draw = int(request.GET.get('draw', 1))
start = int(request.GET.get('start', 0))
length = int(request.GET.get('length', 10))
conn = dbapi.connect(...)
cursorSol2 = conn.cursor()
sql_command = 'SELECT ...'
cursorSol2.execute(sql_command)
total_items = cursorSol2.rowcount
results = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
paginator = Paginator(results, length)
page_results = paginator.get_page(int(start/length)+1)
data = []
for item in page_results:
data.append({
'DOCNUM': item[0],
'DOC_DATE': item[1].strftime("%Y-%m-%d"),
'CARDCODE': item[2],
'CARDNAME': item[3],
'DOCTOTAL': item[4],
'NGUIAS': item[5],
'DOCSTATUS': item[6],
'ADDRESS': item[7],
'ADDRESS2': item[8],
'COMMENTS': item[9],
'URETIRO': item[10]
})
response = {
'draw': draw,
'recordsTotal': total_items,
'recordsFiltered': total_items,
'data': data
}
return JsonResponse(response)
The start, length, and draw parameters are sent automatically by DataTables to the server when requesting data. The start parameter indicates the start index of the current page, the length parameter indicates the number of rows to display on the current page, and the draw parameter is used to ensure that the response matches the current state of the table.
The JsonResponse returns the data in the format expected by DataTables, which includes the draw parameter, the total number of records (recordsTotal), the number of filtered records (recordsFiltered), and the data array.

deleting objects without refresh django

I am trying to delete selected objects in django it works but when I select an item then click delete button it does not delete but after it when I refresh the page selected object is deleted. Here is
views.py
#login_required
def delete_contact(request):
if request.is_ajax():
selected_contacts = request.POST['contact_id']
selected_contacts = json.loads(selected_contacts)
for i, contact in enumerate(selected_contacts):
if contact != '':
ClientContact.objects.filter(author_id__in=selected_contacts).delete()
return redirect('contacts')
in templates
<table class="table table-hover contact-list">
<thead>
</thead>
{% for contact in contacts %}
<tbody>
<tr data-id="{{ contact.id }}" class="clickable-row"
data-href="{% url 'contact-detail' contact.id %}"
style="cursor: pointer; ">
<th scope="row"><input type="checkbox" id="check"></th>
<td>{{ contact.client_name }}</td>
<td>{{ contact.client_company_name }}</td>
<td>{{ contact.email }}</td>
<td>{{ contact.work_phone }}</td>
<td>{{ contact.work_phone }}</td>
</tr>
</tbody>
{% endfor %}
</table>
{% csrf_token %}
</div>
</div>
</div>
</div>
{% include 'users/footer.html' %}
<script type="text/javascript">
$(document).ready(function () {
$(".delete-btn").click(function () {
var selected_rows = [];
$('.contact-list').find('tr').each(function () {
var row = $(this);
if (row.find('input[type="checkbox"]').is(':checked')) {
console.log(row.attr('data-id'));
selected_rows.push(row.attr('data-id'));
}
});
var selected_rows = JSON.stringify(selected_rows);
$.ajax({
url: "{% url 'contact-delete' %}",
type: 'POST',
data: {
'contact_id': selected_rows,
'csrfmiddlewaretoken': $("[name=csrfmiddlewaretoken]").val()
},
});
});
});
</script>
it works fine but with refreshing the page. How can I delete selected objects
as soon as I click the delete button.
Any help please?
Thank you!
You are deleting the objects, what you are missing is that after sending the request to Django, if the request is successful you need to update the HTML accordingly.
The HTML of your page is rendered when you request the view. At this point the for loop in your template gets executed and iterates over all existing contacts
{% for contact in contacts %}
{% endfor %}
Afterwards, when the user clicks the delete button a request is sent through AJAX to Django which effectively deletes the selected objects yet the HTML code is not updated automagically.
When you refresh the page, the template code is executed once more by Django and thus the for loop is run again but this time the list of contacts has changed, that's why you see the changes in this case.
You can solve your issue in different ways:
1) Instead of calling a Django view via AJAX, make a proper HTML+Django form that is posted to a Django view that after processing the form redirects to the same view again. This would require no Javascript or AJAX. You can read more about forms here. In this way your template is re-rendered after every post and therefore you will see your table updated.
2) Probably the worst option, but also the easiest to implement at this point, would be to refresh your page via Javascript after the AJAX request returns successfully. For this you can bind a function to the success property of your $.ajax call that triggers a refresh, something like location.reload();. Note that this is not a good choice since you are making all the effort to call the delete view using AJAX but you are not getting any of its benefits, getting only the worst of both worlds.
3) The third option is to edit your HTML (your DOM actually) using javascript when the AJAX call returns successfully. If you choose to follow this path (which is your intention I presume) and do not know how to do it I suggest that you post another question regarding specifically how to change the rendered HTML via Javascript.
django actually call for every related model pre_delete and post_delete signals. So calling some function for every related models was found not inefficient.
And I think, that not calling delete() method leads to destroy the integrity of data.
For example: we have model myModel with FileField. In case, when we call delete() from some of myModel's object, this object and related file will be deleted.
If we call delete of related to myModel's object, this object will be deleted, but file will remain.

Accessing model data in Django template

I am using the below lines to pass data to my template index.html, model1 being the class in my models.py.
data = model1.objects.all()
return TemplateResponse(request, 'index.html', {'data': data})
I am able to access data on the front end by using a for loop as shown below
{% for x in data %}
<h3>{{x.name}}</h3>
<h4>{{x.department}}</h4>
{% endfor %}
Since there are mutliple objects in this data, my question is if I want to access only the department of particular object with certain name, how can I do that?
For example here I am using a for loop, consider there are two objects in the data. Then the output would be
name1
department1
name2
department2
So now if I need to access only name2 without any loop, how can i do that?
Updating the question: I am updating this question with html, so that the question looks clear.
table id="example" class="table table-striped" cellspacing="1" width="100%">
<thead>
<tr>
<th>ID</th>
<th>Name</th>
<th>Department</th>
<th>View/Edit</th>
</tr>
</thead>
<tbody>
{% for x in data %}
<tr>
<td>{{x.id}}</td>
<td>{{x.name}}</td>
<td>{{x.department}}</td>
<td>View</td>
<button type="button" class="btn-sm btn-primary btn-lg" data-toggle="modal" data-target="#myModal">
view</button></td>
</tr>
{% endfor %}
</tbody>
This is how my html looks, I am using data-table here. So all the data is captured in a table automatically. As you can see every row has a view button that I would implement. Once the user clicks the view button, I should pop up a modal dialog to show all the other details like {{x.dateJoined}} etc which I don't show in the table. But if I use a separate view to pop this dialog, I should send a request to the view from my template saying which row(with some ID) the user has clicked. How can i achieve that? How can I bind the view button with respective rows here?
You need to write custom template tag which will take the queryset and filtering parameters and returns you appropriate object, you can use simple_tag:
myapp/templatetags/myapp_tags.py
from django import template
register = template.Library()
#register.simple_tag
def get_model1_object(queryset, **filters):
if not filters:
raise template.TemplateSyntaxError('`get_model1_object` tag requires filters.')
retrun queryset.filter(**filters).first()
Then in template:
{% load get_model1_object from myapp_tags %}
{% get_model1_object data name='blah' as obj %}
Note: Your filtering criteria might yield multiple results but in get_model1_object i am only returning the first object assuming your criteria will be strict, change it according to your needs.
The first thing to understand is that template rendering happens on the server, before the user sees anything in their browser. The template is not sent to the user, only the HTML that the template generated. When the user is interacting with the page, the original template is not involved. So, you have two choices here:
You can render the data for all the objects in hidden div's on your page, then use javascript with something like a jquery dialog to display them on demand. This is only realistic if you have very few records.
You can create a second view with its own template, which renders just the HTML for the modal dialog contents. You could then, again using javascript/jquery, make an AJAX request to load the contents of the dialog that you need when you need it. In your first view template, the list of departments, include the url of the object you want to fetch, eg:
{% for x in data %}
<tr>
<td>{{x.name}}</td>
<td><a class="deptlink" href="{% url 'dept_detail' x.pk %}">
{{ x.department }}</a></td>
</tr>
{% endfor %}
Where dept_detail is the name of the urls.py entry for your view that supplies the contents of the dialog.
Then in your javascript, hook the a tag so it opens your dialog instead of leaving the page:
$('.deptlink').click(function (event) {
event.preventDefault();
# then the code after this is up to you, but something that'll
# load the url into your dialog and open it.
$('yourdialog').load($(event.target).attr('href'));
Since you say you are an intermediate in javascript I won't get into the details of implementation, but basically, the moral of the story is that the output of django is going to be an HTML page. Anything that you need to have on the client side either has to be in that page, or you will have to make another request to the server to get it... Either way, you'll need some javascript to do this, since you want a modal dialog with client side interaction.

Queryset in view or in template

I am trying to speed up my code. In development, everything ran very smooth, but once I put it in production, and started adding more depth of data into the database, I realize that it is running very slow.
I noticed on django-toolbar that it is running THOUSANDS of queries, where it should only be maybe 10-20. I am wondering if it may be because of the way I have a lot of content being delivered.
For example, I have code that looks like this:
{% if user.profile.is_admin %}
...
{% endif %}
and
{% for stuff in user.profile.get_somestuff %}
...
{{ stuff.info }}
{{ stuff.other_info }}
...
{% endfor %}
Does each one of these execute a new query?
Should I run the query for get_somestuff in the view, pass it through context? I am asking from a performance perspective.
If profile.get_somestuff is an expensive operation and you call it multiple times in the template, and yes, you should call that in the view once and pass the result to the template via context.
def view(request):
...
stuff = request.user.profile.get_somestuff()
return render(request, 'page.html', {'stuff': stuff})
Alternatively, you can use {% with %} tag to create a scope containing the value in its own context:
{% with stuff=user.profile.get_somestuff %}
...
{{ stuff.info }}
{{ stuff.other_info }}
... do some other things with stuff
{% endwith %}
Personally, I would go with the first option because, with the help of django.db.connection.queries, it is relatively easier to monitor db queries you make in the view. Make sure you avoid sending template querysets, lazy expressions etc. as much as possible.
BTW, please note that DEBUG must be set to True for connection.queries to work.
If stuff.info or stuff.other_info are foreign keys to other models then yes, each time you hit each of those for a new stuff obj you could be doing another select query for each one.
select_related might help you here. It'll effectively join the relevant tables on the fk fields you specify in the sql query upfront. The sql query will be more complex than the queries you're running now but far less numerous.

Proper way to consume data from RESTFUL API in django

I'm trying to learn django so while I have a current solution I'm not sure if it follows best practices in django. I would like to display information from a web api on my website. Let's say the api url is as follows:
http://api.example.com/books?author=edwards&year=2009
Thsis would return a list of books by Edwards written in the year 2009. Returned in the following format:
{'results':
[
{
'title':'Book 1',
'Author':'Edwards Man',
'Year':2009
},
{
'title':'Book 2',
'Author':'Edwards Man',
'Year':2009}
]
}
Currently I am consuming the API in my views file as follows:
class BooksPage(generic.TemplateView):
def get(self,request):
r = requests.get('http://api.example.com/books?author=edwards&year=2009')
books = r.json()
books_list = {'books':books['results']}
return render(request,'books.html',books_list)
Normally, we grab data from the database in the models.py file, but I am unsure if I should be grabbing this API data in models.py or views.py. If it should be in models.py, can someone provide an example of how to do this? I wrote the above example sepecifically for stackoverflow, so any bugs are purely a result of writing it here.
I like the approach of putting that kind of logic in a separate service layer (services.py); the data you are rendering is quite not a "model" in the Django ORM sense, and it's more than simple "view" logic. A clean encapsulation ensures you can do things like control the interface to the backing service (i.e., make it look like a Python API vs. URL with parameters), add enhancements such as caching, as #sobolevn mentioned, test the API in isolation, etc.
So I'd suggest a simple services.py, that looks something like this:
def get_books(year, author):
url = 'http://api.example.com/books'
params = {'year': year, 'author': author}
r = requests.get(url, params=params)
books = r.json()
books_list = {'books':books['results']}
return books_list
Note how the parameters get passed (using a capability of the requests package).
Then in views.py:
import services
class BooksPage(generic.TemplateView):
def get(self,request):
books_list = services.get_books('2009', 'edwards')
return render(request,'books.html',books_list)
See also:
Separation of business logic and data access in django
Use the serializer instead of .json, as it gives flexibility to return in a number of formats.As while using rest-api , the provided serializer use is preferred.
Also keep the data handling and get data requests in view.py.The forms are used for templating not as the business logic.
Well, there are several things to keep in mind. First of all, in this case your data is not changing so often. So it is a good practice to cache this kind of responces. There are a lot of caching tools around, but redis is a popular option. Alternatevly, you can choose additional NoSQL database just for caching.
Secondly, what is the purpose of displaying this data? Are you expecting your users to interact with books or authors, etc? If it is just an information, there is no need in forms and models. If not, you must provide proper views, forms and models for both books and authors, etc.
And considering the place where you should call an API request, I would say it dependes heavily on the second question. Choices are:
views.py for just displaying data.
forms.py or still views.py for ineractivity.
<tbody>
{% if libros %}
{% for libro in libros %}
<tr>
<td>{{ libro.id }}</td>
<td>{{ libro.titulo }}</td>
<td>{{ libro.autor }}</td>
<td>{{ libro.eiditorial }}</td>
<td>{{ libro.descripcion }}</td>
<td>{{ libro.cantidad }}</td>
<td>{{ libro.Bodega.nombre }}</td>
<td>
{% if libro.imagen %}
<td><img src= "{{ libro.imagen.url }} "align="center" width="50px" ></td>
{% else %}
<h6>no hay imagen de libros</h6>
{% endif %}
</td>
<td><a class ="btn btn-primary "href="/">Editar</a></td>
<td><a class ="btn btn-danger red"href="/">Eliminar</a></td>
</tr>
{% endfor %}
{% else %}
<h1>no hay registros de libros</h1>
{% endif%}
</tbody>
as I can call it in an html when I already have a list of my local application greetings

Categories

Resources