Develop Reference

How to implement datatable server side from this code using django

This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 20 hours.
Joao Daniel is looking for a canonical answer:
Please I need based on my detailed question one which I can test and implement
Currently I have this code that shows me the normal data table but if I have millions of records it becomes very slow, I understand that serverside of datatable can be used but I don't know how I can implement it
list.html
<table class="table table-striped table-hover responsive" style="width:100%" id="buyer">
<thead>
<th>#id</th>
<th>Fecha</th>
<th>Cod Cliente</th>
<th>Cliente</th>
<th>Dirección</th>
<th>Comentario</th>
<th>Tipo Retiro</th>
<th>Total</th>
<th>Guias a Generar</th>
</thead>
<tbody>
{% for item in ResulOrdenes %}
<tr>
<td>{{item.DOCNUM}}</td>
<td>{{item.DOC_DATE|date:"Y-m-d"}}</td>
<td>{{ item.CARDCODE }}</td>
<td>{{ item.CARDNAME }}</td>
<td>{{ item.ADDRESS }}</td>
<td>{{ item.COMMENTS }}</td>
<td>{{ item.URETIRO }}</td>
<td>{{ item.DOCTOTAL }}</td>
<td>{{ item.NGUIAS }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
{% block content_wrapper %}
<p></p>
{% endblock content_wrapper %}
{% block js_page %}
<script>
$(document).ready(function() {
$('.table').DataTable({
dom: 'Bfrtip',
});
</script>
{% endblock %}
view.py
def Listado_guia(request):
template_name='app_listguia/lista.html'
conn = dbapi.connect(..)
cursorSol2 = conn.cursor()
sql_command = 'select a."DocNum" as "DOCNUM", a."DocDate" as "DOC_DATE", a."CardCode" as "CARDCODE", a."CardName" as "CARDNAME", a."DocTotal" as "DOCTOTAL", CEILING(a."DocTotal"/20000) as "NGUIAS", a."DocStatus" as "DOCSTATUS", a."Address" as "ADDRESS", a."Address2" as "ADDRESS2", a."Comments" as "COMMENTS", a."U_Retiro_mercaderia" as "URETIRO" '
sql_command = sql_command + 'from "'+ connections.databases[ConfigBaseHana]["NAME"] +'"."ODLN" as a '
cursorSol2.execute(sql_command)
RES_ORDENES = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
dnow = today.strftime("%Y-%m-%d")
contexto = { "ResulOrdenes":RES_ORDENES, "dnow":dnow }
return render(request,template_name,contexto)
urls.py
from django.urls import path
from django.contrib.auth import views as auth_views
from app_guia.views import *
urlpatterns = [
path('guiaList',Listado_guia,name='urlListGuia_list'),
]
Can you please give me some light on how to proceed? Thanks

The approach you are using is not suitable for millions of records. First of all, if this is your complete sql, you should stick with normal Django QuerySets and switch your view to use a (generic) class-based view. This gives you a proper paginator, which will reduce the load on the database, django and the resulting html significantly.
If you do not want to have pagination, but rather a continuous list, you need a javascript component to virtually render your table. This way, only a few hundred DOM nodes are present in the clients javascript, and will be populated from server, on the fly, while scrolling. The term to search for is "virtualized table", an examplatory candidate would be react-window.
On a sidenote: if the code shown is your "real" production code (and not a boiled-down demo version for this post), I highly recommend working through a django tutorial, e.g. here or here, and sticking to the default django way as documented in the djangoproject as much as possible. You will save yourself hours of programming and huge problems with future enhancements.

To improve the performance of your data table, you can use the server-side processing mode of the DataTables library.
Add the serverSide option to your DataTable initialization code and
set it to true:
$(document).ready(function() {
$('.table').DataTable({
serverSide: true,
processing: true,
ajax: '/guiaList',
dom: 'Bfrtip',
});
});
In your view function, you need to handle the AJAX request and return
the data in the expected format. You can use the JsonResponse class
from Django to return the data in JSON format
from django.http import JsonResponse
from django.core.paginator import Paginator
def Listado_guia(request):
draw = int(request.GET.get('draw', 1))
start = int(request.GET.get('start', 0))
length = int(request.GET.get('length', 10))
conn = dbapi.connect(...)
cursorSol2 = conn.cursor()
sql_command = 'SELECT ...'
cursorSol2.execute(sql_command)
total_items = cursorSol2.rowcount
results = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
paginator = Paginator(results, length)
page_results = paginator.get_page(int(start/length)+1)
data = []
for item in page_results:
data.append({
'DOCNUM': item[0],
'DOC_DATE': item[1].strftime("%Y-%m-%d"),
'CARDCODE': item[2],
'CARDNAME': item[3],
'DOCTOTAL': item[4],
'NGUIAS': item[5],
'DOCSTATUS': item[6],
'ADDRESS': item[7],
'ADDRESS2': item[8],
'COMMENTS': item[9],
'URETIRO': item[10]
})
response = {
'draw': draw,
'recordsTotal': total_items,
'recordsFiltered': total_items,
'data': data
}
return JsonResponse(response)
The start, length, and draw parameters are sent automatically by DataTables to the server when requesting data. The start parameter indicates the start index of the current page, the length parameter indicates the number of rows to display on the current page, and the draw parameter is used to ensure that the response matches the current state of the table.
The JsonResponse returns the data in the format expected by DataTables, which includes the draw parameter, the total number of records (recordsTotal), the number of filtered records (recordsFiltered), and the data array.

deleting objects without refresh django

I am trying to delete selected objects in django it works but when I select an item then click delete button it does not delete but after it when I refresh the page selected object is deleted. Here is
views.py
#login_required
def delete_contact(request):
if request.is_ajax():
selected_contacts = request.POST['contact_id']
selected_contacts = json.loads(selected_contacts)
for i, contact in enumerate(selected_contacts):
if contact != '':
ClientContact.objects.filter(author_id__in=selected_contacts).delete()
return redirect('contacts')
in templates
<table class="table table-hover contact-list">
<thead>
</thead>
{% for contact in contacts %}
<tbody>
<tr data-id="{{ contact.id }}" class="clickable-row"
data-href="{% url 'contact-detail' contact.id %}"
style="cursor: pointer; ">
<th scope="row"><input type="checkbox" id="check"></th>
<td>{{ contact.client_name }}</td>
<td>{{ contact.client_company_name }}</td>
<td>{{ contact.email }}</td>
<td>{{ contact.work_phone }}</td>
<td>{{ contact.work_phone }}</td>
</tr>
</tbody>
{% endfor %}
</table>
{% csrf_token %}
</div>
</div>
</div>
</div>
{% include 'users/footer.html' %}
<script type="text/javascript">
$(document).ready(function () {
$(".delete-btn").click(function () {
var selected_rows = [];
$('.contact-list').find('tr').each(function () {
var row = $(this);
if (row.find('input[type="checkbox"]').is(':checked')) {
console.log(row.attr('data-id'));
selected_rows.push(row.attr('data-id'));
}
});
var selected_rows = JSON.stringify(selected_rows);
$.ajax({
url: "{% url 'contact-delete' %}",
type: 'POST',
data: {
'contact_id': selected_rows,
'csrfmiddlewaretoken': $("[name=csrfmiddlewaretoken]").val()
},
});
});
});
</script>
it works fine but with refreshing the page. How can I delete selected objects
as soon as I click the delete button.
Any help please?
Thank you!

You are deleting the objects, what you are missing is that after sending the request to Django, if the request is successful you need to update the HTML accordingly.
The HTML of your page is rendered when you request the view. At this point the for loop in your template gets executed and iterates over all existing contacts
{% for contact in contacts %}
{% endfor %}
Afterwards, when the user clicks the delete button a request is sent through AJAX to Django which effectively deletes the selected objects yet the HTML code is not updated automagically.
When you refresh the page, the template code is executed once more by Django and thus the for loop is run again but this time the list of contacts has changed, that's why you see the changes in this case.
You can solve your issue in different ways:
1) Instead of calling a Django view via AJAX, make a proper HTML+Django form that is posted to a Django view that after processing the form redirects to the same view again. This would require no Javascript or AJAX. You can read more about forms here. In this way your template is re-rendered after every post and therefore you will see your table updated.
2) Probably the worst option, but also the easiest to implement at this point, would be to refresh your page via Javascript after the AJAX request returns successfully. For this you can bind a function to the success property of your $.ajax call that triggers a refresh, something like location.reload();. Note that this is not a good choice since you are making all the effort to call the delete view using AJAX but you are not getting any of its benefits, getting only the worst of both worlds.
3) The third option is to edit your HTML (your DOM actually) using javascript when the AJAX call returns successfully. If you choose to follow this path (which is your intention I presume) and do not know how to do it I suggest that you post another question regarding specifically how to change the rendered HTML via Javascript.

django actually call for every related model pre_delete and post_delete signals. So calling some function for every related models was found not inefficient.
And I think, that not calling delete() method leads to destroy the integrity of data.
For example: we have model myModel with FileField. In case, when we call delete() from some of myModel's object, this object and related file will be deleted.
If we call delete of related to myModel's object, this object will be deleted, but file will remain.

save dictionary to database from html template in flask

I will try my best to be as concise as possible.
In my back end, I have a list of dictionaries saved to a variable. Each dictionary represents a post from reddit and includes the score, url, and title.
Respectively, the template will loop through this list and then return the values of each of these keys to the user like so:
<table>
<tr>
{% for x in data[0:5] %}
<td>
{{ x['score'] }}
{{ x['title'] }}
<br>
<a href='/add_to_favorites'> Add To Favorites </a>
</td>
{% endfor %}
</tr>
</table>
As you can see, there's an tag which is linked to a function on my utils.py that is attempting to save the respective dictionary to the database (I have a model that represents the url, title, and score).
I feel as though my template is not representing the dictionary in the correct way, for my link to include the html as when it is pressed I receive a 404 error (though I have this route already defined in views.py - '/add_to_favorites' which calls my 'save_post' function).
def save_post():
data = get_info()
for post in data:
fav= Favorite(title=post.get('title'), url=post.get('url'), score=post.get('score'), user_id=current_user.id)
db.session.add(fav)
db.session.commit()
return redirect(url_for('favorites'))
and:
#app.route('/add_to_favorites')
#login_required
def add_to_favorites():
return save_post()
Am i going about this the wrong way? How can i make sure that the link/button is associated with only the html of the that it is included in?
Just need some guidance into the right direction here, not necessarily the code to fix it. Thank you

Proper way to consume data from RESTFUL API in django

I'm trying to learn django so while I have a current solution I'm not sure if it follows best practices in django. I would like to display information from a web api on my website. Let's say the api url is as follows:
http://api.example.com/books?author=edwards&year=2009
Thsis would return a list of books by Edwards written in the year 2009. Returned in the following format:
{'results':
[
{
'title':'Book 1',
'Author':'Edwards Man',
'Year':2009
},
{
'title':'Book 2',
'Author':'Edwards Man',
'Year':2009}
]
}
Currently I am consuming the API in my views file as follows:
class BooksPage(generic.TemplateView):
def get(self,request):
r = requests.get('http://api.example.com/books?author=edwards&year=2009')
books = r.json()
books_list = {'books':books['results']}
return render(request,'books.html',books_list)
Normally, we grab data from the database in the models.py file, but I am unsure if I should be grabbing this API data in models.py or views.py. If it should be in models.py, can someone provide an example of how to do this? I wrote the above example sepecifically for stackoverflow, so any bugs are purely a result of writing it here.

I like the approach of putting that kind of logic in a separate service layer (services.py); the data you are rendering is quite not a "model" in the Django ORM sense, and it's more than simple "view" logic. A clean encapsulation ensures you can do things like control the interface to the backing service (i.e., make it look like a Python API vs. URL with parameters), add enhancements such as caching, as #sobolevn mentioned, test the API in isolation, etc.
So I'd suggest a simple services.py, that looks something like this:
def get_books(year, author):
url = 'http://api.example.com/books'
params = {'year': year, 'author': author}
r = requests.get(url, params=params)
books = r.json()
books_list = {'books':books['results']}
return books_list
Note how the parameters get passed (using a capability of the requests package).
Then in views.py:
import services
class BooksPage(generic.TemplateView):
def get(self,request):
books_list = services.get_books('2009', 'edwards')
return render(request,'books.html',books_list)
See also:
Separation of business logic and data access in django

Use the serializer instead of .json, as it gives flexibility to return in a number of formats.As while using rest-api , the provided serializer use is preferred.
Also keep the data handling and get data requests in view.py.The forms are used for templating not as the business logic.

Well, there are several things to keep in mind. First of all, in this case your data is not changing so often. So it is a good practice to cache this kind of responces. There are a lot of caching tools around, but redis is a popular option. Alternatevly, you can choose additional NoSQL database just for caching.
Secondly, what is the purpose of displaying this data? Are you expecting your users to interact with books or authors, etc? If it is just an information, there is no need in forms and models. If not, you must provide proper views, forms and models for both books and authors, etc.
And considering the place where you should call an API request, I would say it dependes heavily on the second question. Choices are:
views.py for just displaying data.
forms.py or still views.py for ineractivity.

<tbody>
{% if libros %}
{% for libro in libros %}
<tr>
<td>{{ libro.id }}</td>
<td>{{ libro.titulo }}</td>
<td>{{ libro.autor }}</td>
<td>{{ libro.eiditorial }}</td>
<td>{{ libro.descripcion }}</td>
<td>{{ libro.cantidad }}</td>
<td>{{ libro.Bodega.nombre }}</td>
<td>
{% if libro.imagen %}
<td><img src= "{{ libro.imagen.url }} "align="center" width="50px" ></td>
{% else %}
<h6>no hay imagen de libros</h6>
{% endif %}
</td>
<td><a class ="btn btn-primary "href="/">Editar</a></td>
<td><a class ="btn btn-danger red"href="/">Eliminar</a></td>
</tr>
{% endfor %}
{% else %}
<h1>no hay registros de libros</h1>
{% endif%}
</tbody>
as I can call it in an html when I already have a list of my local application greetings

Django templates and variable attributes

I'm using Google App Engine and Django templates.
I have a table that I want to display the objects look something like:
Object Result:
Items = [item1,item2]
Users = [{name='username',item1=3,item2=4},..]
The Django template is:
<table>
<tr align="center">
<th>user</th>
{% for item in result.items %}
<th>{{item}}</th>
{% endfor %}
</tr>
{% for user in result.users %}
<tr align="center">
<td>{{user.name}}</td>
{% for item in result.items %}
<td>{{ user.item }}</td>
{% endfor %}
</tr>
{% endfor %}
</table>
Now the Django documention states that when it sees a . in variables
It tries several things to get the data, one of which is dictionary lookup which is exactly what I want but doesn't seem to happen...

I found a "nicer"/"better" solution for getting variables inside
Its not the nicest way, but it works.
You install a custom filter into django which gets the key of your dict as a parameter
To make it work in google app-engine you need to add a file to your main directory,
I called mine django_hack.py which contains this little piece of code
from google.appengine.ext import webapp
register = webapp.template.create_template_register()
def hash(h,key):
if key in h:
return h[key]
else:
return None
register.filter(hash)
Now that we have this file, all we need to do is tell the app-engine to use it...
we do that by adding this little line to your main file
webapp.template.register_template_library('django_hack')
and in your template view add this template instead of the usual code
{{ user|hash:item }}
And its should work perfectly =)

I'm assuming that the part the doesn't work is {{ user.item }}.
Django will be trying a dictionary lookup, but using the string "item" and not the value of the item loop variable. Django did the same thing when it resolved {{ user.name }} to the name attribute of the user object, rather than looking for a variable called name.
I think you will need to do some preprocessing of the data in your view before you render it in your template.

Or you can use the default django system which is used to resolve attributes in tempaltes like this :
from django.template import Variable, VariableDoesNotExist
#register.filter
def hash(object, attr):
pseudo_context = { 'object' : object }
try:
value = Variable('object.%s' % attr).resolve(pseudo_context)
except VariableDoesNotExist:
value = None
return value
That just works
in your template :
{{ user|hash:item }}

#Dave Webb (i haven't been rated high enough to comment yet)
The dot lookups can be summarized like this: when the template system encounters a dot in a variable name, it tries the following lookups, in this order:
* Dictionary lookup (e.e., foo["bar"])
* Attribute lookup (e.g., foo.bar)
* Method call (e.g., foo.bar())
* List-index lookup (e.g., foo[bar])
The system uses the first lookup type that works. It’s short-circuit logic.

As a replacement for k,v in user.items on Google App Engine using django templates where user = {'a':1, 'b', 2, 'c', 3}
{% for pair in user.items %}
{% for keyval in pair %} {{ keyval }}{% endfor %}<br>
{% endfor %}
a 1
b 2
c 3
pair = (key, value) for each dictionary item.

shouldn't this:
{{ user.item }}
be this?
{{ item }}
there is no user object in the context within that loop....?