This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 20 hours.
Joao Daniel is looking for a canonical answer:
Please I need based on my detailed question one which I can test and implement
Currently I have this code that shows me the normal data table but if I have millions of records it becomes very slow, I understand that serverside of datatable can be used but I don't know how I can implement it
list.html
<table class="table table-striped table-hover responsive" style="width:100%" id="buyer">
<thead>
<th>#id</th>
<th>Fecha</th>
<th>Cod Cliente</th>
<th>Cliente</th>
<th>Dirección</th>
<th>Comentario</th>
<th>Tipo Retiro</th>
<th>Total</th>
<th>Guias a Generar</th>
</thead>
<tbody>
{% for item in ResulOrdenes %}
<tr>
<td>{{item.DOCNUM}}</td>
<td>{{item.DOC_DATE|date:"Y-m-d"}}</td>
<td>{{ item.CARDCODE }}</td>
<td>{{ item.CARDNAME }}</td>
<td>{{ item.ADDRESS }}</td>
<td>{{ item.COMMENTS }}</td>
<td>{{ item.URETIRO }}</td>
<td>{{ item.DOCTOTAL }}</td>
<td>{{ item.NGUIAS }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
{% block content_wrapper %}
<p></p>
{% endblock content_wrapper %}
{% block js_page %}
<script>
$(document).ready(function() {
$('.table').DataTable({
dom: 'Bfrtip',
});
</script>
{% endblock %}
view.py
def Listado_guia(request):
template_name='app_listguia/lista.html'
conn = dbapi.connect(..)
cursorSol2 = conn.cursor()
sql_command = 'select a."DocNum" as "DOCNUM", a."DocDate" as "DOC_DATE", a."CardCode" as "CARDCODE", a."CardName" as "CARDNAME", a."DocTotal" as "DOCTOTAL", CEILING(a."DocTotal"/20000) as "NGUIAS", a."DocStatus" as "DOCSTATUS", a."Address" as "ADDRESS", a."Address2" as "ADDRESS2", a."Comments" as "COMMENTS", a."U_Retiro_mercaderia" as "URETIRO" '
sql_command = sql_command + 'from "'+ connections.databases[ConfigBaseHana]["NAME"] +'"."ODLN" as a '
cursorSol2.execute(sql_command)
RES_ORDENES = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
dnow = today.strftime("%Y-%m-%d")
contexto = { "ResulOrdenes":RES_ORDENES, "dnow":dnow }
return render(request,template_name,contexto)
urls.py
from django.urls import path
from django.contrib.auth import views as auth_views
from app_guia.views import *
urlpatterns = [
path('guiaList',Listado_guia,name='urlListGuia_list'),
]
Can you please give me some light on how to proceed? Thanks
The approach you are using is not suitable for millions of records. First of all, if this is your complete sql, you should stick with normal Django QuerySets and switch your view to use a (generic) class-based view. This gives you a proper paginator, which will reduce the load on the database, django and the resulting html significantly.
If you do not want to have pagination, but rather a continuous list, you need a javascript component to virtually render your table. This way, only a few hundred DOM nodes are present in the clients javascript, and will be populated from server, on the fly, while scrolling. The term to search for is "virtualized table", an examplatory candidate would be react-window.
On a sidenote: if the code shown is your "real" production code (and not a boiled-down demo version for this post), I highly recommend working through a django tutorial, e.g. here or here, and sticking to the default django way as documented in the djangoproject as much as possible. You will save yourself hours of programming and huge problems with future enhancements.
To improve the performance of your data table, you can use the server-side processing mode of the DataTables library.
Add the serverSide option to your DataTable initialization code and
set it to true:
$(document).ready(function() {
$('.table').DataTable({
serverSide: true,
processing: true,
ajax: '/guiaList',
dom: 'Bfrtip',
});
});
In your view function, you need to handle the AJAX request and return
the data in the expected format. You can use the JsonResponse class
from Django to return the data in JSON format
from django.http import JsonResponse
from django.core.paginator import Paginator
def Listado_guia(request):
draw = int(request.GET.get('draw', 1))
start = int(request.GET.get('start', 0))
length = int(request.GET.get('length', 10))
conn = dbapi.connect(...)
cursorSol2 = conn.cursor()
sql_command = 'SELECT ...'
cursorSol2.execute(sql_command)
total_items = cursorSol2.rowcount
results = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
paginator = Paginator(results, length)
page_results = paginator.get_page(int(start/length)+1)
data = []
for item in page_results:
data.append({
'DOCNUM': item[0],
'DOC_DATE': item[1].strftime("%Y-%m-%d"),
'CARDCODE': item[2],
'CARDNAME': item[3],
'DOCTOTAL': item[4],
'NGUIAS': item[5],
'DOCSTATUS': item[6],
'ADDRESS': item[7],
'ADDRESS2': item[8],
'COMMENTS': item[9],
'URETIRO': item[10]
})
response = {
'draw': draw,
'recordsTotal': total_items,
'recordsFiltered': total_items,
'data': data
}
return JsonResponse(response)
The start, length, and draw parameters are sent automatically by DataTables to the server when requesting data. The start parameter indicates the start index of the current page, the length parameter indicates the number of rows to display on the current page, and the draw parameter is used to ensure that the response matches the current state of the table.
The JsonResponse returns the data in the format expected by DataTables, which includes the draw parameter, the total number of records (recordsTotal), the number of filtered records (recordsFiltered), and the data array.
Related
I am building a Flask application to manage character sheets for an RPG using an SQL database.
Currently, I have the following script that displays the list of a user's characters currently in the database.
#app.route("/<cha_name>")
#login_required
def character_sheet():
characters = db.execute(
"""SELECT
*
FROM
characters
WHERE
user_id = :user_id AND
name = :cha_name
""",
user_id=session["user_id"],
cha_name="???",
)
if not characters:
return render_template("add_char.html")
I would like to include a button that navigates to the character sheet for the specic chosen character. So the page below would detail some of the stats for the character, and then a button would take the user to a populated character sheet on another page.
This is what I have so far for displaying a specific user's characters.
{% extends "main.html" %}
{% block title %}
Characters
{% endblock %}
{% block main %}
<table border = 1>
<thead>
<td>Player</td>
<td>Name</td>
<td>Race</td>
<td>Class</td>
<td>Level</td>
<td>Status</td>
<td>View</td>
</thead>
{% for row in rows %}
<tr>
<td>{{ character["user"] }}</td>
<td>{{ character["name"] }}</td>
<td>{{ character["race"] }}</td>
<td>{{ character["cha_class"] }}</td>
<td>{{ character["level"] }}</td>
<td>{{ character["status"] }}</td>
<td><a href={{ url_for('cha_name') }}>View Sheet</a></td> <!-- HERE -->
</tr>
{% endfor %}
</table>
Add a new Character
Go back to home page
</body>
{% endblock %}
What do I use on the line with <!-- HERE --> to make a link to the character sheet URL?
Whew, there's a lot to unpack here!
Your example is/was invalid, e.g. cha_name= on the end of the db.execute() call, and there are no calls to render_template if a character is found, so it'll never produce a response even with a valid request.
Next, you've told Flask to pass a cha_name parameter to your character_sheet function, but haven't defined a parameter on the function itself.
Finally, in your template you're passing cha_name to the url_for function, which (so far as we can see from the sample code) isn't a route that you've defined, so can't work.
You need to include more information for us to help, such as telling us what error you're seeing. Right now, I imagine the Flask service won't even start due to the syntax error. Once that's fixed, I'd expect to see something like the following:
werkzeug.routing.BuildError: Could not build url for endpoint 'cha_name'. Did you mean 'character_sheet' instead?
I'd suggest you head back to the documentation on URL building and also look at the docs for the route decorator. You'll see on the latter that "Flask itself assumes the name of the view function as endpoint". That means that in order to get Flask to generate a URL for your character_sheet function, you'll need to pass that name to url_for, and then the parameters, like this:
url_for('character_sheet', cha_name=character.name)
If a user were to rename their character, all of the URLs would change, which is a bad user experience — what would happen if they'd bookmarked a particular character, then fixed a typo in the name?
Putting this all together, here's a hopefully-better example:
# app.py
#login_required
#app.route("/<character_id>")
def character_sheet(character_id):
characters = db.execute(
"""SELECT
*
FROM
characters
WHERE
id = :id AND
user_id = :user_id
""",
id=character_id,
user_id=session["user_id"],
)
if not characters:
return abort(404)
return render_template(
"characters/sheet.html",
character=characters[0],
)
<!-- templates/characters/list.html -->
{% extends "main.html" %}
{% block title %}
Characters
{% endblock %}
{% block main %}
<table>
{% for character in characters %}
<tr>
<td>{{ character.name }}</td>
<td><a href={{ url_for('character_sheet', character_id=character.id) }}>View Sheet</a></td>
</tr>
{% endfor %}
</table>
Add a new Character
{% endblock %}
I am trying to delete selected objects in django it works but when I select an item then click delete button it does not delete but after it when I refresh the page selected object is deleted. Here is
views.py
#login_required
def delete_contact(request):
if request.is_ajax():
selected_contacts = request.POST['contact_id']
selected_contacts = json.loads(selected_contacts)
for i, contact in enumerate(selected_contacts):
if contact != '':
ClientContact.objects.filter(author_id__in=selected_contacts).delete()
return redirect('contacts')
in templates
<table class="table table-hover contact-list">
<thead>
</thead>
{% for contact in contacts %}
<tbody>
<tr data-id="{{ contact.id }}" class="clickable-row"
data-href="{% url 'contact-detail' contact.id %}"
style="cursor: pointer; ">
<th scope="row"><input type="checkbox" id="check"></th>
<td>{{ contact.client_name }}</td>
<td>{{ contact.client_company_name }}</td>
<td>{{ contact.email }}</td>
<td>{{ contact.work_phone }}</td>
<td>{{ contact.work_phone }}</td>
</tr>
</tbody>
{% endfor %}
</table>
{% csrf_token %}
</div>
</div>
</div>
</div>
{% include 'users/footer.html' %}
<script type="text/javascript">
$(document).ready(function () {
$(".delete-btn").click(function () {
var selected_rows = [];
$('.contact-list').find('tr').each(function () {
var row = $(this);
if (row.find('input[type="checkbox"]').is(':checked')) {
console.log(row.attr('data-id'));
selected_rows.push(row.attr('data-id'));
}
});
var selected_rows = JSON.stringify(selected_rows);
$.ajax({
url: "{% url 'contact-delete' %}",
type: 'POST',
data: {
'contact_id': selected_rows,
'csrfmiddlewaretoken': $("[name=csrfmiddlewaretoken]").val()
},
});
});
});
</script>
it works fine but with refreshing the page. How can I delete selected objects
as soon as I click the delete button.
Any help please?
Thank you!
You are deleting the objects, what you are missing is that after sending the request to Django, if the request is successful you need to update the HTML accordingly.
The HTML of your page is rendered when you request the view. At this point the for loop in your template gets executed and iterates over all existing contacts
{% for contact in contacts %}
{% endfor %}
Afterwards, when the user clicks the delete button a request is sent through AJAX to Django which effectively deletes the selected objects yet the HTML code is not updated automagically.
When you refresh the page, the template code is executed once more by Django and thus the for loop is run again but this time the list of contacts has changed, that's why you see the changes in this case.
You can solve your issue in different ways:
1) Instead of calling a Django view via AJAX, make a proper HTML+Django form that is posted to a Django view that after processing the form redirects to the same view again. This would require no Javascript or AJAX. You can read more about forms here. In this way your template is re-rendered after every post and therefore you will see your table updated.
2) Probably the worst option, but also the easiest to implement at this point, would be to refresh your page via Javascript after the AJAX request returns successfully. For this you can bind a function to the success property of your $.ajax call that triggers a refresh, something like location.reload();. Note that this is not a good choice since you are making all the effort to call the delete view using AJAX but you are not getting any of its benefits, getting only the worst of both worlds.
3) The third option is to edit your HTML (your DOM actually) using javascript when the AJAX call returns successfully. If you choose to follow this path (which is your intention I presume) and do not know how to do it I suggest that you post another question regarding specifically how to change the rendered HTML via Javascript.
django actually call for every related model pre_delete and post_delete signals. So calling some function for every related models was found not inefficient.
And I think, that not calling delete() method leads to destroy the integrity of data.
For example: we have model myModel with FileField. In case, when we call delete() from some of myModel's object, this object and related file will be deleted.
If we call delete of related to myModel's object, this object will be deleted, but file will remain.
I have a SQL query in a Django view and store the results in a variable. As far I know the result should be stored as a list.
def assignments(request):
cursor = connection.cursor()
cursor.execute("SELECT o.article_id, o.amount, m.id, o.create_date, m.status FROM orders o, producers p, machines ma, matches m WHERE ma.producer_id=1 AND m.machine_id = ma.id AND m.order_id = o.id")
articles = cursor.fetchall()
context = {"article_list": articles}
return render(request, 'assignments.html', context)
Then I want to transfer that data row by row in a table in my template.
{% block body %}
<div class="container">
<table class="table">
<thead>...</thead>
<tbody>
{% for articles in article_list %}
<tr>
<td>{{ articles.article_id }}</td>
<td>{{ articles.amount }}</td>
<td>{{ articles.id}}</td>
<td>{{ articles.create_date }}</td>
<td>{{ articles.status }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
Unfortunately the table is empty and is not showing any results.
The query itself should be fine. I tested the query in my database workbench and it is showing the correct results.
How can I access the data stored in variable articles from my template?
PS: I'm far from being a programmer. So I don't really know any programming concepts/language.
Any help is highly appreciated. Thanks!
You have a list of lists. Each element in articles_list is just a list of values from the database; it does not have any knowledge of attributes like "id" or "amount".
As Arpit says in the comments, you should be using Django models for this rather than raw SQL.
I'm trying to learn django so while I have a current solution I'm not sure if it follows best practices in django. I would like to display information from a web api on my website. Let's say the api url is as follows:
http://api.example.com/books?author=edwards&year=2009
Thsis would return a list of books by Edwards written in the year 2009. Returned in the following format:
{'results':
[
{
'title':'Book 1',
'Author':'Edwards Man',
'Year':2009
},
{
'title':'Book 2',
'Author':'Edwards Man',
'Year':2009}
]
}
Currently I am consuming the API in my views file as follows:
class BooksPage(generic.TemplateView):
def get(self,request):
r = requests.get('http://api.example.com/books?author=edwards&year=2009')
books = r.json()
books_list = {'books':books['results']}
return render(request,'books.html',books_list)
Normally, we grab data from the database in the models.py file, but I am unsure if I should be grabbing this API data in models.py or views.py. If it should be in models.py, can someone provide an example of how to do this? I wrote the above example sepecifically for stackoverflow, so any bugs are purely a result of writing it here.
I like the approach of putting that kind of logic in a separate service layer (services.py); the data you are rendering is quite not a "model" in the Django ORM sense, and it's more than simple "view" logic. A clean encapsulation ensures you can do things like control the interface to the backing service (i.e., make it look like a Python API vs. URL with parameters), add enhancements such as caching, as #sobolevn mentioned, test the API in isolation, etc.
So I'd suggest a simple services.py, that looks something like this:
def get_books(year, author):
url = 'http://api.example.com/books'
params = {'year': year, 'author': author}
r = requests.get(url, params=params)
books = r.json()
books_list = {'books':books['results']}
return books_list
Note how the parameters get passed (using a capability of the requests package).
Then in views.py:
import services
class BooksPage(generic.TemplateView):
def get(self,request):
books_list = services.get_books('2009', 'edwards')
return render(request,'books.html',books_list)
See also:
Separation of business logic and data access in django
Use the serializer instead of .json, as it gives flexibility to return in a number of formats.As while using rest-api , the provided serializer use is preferred.
Also keep the data handling and get data requests in view.py.The forms are used for templating not as the business logic.
Well, there are several things to keep in mind. First of all, in this case your data is not changing so often. So it is a good practice to cache this kind of responces. There are a lot of caching tools around, but redis is a popular option. Alternatevly, you can choose additional NoSQL database just for caching.
Secondly, what is the purpose of displaying this data? Are you expecting your users to interact with books or authors, etc? If it is just an information, there is no need in forms and models. If not, you must provide proper views, forms and models for both books and authors, etc.
And considering the place where you should call an API request, I would say it dependes heavily on the second question. Choices are:
views.py for just displaying data.
forms.py or still views.py for ineractivity.
<tbody>
{% if libros %}
{% for libro in libros %}
<tr>
<td>{{ libro.id }}</td>
<td>{{ libro.titulo }}</td>
<td>{{ libro.autor }}</td>
<td>{{ libro.eiditorial }}</td>
<td>{{ libro.descripcion }}</td>
<td>{{ libro.cantidad }}</td>
<td>{{ libro.Bodega.nombre }}</td>
<td>
{% if libro.imagen %}
<td><img src= "{{ libro.imagen.url }} "align="center" width="50px" ></td>
{% else %}
<h6>no hay imagen de libros</h6>
{% endif %}
</td>
<td><a class ="btn btn-primary "href="/">Editar</a></td>
<td><a class ="btn btn-danger red"href="/">Eliminar</a></td>
</tr>
{% endfor %}
{% else %}
<h1>no hay registros de libros</h1>
{% endif%}
</tbody>
as I can call it in an html when I already have a list of my local application greetings
I using nginx as server.
While loading the simple html page without any queries using Django , It takes only 34 milliseconds to load.
<html><body><h1> Test</h1></body></html>
If run some queries ( having 500 rows in query) it take 4 seconds to load the page.
views.py
def testing3(request):
context = User.objects.all()
return render_to_response('pages/index1.html', {'users': context,})
index1.html
<html>
<head> <title>Testing </title> </head>
<body>
Testing
<h1>Testing</h1>
{% for e in users %}
<table>
<tr> <td class="active">{{ e.username }}</td>
<td>{{ e.email }}</td> </tr>
</table>
{% endfor %}
</body>
</html>
How to reduce the loading time.
How to reduce this waiting time?
First : optimize your db queries.
If you only want to display a couple db fields (user.username and user.email in your example) and don't need any of the model's method, use a ValuesQuerySet instead:
def myview(request):
users = User.objects.values("username", "email")
return render_to_response('pages/index1.html', {'users': users,})
OTHO if you need related objects, use the "select_related" keyword arg in your query to avoid the "n+1 queries" problem.
Once this is done, if your query returns hundreds or more records, you want to paginate the queryset. This is specially efficient with querysets as it uses the SQL offset and limit clauses so pagination is in fact done at the db level.
At this point you should have a significant boost already at least at the ORM/db level. If the rendering itself (template level) still takes too much time, it will be time to consider using the cache.