Django Running Python Function to Update Page w/o Refresh - python

I am making a website that tracks population statistics. The site needs to update about every 5 seconds with the latest information.
Here is the relevant code for displaying the pandas df on the page (in a file titled "home.html"):
{% block content %}
<h1>Population Tracker</h1>
{% for index, label,data in pop_df.itertuples %}
<div class = "row__data">
<p>{{label}}: {{data}}</p>
</div>
{% endfor %}
{% endblock content %}
Here is the code for my scraper (in a separate file called "scraper.py")
class Scraper():
def __init__(self):
self.URL = "https://countrymeters.info/en/Japan"
def scrape(self):
"Scrapes the population data once"
page = requests.get(self.URL)
soup = BeautifulSoup(page.content,'html.parser')
data_div = soup.find_all('div',class_="data_div")[0]
table = data_div.findAll("table")[0]
tr = table.findAll('tr')
labels = []
numbers = []
for n, i in enumerate(tr):
number = i.findAll('td',{"class":"counter"})[0].text # numbers
label = i.findAll('td',{"class":"data_name"})[0].text # labels
labels.append(label)
numbers.append(number)
pop_df = pd.DataFrame(
{
'Labels':labels,
'Data': numbers
}
)
return pop_df
In my views.py file, here is what I have done:
from django.shortcuts import render
from bsoup_tracker.scraper import Scraper
scraper = Scraper()
df = scraper.scrape()
def home(request):
context = {
'pop_df':df
}
return render(request,'tracker/home.html',context)
Basically, I would like to be able to call the render onto my home.html page every 5 seconds to reupdate the page, without needing refreshes. I have tried to look elsewhere and see that Ajax could help; however I do not know where to begin.

Instead of using Django to render the page, create API and call every after 5 minutes and after getting the results, refresh the HTML content using JavaScript.
If you need more information please let me know.

AJAX stands for "asynchronous JavaScript and XML" so as you thought that would be the way to go if you need to fetch data from your backend and refresh the interface.
The base component to do so in the XmlHttpRequest object in vanilla JavaScript. However, I strongly advice using a library like jQuery, to me it's really easier to use. With vanilla JS, jQuery or any other library you choose, you can modify DOM to expose data you got from your backend. The major drawback is that you will probably end up with not so clean code which will get harder and harder to maintain.
Nowadays the most common solution would be to use djangorestframework (not mandatory, you can also use django's JsonResponse) to create an API along with a nodeJS framework like React or VueJS to create your interface using API's data. That way you will have a lot more control on your interface.
Finally, if you need to have some sort of live website (pulling data and refreshing interface every 5 seconds seems like a poor design pattern to me), you should use websockets for your frontend and ASGI in backend (instead of WSGI). Django-channel is a nice package to do so, but just Google "django websockets" and you will find a lot of documentation.

Related

Calling a function on HTML page, not found

I am writing a tool to record and monitor downtime on a range of equipment.
I have my file structure as below:
File Structure
Sites is just a subfolder containing individual HTMLS for where the equipment is located.
Currently, flask runs webapp.py which contains:
>from . import app
>#app = (__init__.app)
>from . import views
>from . import ReportingTool
views.py has all of my #app.route's in it, up until the [site].html files. From there, on the [site].html file I ask for input from the user. I haven't started writing code to record the user input in any meaningful way, just want to get the data to a python script and commit them to variables. To this end, in the html file I have
<body>
<div class="menu">
<form method="post" enctype="multipart\form-data" action="{{ url_for('downTime') }}">
<fieldset class="datafieldset">
This then requests different data from the user in the form of multiple field sets as seen here: fieldsets
as you see in the code snippet above I set the action to be url_for('downTime'), downTime is a function in my python file ReportingTool.py. this throws out an error, "werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'downTime'. Did you mean 'supportguide' instead?" traceback
Is there something I need to add or specify on the html document to enable this page (and the other [site].html pages to call functions from the ReportingTool.py file? the #app.route that calls the [site].html file is this and that is called with a redirected from here I've only got it setup like that becuase I wanted the name for the site to appear in the address bar.
Thanks in advance.
I am not sure on steps to fix as I am kind of throwing myself in the deep end to learn basic coding by creating an application for my workplace to replace an excel spreadsheet I created.
You are not reaching the downTime function in the ReportingTool.py file. I suggest trying add_url_rule in your views.py by adding the /reported endpoint referencing the downTime function in ReportingTool.py. Something like this;
app.add_url_rule('/reported', 'ReportingTool.downTime', view_func=ReportingTool.downTime, methods=METHODS)
This answer is based on the responds for this question. You are trying to reach a function in a different file from your main view file. Assuming you are calling the page with the form from a function in the views.py file.
Solved with info from Kakedis' input, and the links they provided.
I added:
app.add_url_rule('/reported', 'ReportingTool.downTime', view_func=ReportingTool.downTime, methods=METHODS)
to webbapp.py, then:
#app.route('/reported')
def downTime():
try:
DTref = request.form['refDT']
except:
DTref = "No Reference"
print(DTref)
print("reported")
return(render_template("/UserRip.html"))
to ReportingTool.py
This now prints the above to console to confirm it's pulling the correct func and brings the user back to the starting page.

Testing the html elements in flask routing templates unittest

I have a login page for a flask app with cloud database, I want to test the results after logging in, specifically, I want to test the HTML elements after logging in. I have seen people test return status code or using assertIn to check if data exist.
Is there a way for me to target a specific HTML tag, like <h1 id="userTitle"> </h1> from rendered templates after POST username, password to the route function login()
def test_users_login(self):
result = self.app.post('/login', data=dict(username='Nicole', password='abc123'), follow_redirects=True)
# I want to check the HTML tag's text value data after logging in
self.assertEqual(result.data.getTag("h1", b"Nicole") #What I imagined using <h1>
self.assertEqual(result.data.getId("user", b"Nicole") #What I imagined using id
#This returns true which is okay, because 'Nicole' exists in the whole page
self.assertIn(b'Nicole', result.data)
In my rendered jinja2 template I have this which is after logging in.
<h1 id="userTitle">{{ session['username'] }},Welcome!</h1>
I guess assertIn works well, but I just want to know how to test an HTML tag without running a browser test.
Although I didn't get a correct answer from here, but I just managed to do the unit-test with just assertIn, by checking the contents of the page.
Thanks everyone

Python + webapp2 + Modify the URL without reloading the page [duplicate]

Is there a way I can modify the URL of the current page without reloading the page?
I would like to access the portion before the # hash if possible.
I only need to change the portion after the domain, so it's not like I'm violating cross-domain policies.
window.location.href = "www.mysite.com/page2.php"; // this reloads
This can now be done in Chrome, Safari, Firefox 4+, and Internet Explorer 10pp4+!
See this question's answer for more information:
Updating address bar with new URL without hash or reloading the page
Example:
function processAjaxData(response, urlPath){
document.getElementById("content").innerHTML = response.html;
document.title = response.pageTitle;
window.history.pushState({"html":response.html,"pageTitle":response.pageTitle},"", urlPath);
}
You can then use window.onpopstate to detect the back/forward button navigation:
window.onpopstate = function(e){
if(e.state){
document.getElementById("content").innerHTML = e.state.html;
document.title = e.state.pageTitle;
}
};
For a more in-depth look at manipulating browser history, see this MDN article.
HTML5 introduced the history.pushState() and history.replaceState() methods, which allow you to add and modify history entries, respectively.
window.history.pushState('page2', 'Title', '/page2.php');
Read more about this from here
You can also use HTML5 replaceState if you want to change the url but don't want to add the entry to the browser history:
if (window.history.replaceState) {
//prevents browser from storing history with each change:
window.history.replaceState(statedata, title, url);
}
This would 'break' the back button functionality. This may be required in some instances such as an image gallery (where you want the back button to return back to the gallery index page instead of moving back through each and every image you viewed) whilst giving each image its own unique url.
Here is my solution (newUrl is your new URL which you want to replace with the current one):
history.pushState({}, null, newUrl);
NOTE: If you are working with an HTML5 browser then you should ignore this answer. This is now possible as can be seen in the other answers.
There is no way to modify the URL in the browser without reloading the page. The URL represents what the last loaded page was. If you change it (document.location) then it will reload the page.
One obvious reason being, you write a site on www.mysite.com that looks like a bank login page. Then you change the browser URL bar to say www.mybank.com. The user will be totally unaware that they are really looking at www.mysite.com.
parent.location.hash = "hello";
In modern browsers and HTML5, there is a method called pushState on window history. That will change the URL and push it to the history without loading the page.
You can use it like this, it will take 3 parameters, 1) state object 2) title and a URL):
window.history.pushState({page: "another"}, "another page", "example.html");
This will change the URL, but not reload the page. Also, it doesn't check if the page exists, so if you do some JavaScript code that is reacting to the URL, you can work with them like this.
Also, there is history.replaceState() which does exactly the same thing, except it will modify the current history instead of creating a new one!
Also you can create a function to check if history.pushState exist, then carry on with the rest like this:
function goTo(page, title, url) {
if ("undefined" !== typeof history.pushState) {
history.pushState({page: page}, title, url);
} else {
window.location.assign(url);
}
}
goTo("another page", "example", 'example.html');
Also, you can change the # for <HTML5 browsers, which won't reload the page. That's the way Angular uses to do SPA according to hashtag...
Changing # is quite easy, doing like:
window.location.hash = "example";
And you can detect it like this:
window.onhashchange = function () {
console.log("#changed", window.location.hash);
}
The HTML5 replaceState is the answer, as already mentioned by Vivart and geo1701. However it is not supported in all browsers/versions.
History.js wraps HTML5 state features and provides additional support for HTML4 browsers.
Before HTML5 we can use:
parent.location.hash = "hello";
and:
window.location.replace("http:www.example.com");
This method will reload your page, but HTML5 introduced the history.pushState(page, caption, replace_url) that should not reload your page.
If what you're trying to do is allow users to bookmark/share pages, and you don't need it to be exactly the right URL, and you're not using hash anchors for anything else, then you can do this in two parts; you use the location. hash discussed above, and then implement a check on the home page, to look for a URL with a hash anchor in it, and redirect you to the subsequent result.
For instance:
User is on www.site.com/section/page/4
User does some action which changes the URL to www.site.com/#/section/page/6 (with the hash). Say you've loaded the correct content for page 6 into the page, so apart from the hash the user is not too disturbed.
User passes this URL on to someone else, or bookmarks it
Someone else, or the same user at a later date, goes to www.site.com/#/section/page/6
Code on www.site.com/ redirects the user to www.site.com/section/page/6, using something like this:
if (window.location.hash.length > 0){
window.location = window.location.hash.substring(1);
}
Hope that makes sense! It's a useful approach for some situations.
Below is the function to change the URL without reloading the page. It is only supported for HTML5.
function ChangeUrl(page, url) {
if (typeof (history.pushState) != "undefined") {
var obj = {Page: page, Url: url};
history.pushState(obj, obj.Page, obj.Url);
} else {
window.location.href = "homePage";
// alert("Browser does not support HTML5.");
}
}
ChangeUrl('Page1', 'homePage');
You can use this beautiful and simple function to do so anywhere on your application.
function changeurl(url, title) {
var new_url = '/' + url;
window.history.pushState('data', title, new_url);
}
You can not only edit the URL but you can update the title along with it.
Any changes of the loction (either window.location or document.location) will cause a request on that new URL, if you’re not just changing the URL fragment. If you change the URL, you change the URL.
Use server-side URL rewrite techniques like Apache’s mod_rewrite if you don’t like the URLs you are currently using.
You can add anchor tags. I use this on my site so that I can track with Google Analytics what people are visiting on the page.
I just add an anchor tag and then the part of the page I want to track:
var trackCode = "/#" + urlencode($("myDiv").text());
window.location.href = "http://www.piano-chords.net" + trackCode;
pageTracker._trackPageview(trackCode);
As pointed out by Thomas Stjernegaard Jeppesen, you could use History.js to modify URL parameters whilst the user navigates through your Ajax links and apps.
Almost an year has passed since that answer, and History.js grew and became more stable and cross-browser. Now it can be used to manage history states in HTML5-compliant as well as in many HTML4-only browsers. In this demo You can see an example of how it works (as well as being able to try its functionalities and limits.
Should you need any help in how to use and implement this library, i suggest you to take a look at the source code of the demo page: you will see it's very easy to do.
Finally, for a comprehensive explanation of what can be the issues about using hashes (and hashbangs), check out this link by Benjamin Lupton.
Use history.pushState() from the HTML 5 History API.
Refer to the HTML5 History API for more details.
Your new url.
let newUrlIS = window.location.origin + '/user/profile/management';
In a sense, calling pushState() is similar to setting window.location = "#foo", in that both will also create and activate another history entry associated with the current document. But pushState() has a few advantages:
history.pushState({}, null, newUrlIS);
You can check out the root: https://developer.mozilla.org/en-US/docs/Web/API/History_API
This code works for me. I used it into my application in ajax.
history.pushState({ foo: 'bar' }, '', '/bank');
Once a page load into an ID using ajax, It does change the browser url automatically without reloading the page.
This is ajax function bellow.
function showData(){
$.ajax({
type: "POST",
url: "Bank.php",
data: {},
success: function(html){
$("#viewpage").html(html).show();
$("#viewpage").css("margin-left","0px");
}
});
}
Example: From any page or controller like "Dashboard", When I click on the bank, it loads bank list using the ajax code without reloading the page. At this time, browser URL will not be changed.
history.pushState({ foo: 'bar' }, '', '/bank');
But when I use this code into the ajax, it change the browser url without reloading the page.
This is the full ajax code here in the bellow.
function showData(){
$.ajax({
type: "POST",
url: "Bank.php",
data: {},
success: function(html){
$("#viewpage").html(html).show();
$("#viewpage").css("margin-left","0px");
history.pushState({ foo: 'bar' }, '', '/bank');
}
});
}
This is all you will need to navigate without reload
// add setting without reload
location.hash = "setting";
// if url change with hash do somthing
window.addEventListener('hashchange', () => {
console.log('url hash changed!');
});
// if url change do somthing (dont detect changes with hash)
//window.addEventListener('locationchange', function(){
// console.log('url changed!');
//})
// remove #setting without reload
history.back();
Simply use, it will not reload the page, but just the URL :
$('#form_name').attr('action', '/shop/index.htm').submit();

Posting dynamic values in Locust

I have an application that I am trying to load test with Locust. If I know the parameters of a post in advance, I can add them to a post and that works fine:
self.client.post("/Login", {"Username":"user", "Password":"a"})
The application uses a bunch of hidden fields that get sent when the page is posted interactively. The content of these fields is dynamic and assigned by the server at runtime to manage sessions etc. e.g.
<input type="hidden" name="$$submitid" value="view:xid1:xid2:xid143:xid358">
Is there a way I can pick these up to add to my post data? I know the names of the hidden inputs.
You write a function to extract this data by using PyQuery. You just need to call it before sending post request. If you want to create a bunch of data you can call it in on_start function store them in an array, then use it in tasks. See the example below:
from locust import HttpLocust, TaskSet, task
from pyquery import PyQuery
class UserBehaviour(TaskSet):
def get_data(self, url, locator):
data = []
request = self.client.get(url)
pq = PyQuery(request.content)
link_elements = pq(locator)
for link in link_elements:
if key in link.attrib and "http" not in link.attrib[key]:
data.append(link.attrib[key])
return data
#task
def test_get_thing(self):
data_ = self.get_data("/url/to/send/request", "#review-ul > li > div > a", "href")
self.client.post("url", data = data_)

How to store/deal with django exception webpages in acceptance testing

I am running a series of selenium functional tests of a django site for acceptance testing purposes. I notice that when I run these and exceptions occur, I get back an entire page ( eg a HTTP status 500 ).
I am running acceptance testing using a simple loop and storing the outputted html to a db using the django orm:
def my_functional_tests(request):
import requests
from mytests.models import Entry
for i in range(3):
p1 = { ....... }
r1 = requests.post('http://127.0.0.1:8000/testfunction1/',data=p1)
..............
entry = Entry(output1 = r1.text, output2 = r2.text, output3 = r3.text)
entry.save()
return HttpResponse("completed")
My Model is defined as (where the outputs are the HTML results of 3 functional tests ):
class Entry(models.Model):
output1 = models.CharField(max_length=240)
output2 = models.CharField(max_length=240)
output3 = models.CharField(max_length=240)
When I get an error, the resulting approximately 65K webpage causes an exception on saving, and breaks the testing. I want to get as much info as possible, so I could increase the max_length to lets say 70,000 to store the entire page, but is there a more concise way to capture and store relevant data including the specific errors to the db ?
If you did this with Django's testing client, you could get more concise information--but by using requests, you're really hitting your page as a web browser would, so the full page is what you get (but 65K for a 500 Error page? Wow).
Could you embed in the error page an HTML comment with a marker and concise explanation?
<html>
<h1>Error</h1>
... 64k of stuff follows ...
<!-- ERR:"info about error" -->
</html>
That way, you could parse the results for that error code and store just that.
Of course, you'll want to make sure you don't put anything confidential in that error message or, if you do, that you emit it only when in DEBUG mode or when the request comes from localhost, or logged in as staff, or whatever other security constraint would work.
Slightly prettier would be to write a piece of middleware that emits the error-info as an HTTP Header; then your page could stay the same and you could look at the response headers for your error info.

Categories

Resources