I am running a series of selenium functional tests of a django site for acceptance testing purposes. I notice that when I run these and exceptions occur, I get back an entire page ( eg a HTTP status 500 ).
I am running acceptance testing using a simple loop and storing the outputted html to a db using the django orm:
def my_functional_tests(request):
import requests
from mytests.models import Entry
for i in range(3):
p1 = { ....... }
r1 = requests.post('http://127.0.0.1:8000/testfunction1/',data=p1)
..............
entry = Entry(output1 = r1.text, output2 = r2.text, output3 = r3.text)
entry.save()
return HttpResponse("completed")
My Model is defined as (where the outputs are the HTML results of 3 functional tests ):
class Entry(models.Model):
output1 = models.CharField(max_length=240)
output2 = models.CharField(max_length=240)
output3 = models.CharField(max_length=240)
When I get an error, the resulting approximately 65K webpage causes an exception on saving, and breaks the testing. I want to get as much info as possible, so I could increase the max_length to lets say 70,000 to store the entire page, but is there a more concise way to capture and store relevant data including the specific errors to the db ?
If you did this with Django's testing client, you could get more concise information--but by using requests, you're really hitting your page as a web browser would, so the full page is what you get (but 65K for a 500 Error page? Wow).
Could you embed in the error page an HTML comment with a marker and concise explanation?
<html>
<h1>Error</h1>
... 64k of stuff follows ...
<!-- ERR:"info about error" -->
</html>
That way, you could parse the results for that error code and store just that.
Of course, you'll want to make sure you don't put anything confidential in that error message or, if you do, that you emit it only when in DEBUG mode or when the request comes from localhost, or logged in as staff, or whatever other security constraint would work.
Slightly prettier would be to write a piece of middleware that emits the error-info as an HTTP Header; then your page could stay the same and you could look at the response headers for your error info.
Related
I am writing a tool to record and monitor downtime on a range of equipment.
I have my file structure as below:
File Structure
Sites is just a subfolder containing individual HTMLS for where the equipment is located.
Currently, flask runs webapp.py which contains:
>from . import app
>#app = (__init__.app)
>from . import views
>from . import ReportingTool
views.py has all of my #app.route's in it, up until the [site].html files. From there, on the [site].html file I ask for input from the user. I haven't started writing code to record the user input in any meaningful way, just want to get the data to a python script and commit them to variables. To this end, in the html file I have
<body>
<div class="menu">
<form method="post" enctype="multipart\form-data" action="{{ url_for('downTime') }}">
<fieldset class="datafieldset">
This then requests different data from the user in the form of multiple field sets as seen here: fieldsets
as you see in the code snippet above I set the action to be url_for('downTime'), downTime is a function in my python file ReportingTool.py. this throws out an error, "werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'downTime'. Did you mean 'supportguide' instead?" traceback
Is there something I need to add or specify on the html document to enable this page (and the other [site].html pages to call functions from the ReportingTool.py file? the #app.route that calls the [site].html file is this and that is called with a redirected from here I've only got it setup like that becuase I wanted the name for the site to appear in the address bar.
Thanks in advance.
I am not sure on steps to fix as I am kind of throwing myself in the deep end to learn basic coding by creating an application for my workplace to replace an excel spreadsheet I created.
You are not reaching the downTime function in the ReportingTool.py file. I suggest trying add_url_rule in your views.py by adding the /reported endpoint referencing the downTime function in ReportingTool.py. Something like this;
app.add_url_rule('/reported', 'ReportingTool.downTime', view_func=ReportingTool.downTime, methods=METHODS)
This answer is based on the responds for this question. You are trying to reach a function in a different file from your main view file. Assuming you are calling the page with the form from a function in the views.py file.
Solved with info from Kakedis' input, and the links they provided.
I added:
app.add_url_rule('/reported', 'ReportingTool.downTime', view_func=ReportingTool.downTime, methods=METHODS)
to webbapp.py, then:
#app.route('/reported')
def downTime():
try:
DTref = request.form['refDT']
except:
DTref = "No Reference"
print(DTref)
print("reported")
return(render_template("/UserRip.html"))
to ReportingTool.py
This now prints the above to console to confirm it's pulling the correct func and brings the user back to the starting page.
I am making a website that tracks population statistics. The site needs to update about every 5 seconds with the latest information.
Here is the relevant code for displaying the pandas df on the page (in a file titled "home.html"):
{% block content %}
<h1>Population Tracker</h1>
{% for index, label,data in pop_df.itertuples %}
<div class = "row__data">
<p>{{label}}: {{data}}</p>
</div>
{% endfor %}
{% endblock content %}
Here is the code for my scraper (in a separate file called "scraper.py")
class Scraper():
def __init__(self):
self.URL = "https://countrymeters.info/en/Japan"
def scrape(self):
"Scrapes the population data once"
page = requests.get(self.URL)
soup = BeautifulSoup(page.content,'html.parser')
data_div = soup.find_all('div',class_="data_div")[0]
table = data_div.findAll("table")[0]
tr = table.findAll('tr')
labels = []
numbers = []
for n, i in enumerate(tr):
number = i.findAll('td',{"class":"counter"})[0].text # numbers
label = i.findAll('td',{"class":"data_name"})[0].text # labels
labels.append(label)
numbers.append(number)
pop_df = pd.DataFrame(
{
'Labels':labels,
'Data': numbers
}
)
return pop_df
In my views.py file, here is what I have done:
from django.shortcuts import render
from bsoup_tracker.scraper import Scraper
scraper = Scraper()
df = scraper.scrape()
def home(request):
context = {
'pop_df':df
}
return render(request,'tracker/home.html',context)
Basically, I would like to be able to call the render onto my home.html page every 5 seconds to reupdate the page, without needing refreshes. I have tried to look elsewhere and see that Ajax could help; however I do not know where to begin.
Instead of using Django to render the page, create API and call every after 5 minutes and after getting the results, refresh the HTML content using JavaScript.
If you need more information please let me know.
AJAX stands for "asynchronous JavaScript and XML" so as you thought that would be the way to go if you need to fetch data from your backend and refresh the interface.
The base component to do so in the XmlHttpRequest object in vanilla JavaScript. However, I strongly advice using a library like jQuery, to me it's really easier to use. With vanilla JS, jQuery or any other library you choose, you can modify DOM to expose data you got from your backend. The major drawback is that you will probably end up with not so clean code which will get harder and harder to maintain.
Nowadays the most common solution would be to use djangorestframework (not mandatory, you can also use django's JsonResponse) to create an API along with a nodeJS framework like React or VueJS to create your interface using API's data. That way you will have a lot more control on your interface.
Finally, if you need to have some sort of live website (pulling data and refreshing interface every 5 seconds seems like a poor design pattern to me), you should use websockets for your frontend and ASGI in backend (instead of WSGI). Django-channel is a nice package to do so, but just Google "django websockets" and you will find a lot of documentation.
I have a login page for a flask app with cloud database, I want to test the results after logging in, specifically, I want to test the HTML elements after logging in. I have seen people test return status code or using assertIn to check if data exist.
Is there a way for me to target a specific HTML tag, like <h1 id="userTitle"> </h1> from rendered templates after POST username, password to the route function login()
def test_users_login(self):
result = self.app.post('/login', data=dict(username='Nicole', password='abc123'), follow_redirects=True)
# I want to check the HTML tag's text value data after logging in
self.assertEqual(result.data.getTag("h1", b"Nicole") #What I imagined using <h1>
self.assertEqual(result.data.getId("user", b"Nicole") #What I imagined using id
#This returns true which is okay, because 'Nicole' exists in the whole page
self.assertIn(b'Nicole', result.data)
In my rendered jinja2 template I have this which is after logging in.
<h1 id="userTitle">{{ session['username'] }},Welcome!</h1>
I guess assertIn works well, but I just want to know how to test an HTML tag without running a browser test.
Although I didn't get a correct answer from here, but I just managed to do the unit-test with just assertIn, by checking the contents of the page.
Thanks everyone
I want to submit a protein sequence online for HMM comparison using HHpred tool and collect its job id so that I can collect the output later and process. But I have 1200 such sequences and I need to automate the process very badly. I tried to use mechanize package in Python but I couldn't understand properly as there is no documentation available.
import mechanize
ur = "http://toolkit.tuebingen.mpg.de/hhpred/"
request = mechanize.Request(ur)
response = mechanize.urlopen(request)
forms = mechanize.ParseResponse(response, backwards_compat=False)
print response.code
form = forms[0]
print form
original_text = form["jobid"]
form["jobid"] = '''MNDKSKNMMKNFIRTFAGLLLAILLILGFFLLVFPKAGDRFLADKKVSTLSAKNLTYAALGDSLTEGVGDATGQGGF VPLFAKDIENKTDSSVSSQNFGKAGDTSTQIYNRMMKSKKITDGLKKADIITITIGGNDVLKV
IRDNVSKLSSMTEKDFTKPEELYQARVKKLLDKIREDNPKAQIYVLGIYNPFYLNFPELTVMQNVIDSWNTATAGVVSQE KNTYFIPINDLLYKGSGDKQAVESGSTSDAVSNNLLYTEDHFHPNNVGYQLMADAVFASY
KEVNQK'''
control = form.find_control("jobid")
print control.name, control.value, control.type
control = form.find_control("showres")
print control.name, control.value, control.type
print control.disabled
request2 = form.click("showres")
response2 = mechanize.urlopen(request2)
forms2 = mechanize.ParseResponse(response2, backwards_compat=False)
form2 = forms2[0]
print form2
The website http://toolkit.tuebingen.mpg.de/hhpred/ has many input fields but i could see only the "jobid" and "showres" in the control list using 'mechanize' parser. Code above is what i tried to do but its totally incorrect.
I actually want to paste the sequence in the text box and hit on submit and if possible give my own job-id at the bottom. And save the url of the resulting page after hitting submit.
Kindly help me. (I'm using windows)
I'm currently trying to get a grasp on pycurl. I'm attempting to login to a website. After logging into the site it should redirect to the main page. However when trying this script it just gets returned to the login page. What might I be doing wrong?
import pycurl
import urllib
import StringIO
pf = {'username' : 'user', 'password' : 'pass' }
fields = urllib.urlencode(pf)
pageContents = StringIO.StringIO()
p = pycurl.Curl()
p.setopt(pycurl.FOLLOWLOCATION, 1)
p.setopt(pycurl.COOKIEFILE, './cookie_test.txt')
p.setopt(pycurl.COOKIEJAR, './cookie_test.txt')
p.setopt(pycurl.POST, 1)
p.setopt(pycurl.POSTFIELDS, fields)
p.setopt(pycurl.WRITEFUNCTION, pageContents.write)
p.setopt(pycurl.URL, 'http://localhost')
p.perform()
pageContents.seek(0)
print pageContents.readlines()
EDIT: As pointed out by Peter the URL should point to a login URL but the site I'm trying to get this to work for fails to show me what URL this would be. The form's action just points to the home page ( /index.html )
As you're troubleshooting this problem, I suggest getting a browser plugin like FireBug or LiveHTTPHeaders (I suggest Firefox plugins, but there are similar plugins for other browsers as well). Then you can exercise a request to the site and see what action (URL), method, and form parameters are being passed to the target server. This will likely help elucidate the crux of the problem.
If that's no help, you may consider using a different tool for your mechanization. I've used ClientForm and BeautifulSoup to perform similar operations. Based on what I've read in the pycURL docs and your code above, ClientForm might be a better tool to use. ClientForm will parse your HTML page, locate the forms on it (including login forms), and construct the appropriate request for you based on the answers you supply to the form. You could even use ClientForm with pycURL... but at least ClientForm will provide you with the appropriate action to which to POST, and construct all of the appropriate parameters.
Be aware, though, that if there is JavaScript handling any necessary part of the login form, even ClientForm can't help you there. You will need something that interprets the JavaScript to effectively automate the login. In that case, I've used SeleniumRC to control a browser (and I let the browser handle the JavaScript).
One of the golden rule, you need to 'brake the ice', have debugging enabled when trying to solve pycurl example:
Note: don't forget to use p.close() after p.perform()
def test(debug_type, debug_msg):
if len(debug_msg) < 300:
print "debug(%d): %s" % (debug_type, debug_msg.strip())
p.setopt(pycurl.VERBOSE, True)
p.setopt(pycurl.DEBUGFUNCTION, test)
Now you can see how your code is breathing, because you have debugging enabled
import pycurl
import urllib
import StringIO
def test(debug_type, debug_msg):
if len(debug_msg) < 300:
print "debug(%d): %s" % (debug_type, debug_msg.strip())
pf = {'username' : 'user', 'password' : 'pass' }
fields = urllib.urlencode(pf)
pageContents = StringIO.StringIO()
p = pycurl.Curl()
p.setopt(pycurl.FOLLOWLOCATION, 1)
p.setopt(pycurl.COOKIEFILE, './cookie_test.txt')
p.setopt(pycurl.COOKIEJAR, './cookie_test.txt')
p.setopt(pycurl.POST, 1)
p.setopt(pycurl.POSTFIELDS, fields)
p.setopt(pycurl.WRITEFUNCTION, pageContents.write)
p.setopt(pycurl.VERBOSE, True)
p.setopt(pycurl.DEBUGFUNCTION, test)
p.setopt(pycurl.URL, 'http://localhost')
p.perform()
p.close() # This is mandatory.
pageContents.seek(0)
print pageContents.readlines()