Develop Reference

How to parse a html string using python scrapy

I have a list of html input elements as below.
lists=[<input type="hidden" name="csrf_token" value="jZdkrMumEBeXQlUTbOWfInDwNhtVHGSxKyPvaipoAFsYqCgRLJzc">,
<input type="text" class="form-control" id="username" name="username">,
<input type="password" class="form-control" id="password" name="password">,
<input type="submit" value="Login" class="btn btn-primary">]
From these I need to extract the attribute values of name, type, and value
For eg:
Consider the input <input type="hidden" name="csrf_token" value="jZdkrMumEBeXQlUTbOWfInDwNhtVHGSxKyPvaipoAFsYqCgRLJzc">
then I need output as following dictionary format
{'csrf_token':('hidden',"jZdkrMumEBeXQlUTbOWfInDwNhtVHGSxKyPvaipoAFsYqCgRLJzc")}
Could anyone please a guidance to solve this

I recommend you to use the Beautiful Soup Python library (https://pypi.org/project/beautifulsoup4/) to get the HTML content and the values of the elements. There are functions already created for that purpose.

Django1.8 and Python 2.7: Radio button stores the first option whatever is my choice

I'm using Django 1.8 and python 2.7
In my template I have a radio button.
<div class="form-group">
<label class="col-md-4 control-label" for="radios">Liquids:</label>
<div class="col-md-4">
<label class="radio-inline" for="radios">
<input type="radio" name="liquids" id="liquids" value="1" disabled>
With cough
</label>
<label class="radio-inline" for="radios">
<input type="radio" name="liquids" id="liquids" value="0" >
No cough
</label>
</div>
</div>
When I push the submit button whatever is my choice, the first option is taken as selected. Do you know how to fix this?
How do I get the correct value of radio choice in views.py?
Also, when I render the template having filled values with selected the 2nd option, the radio button for the first option is selected in template.

Can you provide your complete HTML page? Probably you forgot to use <form></form> HTML tag.
When you need to get the selected values of an input element you can use jQuery. The function .serializeArray(); will come in handy.
var fields = $( ":input" ).serializeArray();

how to locate an element inside expanded list

I have a little problem. I am writing an automatic test, that has to go to webpage, enter some keys to search, then go to advanced search options, choose one from a list and search. The point is, that I have absolutely no idea on how to locate element on a list. It looks like this: http://www.e-podroznik.pl/public/searcherFinal.do. If you click on "więcej opcji" you'll se a list of options about journey. My question: how to locate element on this expanded "więcej opcji"-list using python selenium? I have tried everything, from find_element_by* to execute_script; nothing works. I get errors like 'unable to locate element', 'compound not permitted' etc.
The HTML:
<div class="advanced-searcher dNone more-options" style="display: block;">
<label class="lblDepartureArrival"></label>
<label class="lblRadio frmButtonFlat lblDeparture"></label>
<label class="lblRadio frmButtonFlat lblDeparture"></label>
<span class="lblOmmit"></span>
<label class="lblCaptionCarrierType"></label>
<label class="lblCarrierType lblCheckbox frmButtonFlat carrierType-bus"></label>
<label class="lblCarrierType lblCheckbox frmButtonFlat carrierType-rail"></label>
<label class="lblCarrierType lblCheckbox frmButtonFlat carrierType-aut"></label>
<label class="lblCarrierType lblCheckbox frmButtonFlat carrierType-city"></label>
<label class="lblCaptionJourneyMode"></label>
<span class=""></span>
<span class=""></span>
<span class=""></span>
<label class="lblSort">
<span class="fldCaption">
Sortuj według:
</span>
<span class="fldContainer">
<select id="sortTypeV_1399324651976" name="formCompositeSearchingResults.formCompositeSearcherFinalH.sortTypeV" style="display: none;">
<option selected="selected" value="departure"></option>
<option value="time"></option>
<option value="price"></option>
</select>
<span class="frmDropdown"></span>
</span>
</label>
<a class="lnkSwitchSearcherType" title="Wyszukiwarka zaawansowana" href="/public/searcher.do?method=task&advanced=false"></a>
</div>
I specifically need to expand list:
"<select id="sortTypeV_1399324651976" name="formCompositeSearchingResults.formCompositeSearcherFinalH.sortTypeV" style="display: none;">" and then click "<option value="time"></option>
This doesn't work:
def do_the_advanced_search(self):
self.driver.find_element_by_xpath('//*[#id="frm_1399323992507"]/fieldset/div/div[2]/label[3]').click()
time.sleep(3)
neither this:
self.driver.execute_script("document.getElementById('sortTypeV_1399324651976').style=='display: inline block';")
Thanks for any help.

To expand the part that contains the dropdown, you can use this:
self.driver.find_element_by_css_selector("a>span.imageSprite.sprite-label-left").click()
Then you can use execute_script to set the value:
self.driver.execute_script("document.getElementsByClassName('current').value = 'Najkrótszy czas';")
The select itself is hidden via style=display:none; so it cannot be manipulated directly.

Handle Multiple Checkboxes with a Single Serverside Variable

I have the following HTML code:
<form method="post">
<h5>Sports you play:</h5>
<input type="checkbox" name="sports_played" value="basketball"> basketball<br>
<input type="checkbox" name="sports_played" value="football"> football<br>
<input type="checkbox" name="sports_played" value="baseball"> baseball<br>
<input type="checkbox" name="sports_played" value="soccer"> tennis<br>
<input type="checkbox" name="sports_played" value="mma"> MMA<br>
<input type="checkbox" name="sports_played" value="hockey"> hockey<br>
<br>
<input class="btn" type="submit">
</form>
And then ideally I would like to have the following python serverside code:
class MyHandler(ParentHandler):
def post(self):
sports_played = self.request.get('sports_played')
#sports_played is a list or array of all the selected checkboxes that I can iterate through
I tried doing this by making the HTML sports_played name and array, sports_played[], but that didn't do anything and right now it just always returns the first selected item.
Is this possible? Really I just don't want to have to do a self.request.get('HTML_item') for each and every checkbox incase I need to alter the HTML I don't want to have to change the python.
Thanks!

The answer is shown in the webapp2 docs for the request object:
self.request.get('sports_played', allow_multiple=True)
Alternatively you can use
self.request.POST.getall('sports_played')

The name of the inputs should have [] at the end so that they are set to the server as an array. Right now, your multiple checkboxes are being sent to the server as many variables with the same name, so only one is recognized. It should look like this:
<form method="post">
<h5>Sports you play:</h5>
<input type="checkbox" name="sports_played[]" value="basketball"> basketball<br>
<input type="checkbox" name="sports_played[]" value="football"> football<br>
<input type="checkbox" name="sports_played[]" value="baseball"> baseball<br>
<input type="checkbox" name="sports_played[]" value="soccer"> tennis<br>
<input type="checkbox" name="sports_played[]" value="mma"> MMA<br>
<input type="checkbox" name="sports_played[]" value="hockey"> hockey<br>
<br>
<input class="btn" type="submit">
</form>
Now, if you select more than one, the values will be sent as an array.

Although this answer is not related to this question, but it may help all the django developers who is walking here and there.
In Django request.POST is a QueryDict object. So you can get all the values as list by following way
request.POST.getlist('sports_played')
N.B: This only works in Django

How do I determine the value of a dropdown?

This is the Html form:
<form action='use.py'>
<div><input type='text' name='etc'></div>
<p><input type='submit' value='etc!'></p>
</form>
And this is the python for it
colour = form["colour"].value
The top html form is a text box that users can type something in.
If I were to have a drop down box / radio button form like this:
<form action='etc.py'>
<p>List Type:</p>
<div><input type='radio' name='status' value='bulleted' checked>
Bulleted
</div>
<div><input type='radio' name='status' value='numbered'>
Numbered
</div>
<p>Text style:</p>
<div><input type='checkbox' name='bold'>
<b>Bold</b>
</div>
<div><input type='checkbox' name='italic'>
<i>Italic</i>
</div>
<p>Movies to display:</p>
<div>
<select name='type'>
<option>Only numbers</option>
<option>Only names</option>
</select>
</div>
<p><input type='submit' value='Display'></p>
</form>
How would I write my python? Like this?
only_numbers = form["Only numbers"].value

You can write a Python CGI script, but most Python frameworks are built upon WSGI and have a concept of request/response objects. For example, in Django the value would be available at request.POST['fieldname'].
Using the python CGI module, it is like:
form = cgi.FieldStorage()
# for multiple elements with same name or elements with multiple values
a_list = form.getlist("multiple_select_input")
# For single value elements
if "single_value_input" in form:
x = form["single_value_input"].value
else:
x = "default value"
# or
x = form.getvalue("single_value_input", "default value")