I am trying to extract data from a HTML file using python. I am trying to extract the table content from the file.
Below is the HTML content of the table:
<table class="radiobutton" id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay" onclick="return false;">
<tbody>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_0" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="1" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_0">Fitting</label>
</td>
</tr>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_1" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="2" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_1">Material</label>
</td>
</tr>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_2" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="4" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_2">Appliance</label>
</td>
</tr>
<tr>
<td>
<input checked="checked" id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_3" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="8" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_3">Apparatus</label>
</td>
</tr>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_4" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="16" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_4">Other procedures</label>
</td>
</tr>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_5" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="32" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_5">Alternative fuel oils</label>
</td>
</tr>
<tr>
<td>
<input id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_6" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="64" />
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_6">Other compliance method:</label>
</td>
</tr>
</tbody>
</table>
Below is the python code to print the properties from the tags.
from bs4 import BeautifulSoup
from pyparsing import makeHTMLTags
with open('.\ABC.html', 'r') as read_file:
data = read_file.read()
soup = BeautifulSoup(data, 'html.parser')
table = soup.find("table", attrs={"id":"ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay"})
spotterTag, spotterEndTag = makeHTMLTags("input")
for spotter in spotterTag.searchString(table):
print(spotter.checked)
print(spotter.id)
How can I print the label of the radio buttons along with checked property?
Examle: For below tag, it should print : Fitting
And "checked" for Input tag mentioned below:
<label for="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_0">Fitting</label>
<input checked="checked" id="ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay_3" name="ctl00$bodyPlaceHolder$ctl00$Reg$rblTypeDisplay" type="radio" value="8"/>
Below code works but needs a better solution:
from bs4 import BeautifulSoup
from pyparsing import makeHTMLTags
with open('.\ABC.html', 'r') as read_file:
data = read_file.read()
soup = BeautifulSoup(data, 'html.parser')
table = soup.find("table", attrs={"id":"ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay"})
spotterTag, spotterEndTag = makeHTMLTags("input")
for spotter in spotterTag.searchString(table):
if spotter.checked == 'checked':
label = soup.find("label", attrs={"for":spotter.id})
print(str(label)[str(label).find('>')+1:str(label).find('<',2)])
print(spotter.checked)
Thanks in advance for help!
I'm not sure if I understand you correctly, but do you want to zip input and labels together? If yes, you can use zip() function. For example (data is your HTML string):
from bs4 import BeautifulSoup
soup = BeautifulSoup(data, 'html.parser')
print('{:^25} {:^15} {:^15}'.format('Text', 'Value', 'Checked'))
for inp, lbl in zip(soup.select('table#ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay input'),
soup.select('table#ctl00_bodyPlaceHolder_ctl00_Reg_rblTypeDisplay label')):
print('{:<25} {:^15} {:^15}'.format(lbl.text, inp['value'], 'checked' if 'checked' in inp.attrs else '-'))
Prints:
Text Value Checked
Fitting 1 -
Material 2 -
Appliance 4 -
Apparatus 8 checked
Other procedures 16 -
Alternative fuel oils 32 -
Other compliance method: 64 -
Related
I'm trying to scrape the links connected to View Bill button within a table from a local html file. Here is the file link.
This is how the first container looks like:
<tr class="clickable collapsed ng-isolate-scope blue-row" data-parent="#parent-table-body" data-target="#tr1" data-toggle="collapse" role="button">
<td>
<div class="accordion-icon" ng-enterclick="" tabindex="0"></div>
</td>
<td>09/18/2020</td>
<td>$183.47</td>
<td>10/02/2020</td>
<td>29</td>
<td>$0.00</td>
<td>
<form action="/my-account/view-bill" class="hiddenForm ng-pristine ng-valid" method="post" target="_blank">
<input name="actionType" type="hidden" value="View Bill" autocomplete="off">
<input name="billDate" type="hidden" value="2020-09-18" autocomplete="off">
<a class="link" data-action="View Bill " data-category="billing_payment_history" data-label="Billing & Payment History" href="https://www.duke-energy.com/?_ga=2.36159203.2592906.1601114887-735893428.1601114887" onclick="this.parentNode.submit(); return false;">View Bill </a>
</form>
</td>
</tr>
I've tried with:
from bs4 import BeautifulSoup
local_file = r"C:\Users\WCS\Desktop\htmlfile.html"
with open(local_file,"r") as f:
page_content = f.read()
soup = BeautifulSoup(page_content,"lxml")
for item in soup.select("#parent-table-body a.link:contains('View Bill')"):
print(item)
break
Output I'm getting from the first container:
<a class="link" data-action="View Bill " data-category="billing_payment_history" data-label="Billing & Payment History" href="" onclick="this.parentNode.submit(); return false;">View Bill </a>
So, you can see that there is no link in the output above.
How can I parse the links from the view bill buttons?
from bs4 import BeautifulSoup
with open('htmlfile.html') as f:
soup = BeautifulSoup(f, 'html.parser')
target = [x['href'] for x in soup.select("a[data-action^=View]")]
print(target)
Try this:
import re
from bs4 import BeautifulSoup
html = """
<tr class="clickable collapsed ng-isolate-scope blue-row" data-parent="#parent-table-body" data-target="#tr1" data-toggle="collapse" role="button">
<td>
<div class="accordion-icon" ng-enterclick="" tabindex="0"></div>
</td>
<td>09/18/2020</td>
<td>$183.47</td>
<td>10/02/2020</td>
<td>29</td>
<td>$0.00</td>
<td>
<form action="/my-account/view-bill" class="hiddenForm ng-pristine ng-valid" method="post" target="_blank">
<input name="actionType" type="hidden" value="View Bill" autocomplete="off">
<input name="billDate" type="hidden" value="2020-09-18" autocomplete="off">
<a class="link" data-action="View Bill " data-category="billing_payment_history" data-label="Billing & Payment History" href="https://www.duke-energy.com/?_ga=2.36159203.2592906.1601114887-735893428.1601114887" onclick="this.parentNode.submit(); return false;">View Bill </a>
</form>
</td>
</tr>"""
soup = BeautifulSoup(html, "html.parser")
for anchor in soup.find_all(lambda t: t.name == 'a' and re.search(r'View Bill\s+', t.text)):
print(f"{anchor.text.strip()} - {anchor.get('href')}")
Output:
View Bill - https://www.duke-energy.com/?_ga=2.36159203.2592906.1601114887-735893428.1601114887
I'm scraping a webpage for a table using BeautifulSoup, but for some reason it is only scraping half the table. The half I'm getting is the part that doesn't contain the input fields. Here is the html data:
<table class="commonTable1" cellpadding="0" cellspacing="0" border="0" width="100%" id="portAllocTable">
<tbody>
<tr>
<th class="commonTableHeaderLastCell" colspan="2"><span class="commonBold"> Portfolio Allocation (%) </span></th>
</tr>
<tr>
<td colspan="2" class="commonHeaderContentSeparator"><img src="/fees-web/common/images/spacer.gif" height="1" style="display: block"></td>
</tr>
<tr>
<td>
<span>AdvisorGuided (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[0].feeCollectionRate" value="100" id="selText_1"><input type="text" name="portfolioChargeList[0].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="100" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>AdvisorGuided 2 (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[1].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[1].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Client Directed (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[2].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[2].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Holding MMKT (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[3].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[3].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Total</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<input type="hidden" name="portfolioChargeList[4].feeCollectionRate" value="100" id="selText_1Total"><input type="text" name="portfolioChargeList[4].feeCollectionRateINPUT" maxlength="3" value="100" maxvalue="100" decimals="0" blankifzero="true" id="selText_1TotalINPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
</tbody>
</table>
Here is my code:
url = driver.page_source
soup = BeautifulSoup(url, "lxml")
table = soup.find('table', id="portAllocTable")
rows = table.findAll('td')
list_of_rows = []
for row in table.findAll('tr'):
list_of_cells = []
for cell in row.findAll(["th","td"]):
text = cell.text
list_of_cells.append(text)
list_of_rows.append(list_of_cells)
for item in list_of_rows:
print(' '.join(item))
What am I doing wrong? Why is it only printing the left side of the table? Any recommendations about what to change would be much appreciated.
Results:
Portfolio Allocation (%)
AdvisorGuided (Capital Portfolio)
100 100
AdvisorGuided 2 (Capital Portfolio)
0 100
Client Directed (Capital Portfolio)
0 100
Holding MMKT (Capital Portfolio)
0 100
Total
100 100
You'll have to go further into the child and sibling nodes and pull out the attributes (those values aren't actual text/content.
import pandas as pd
import bs4
html = '''<table class="commonTable1" cellpadding="0" cellspacing="0" border="0" width="100%" id="portAllocTable">
<tbody>
<tr>
<th class="commonTableHeaderLastCell" colspan="2"><span class="commonBold"> Portfolio Allocation (%) </span></th>
</tr>
<tr>
<td colspan="2" class="commonHeaderContentSeparator"><img src="/fees-web/common/images/spacer.gif" height="1" style="display: block"></td>
</tr>
<tr>
<td>
<span>AdvisorGuided (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[0].feeCollectionRate" value="100" id="selText_1"><input type="text" name="portfolioChargeList[0].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="100" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>AdvisorGuided 2 (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[1].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[1].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Client Directed (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[2].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[2].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Holding MMKT (Capital Portfolio)</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<!-- When collection method is invoice, the portfolio to charge table should be diabled.
Else work as it was-->
<input type="hidden" name="portfolioChargeList[3].feeCollectionRate" value="0" id="selText_1"><input type="text" name="portfolioChargeList[3].feeCollectionRateINPUT" maxlength="3" onkeypress="return disableMinus();" onblur="updateTotal(1);" value="0" maxvalue="100" decimals="0" showalertdialog="true" blankifzero="true" id="selText_1INPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
<tr>
<td>
<span>Total</span>
</td>
<td class="commonTableBodyLastCell" align="right">
<span>
<input type="hidden" name="portfolioChargeList[4].feeCollectionRate" value="100" id="selText_1Total"><input type="text" name="portfolioChargeList[4].feeCollectionRateINPUT" maxlength="3" value="100" maxvalue="100" decimals="0" blankifzero="true" id="selText_1TotalINPUT" style="text-align:right;width:50px" class="commonTextBoxAmount">
</span>
</td>
</tr>
</tbody>
</table>'''
soup = bs4.BeautifulSoup(html, "lxml")
table = soup.find('table', id="portAllocTable")
rows = table.findAll('td')
list_of_rows = []
for row in table.findAll('tr'):
list_of_cells = []
for cell in row.find_all(["th","td"]):
text = cell.text
try:
val = cell.find('input')['value']
max_val = cell.find('input').next_sibling['maxvalue']
list_of_cells.append(val)
list_of_cells.append(max_val)
except:
pass
list_of_cells.append(text)
list_of_rows.append(list_of_cells)
for item in list_of_rows:
print(' '.join(item))
To make a table, you could do something like this. You'll have to do a bitt of clean up, but should get you going:
results = pd.DataFrame()
for row in table.findAll('tr'):
for cell in row.find_all(["th","td"]):
text = cell.text
try:
val = cell.find('input')['value']
max_val = cell.find('input').next_sibling['maxvalue']
except:
val = ''
max_val = ''
pass
temp_df = pd.DataFrame([[text, val, max_val]], columns=['text','value','maxvalue'])
results = results.append(temp_df).reset_index(drop=True)
A few things come to mind.
First: it should be rows = table.findAll('tr') as the tr HTML tag designates rows. Subsequently, it should for row in table.findAll('td'): as the td HTML tag is the cell tag. But you're not even using the rows variable, so the point is moot. If you want you could do something like this:
soup = BeautifulSoup(url, "lxml")
table = soup.find('table', id="portAllocTable")
rows = table.findAll("tr")
list_of_rows = []
for row in rows:
list_of_cells = []
for cell in row.findAll(['th', 'td']):
text = cell.text
list_of_cells.append(text)
list_of_rows.append(list_of_cells)
for item in list_of_rows:
print(' '.join(item))
Second, this code wouldn't get the text in the input fields, so this is probably why you only see the text on the left side.
Finally, you could try a difference parser, such as html5lib.
I need to get the name and the price of each row in the sample html below, however when I'm using beatifulsoup to find_all('tr') it returns all the tr of the main table and the nested tables. what is the best way for extracting only the value and the price of each row?
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
Above is what I have and it returns "all" the tr including the nested tables.
What I need is to get all the names and the price of each item in front of it, and finally save them in a csv file
<table class="table table-hover table-responsive">
<tbody><tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Contact Energy Saver Plus</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$242.85 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.334</td><td>per kWh</td><tr><td>Daily</td><td>$0.333</td><td>per day</td><tr><td>EA Levy</td><td>$0.0013</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$63.14 (26%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>Provides fixed pricing until 31 June 2021 unless there are changes to taxes and levies.</li><li>24% Prompt Payment Discount when you pay on time. additional 1% discount for paying by direct debit (excl. credit card), and 1% discount for getting bills and correspondence by email. Up to 26% PPD available.</li><li>An early termination fee of $150 per contracted ICP if you terminate the contract before the end date�(31/06/2021). Fee may be waived if you are moving house and take Contact Energy to the new property.</li><li>Not available to prepay customers.</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w0" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form> <div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
<td style="text-align: center" class="hidden-xs">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
<td class="hidden-xs">
<form id="w1" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch">Switch Now!</button> </form> <div class="wannatalk">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
<tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Powershop Saver</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$183.40 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.2508</td><td>per kWh</td><tr><td>Daily</td><td>$0.30</td><td>per day</td><tr><td>EA Levy</td><td>$0.00</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$0.00 (0%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>The price estimate is based on forecast charges from Powershop for the next 12 months.</li><li>It assumes you purchase the Powershop Simple Saver powerpack once a month and special powerpacks that are made available from time to time.</li><li>This offer does not require a contract or a minimum supply period.</li><li>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, $10 on the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).�</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$183.40</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Get $150 off your bill over 12 months!<br /> <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="<div><div>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, then $10 for the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).</div><div>�</div></div><div><br></div><div><br></div>">More Info</a> </div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w2" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="53" /> <input type="hidden" name="plan_stage_id" value="273" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form><div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
so the output should be:
from td[3] and td[4] in first row:
Contact Energy Saver Plus
$179.71
and then the next row:
Powershop Saver
$183.40
and so on until the last row ( of the main table).
Similar process to that given in comments but different selectors
from bs4 import BeautifulSoup as bs
html = '''yourhtml'''
soup = bs(html, 'lxml')
names = [item.text for item in soup.select('.table h4 ')]
prices = [item.text for item in soup.select('[colspan="2"] > .total')]
results = list(zip(names, prices))
print(results)
I actually managed to solve this with using regex. I like the approach in the above answer much better specially using zip(), but I though pasting my solution here in case it becomes handy to some other readers.
deals=[]
prices=[]
results={}
with open("prices.html", "r") as f:
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
tbody = priceTable.find('tbody')
pplanPattern = '<td\ colspan="3"><h4>([^<]+)<\/h4><\/td>'
pricePatterns = '<h3 class="total">([^<]+)<\/h3>'
for rw in tbody:
plan = re.search(pplanPattern, rw)
price = re.search(pricePatterns, rw)
if plan:
deals.append(plan.group(1))
if price:
deals.append(price.group(1))
results[plan.group(1)] = price.group(1)
I am creating a flask form which requires login, and after login it goes to the entry form where we have check boxes and text entry.
I am facing a specific problem: I am unable to get the value of text boxes but getting the value of checkboxes.
I using flask and using every request method to print my text boxes but not getting the values.
below is my code for main file:
from flask import Flask, render_template
import os
from flask import redirect, url_for, request
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"]="sqlite:////OtrsSummary.db"
app.config["SECRET_KEY"]="thisiskey"
db = SQLAlchemy(app)
# #app.route("/ndex")
# def home():
# names = os.getlogin().split(".")[0].title()
# return render_template("index.html", name=names)
#app.route("/welcome", methods=['GET', 'POST'])
def welcome():
if request.method=="POST":
try:
phase = request.form.get("phase")
rphase = phase.replace("on", "1")
print(rphase)
sale = request.form.get("sale")
rsale = sale.replace("on", "1")
print(rsale)
floor = request.form.get("floor")
rfloor = floor.replace("on", "1")
options = request.form.get("options")
roptions = options.replace("on", "1")
image = request.form.get("image")
rimage = image.replace("on", "1")
video = request.form.get("video")
rvideo = video.replace("on", "1")
possession = request.form.get("possession")
rpossession = possession.replace("on", "1")
amenities = request.form.get("amenities")
ramenities = amenities.replace("on", "1")
prdeactivation = request.form.get("prdeactivation")
rprdeactivation = prdeactivation.replace("on", "1")
np = request.form.get("np")
rnp = np.replace("on", "1")
newbooking = request.form.get("newbooking")
rnewbooking = newbooking.replace("on", "1")
bank = request.form.get("bank")
rbank = bank.replace("on", "1")
lat = request.form.get("lat")
rlat = lat.replace("on", "1")
usp = request.form.get("usp")
rusp = usp.replace("on", "1")
fact = request.form.get("fact")
rfact = fact.replace("on", "1")
prname = request.form.get("prname")
rprname = prname.replace("on", "1")
prdescription = request.form.get("prdescription")
rprdescription = prdescription.replace("on", "1")
prspecification = request.form.get("prspecification")
rprspecification = prspecification.replace("on", "1")
builderdetails = request.form.get("builderdetails")
rbuilerdetails = builderdetails.replace("on", "1")
tco = request.form.get("tco")
rtco = tco.replace("on", "1")
npdeactivation = request.form.get("npdeactivation")
rnpdeactivation = npdeactivation.replace("on", "1")
constuctionimages = request.form.get("constuctionimages")
rconstuctionimages = constuctionimages.replace("on", "1")
brochure = request.form.get("brochure")
rbrochure = brochure.replace("on", "1")
rera = request.form.get("rera")
rrera = rera.replace("on", "1")
rticketnumber = request.form["ticketnumber"]##here not getting the value
rxidnumber = request.form["xidnumber"]##here not getting the value
rreranumber = request.form["reranumber"]##here not getting the value
print(rxidnumber)
print(rticketnumber)
msg = "Entry Submitted Successfully"
except AttributeError:
msg = "Please Do Not Submit Blank Form"
#con.close()
return render_template("same.html")
#app.route("/", methods=["GET", "POST"])
def log():
names = os.getlogin().split(".")[0].title()
error= None
if request.method == "POST":
if request.form["username"]!= os.getlogin() or request.form["password"]!="1234":
error = "Invalid Credentials.Please Try again."
else:
return redirect(url_for("welcome"))
return render_template("index.html", error=error, name=names)
app.run(debug=True)
This is my html:
<!doctype <!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Page Title</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- <link rel="stylesheet" type="text/css" media="screen" href="main.css" /> -->
<!-- <script src="main.js"></script> -->
</head>
<body>
<form action="" ALIGN="CENTRE" method="POST">
<h3>OTRS Basic Information form</h3>
<TABLE>
<TR>
<TD>
Phases and Tower</TD>
<TD>
<input type="checkbox" name="phase">
</TD>
<TD>
Saleable Mapping </TD>
<TD>
<input type="checkbox" name="sale">
</TD>
</TR>
<TR>
<TD>
Floor Plan </TD>
<TD>
<input type="checkbox" name="floor">
</TD>
<TD>
Options </TD>
<TD>
<input type="checkbox" name="options">
</TD>
</TR>
<TR>
<TD>
Images </TD>
<TD>
<input type="checkbox" name="image">
</TD>
<TD>
Video </TD>
<TD>
<input type="checkbox" name="video">
</TD>
</TR>
<TR>
<TD>
Possession Status/Date </TD>
<TD>
<input type="checkbox" name="possession">
</TD>
<TD>
Amenities </TD>
<TD>
<input type="checkbox" name="amenities">
</TD>
</TR>
<TR>
<TD>
Project Deactivation </TD>
<TD>
<input type="checkbox" name="prdeactivation">
</TD>
<TD>
Np Slot Changes/Refresh </TD>
<TD>
<input type="checkbox" name="np">
</TD>
</TR>
<TR>
<TD>
New Booking/Resale Lock </TD>
<TD>
<input type="checkbox" name="newbooking">
</TD>
<TD>
Bank
</TD>
<TD>
<input type="checkbox" name="bank">
</TD>
</TR>
<TR>
<TD>
Lat Long/Location </TD>
<TD>
<input type="checkbox" name="lat">
</TD>
<TD>
USP </TD>
<TD>
<input type="checkbox" name="usp">
</TD>
</TR>
<TR>
<TD>
Fact Table </TD>
<TD>
<input type="checkbox" name="fact">
</TD>
<TD>
Project Name </TD>
<TD>
<input type="checkbox" name="prname">
</TD>
</TR>
<TR>
<TD>
Project Description </TD>
<TD>
<input type="checkbox" name="prdescription">
</TD>
<TD>
Project Specification </TD>
<TD>
<input type="checkbox" name="prspecification">
</TD>
</TR>
<TR>
<TD>
Builder Details </TD>
<TD>
<input type="checkbox" name="builderdetails">
</TD>
<TD>
TCO/Payment Plan</TD>
<TD>
<input type="checkbox" name="tco">
</TD>
</TR>
<TR>
<TD>
NP Deactivation </TD>
<TD>
<input type="checkbox" name="npdeactivation">
</TD>
<TD>
Construction Images </TD>
<TD>
<input type="checkbox" name="constuctionimages">
</TD>
</TR>
<TR>
<TD>
Brochure </TD>
<TD>
<input type="checkbox" name="brochure">
</TD>
<TD>
Rera Available </TD>
<TD>
<input type="checkbox" name="rera">
</TD>
</TR>
<TR>
<TD>
Ticket Number </TD>
<TD><input type="text" name="ticketnumber">
</TD>
<TD>
XID Number </TD>
<TD>
<input type="text" name="xidnumber">
</TD>
<TD>
Rera Number </TD>
<TD>
<input type="text" name="reranumber">
</TD>
</TABLE>
<input type="submit" value="submit"><br>{{msg}}
</form>
</body>
</html>
Can anyone please suggest some solutions?
Actually I tried running your code and I'll get the values of textbox only if all the above checkbox is checked.
Actually you are catching the attribute exception but not handling it properly.In your code if any of the checkbox goes unchecked it'll give the exception 'NoneType' object has no attribute 'replace'.which is not handled and because of that it will not execute the next lines of code.
My suggestion on working with checkbox is make a hidden type with same name and value='off' so if it is unchecked it'll give the off value.
<input type='hidden' name="checkbox_name" value="off">
request.form is a dictionary, so you can check if your checkbox has been checked like this:
sale = 'checked' if 'sale' in request.form else 'not_checked'
I'm new with python and beautifulsopu lib. I have tried many things, but no luck.
My html code could be like:
<form method = "post" id="FORM1" name="FORM1">
<table cellpadding=0 cellspacing=1 border=0 align="center" bgcolor="#cccccc">
<tr>
<td class="producto"><b>Club</b><br>
<input value="CLUB TENIS DE MESA PORTOBAIL" disabled class="txtmascaraform" type="TEXT" name="txtClub" size="60" maxlength="55">
</td>
<tr>
<td colspan="2" class="producto"><b>Nombre Equipo</b><br>
<input value="C.T.M. PORTOBAIL" disabled class="txtmascaraform" type="TEXT" name="txtNomEqu" size="100" maxlength="80">
</td>
</tr>
<tr>
<td class="producto"><b>Telefono fijo</b><br>
<input value="63097005534" disabled class="txtmascaraform" type="TEXT" name="txtTelf" size="15" maxlength="10">
</td
and I need JUST to take what is within <"b"><"/b"> and its "input value" .
Many thanks!!
First find() your form by id, then find_all() inputs inside and get the value of value attribute:
from bs4 import BeautifulSoup
data = """<form method = "post" id="FORM1" name="FORM1">
<table cellpadding=0 cellspacing=1 border=0 align="center" bgcolor="#cccccc">
<tr>
<td class="producto"><b>Club</b><br>
<input value="CLUB TENIS DE MESA PORTOBAIL" disabled class="txtmascaraform" type="TEXT" name="txtClub" size="60" maxlength="55">
</td>
<tr>
<td colspan="2" class="producto"><b>Nombre Equipo</b><br>
<input value="C.T.M. PORTOBAIL" disabled class="txtmascaraform" type="TEXT" name="txtNomEqu" size="100" maxlength="80">
</td>
</tr>
<tr>
<td class="producto"><b>Telefono fijo</b><br>
<input value="63097005534" disabled class="txtmascaraform" type="TEXT" name="txtTelf" size="15" maxlength="10">
</td>
</tr>
</table>
</form>"""
soup = BeautifulSoup(data)
form = soup.find("form", {'id': "FORM1"})
print [item.get('value') for item in form.find_all('input')]
# UPDATE for getting table cell values
table = form.find("table")
print [item.text.strip() for item in table.find_all('td')]
prints:
['CLUB TENIS DE MESA PORTOBAIL', 'C.T.M. PORTOBAIL', '63097005534']
[u'Club', u'Nombre Equipo', u'Telefono fijo']