I've just parsed a web page via pandas:
r = requests.post("https://www.eigroup.co.uk/clients/auctions/fulldetails.aspx?auctionid=17999 ", params=payload)
parsed_page = pd.read_html(r.text, attrs={"class": "table-search-result"})
(an example of the HTML being parsed)
<table cellspacing="0" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1" style="width:100%;border-collapse:collapse;">
<tr>
<td colspan="2">
<table class="table-search-result">
<tr>
<th>66D Charlwood Street, Pimlico, London, SW1V 4PQ</th>
<th style="text-align: right; white-space: nowrap;">
<a href="http://www.englishhouseprices.com/results.aspx?postcode=SW1V 4PQ" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A2" class="icon" target="_blank">
<img src="/content/images/icons/32/houseprices.png" alt="Compare with Property Prices" title="Compare with Property Prices in this Postcode" /></a>
<a id="" title="View Auction Details" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank"><img title="View Auction Details" src="/content/images/icons/32/auctiondetails.png" alt="" /></a>
<a id="" title="Trend Analysis" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/lots/trend-analysis.aspx?lotid=756425" target="_blank"><img title="Trend Analysis" src="/content/images/icons/32/piechart.png" alt="" /></a>
<a href='http://maps.google.co.uk?q=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageLocationMap" title="Location Map" class="icon" src="/content/images/icons/32/compass.png" /></a>
<a href='http://www.multimap.com/map/photo.cgi?scale=5000&mapsize=big&pc=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageAerialPhoto" title="Aerial Photo" class="icon" src="/content/images/icons/32/camera.png" /></a>
<a href='/clients/search/search-results.aspx?searchtype=comparable&lotid=756425' title="Find similar properties like this one">
<img src="/content/images/icons/32/find.png" alt="Find other properties matching this tenant" title="Find similar properties like this one" class="icon" /></a>
<a href='/clients/search/search-results.aspx?searchtype=history&lotid=756425'>
<img src="/content/images/icons/32/history.png" alt="Find history of property in this street" title="Find history of property in this street" class="icon" /></a>
<a id="" title="Add to one of my portfolios" class="icon" Title="Add to portfolio" onclick="return o(this,650,500,1,1)" href="/clients/portfolios/lot.aspx?lotid=756425" target="_blank"><img title="Add to one of my portfolios" src="/content/images/icons/32/briefcase.png" alt="" /></a>
<a href="https://www.eigroup.co.uk/files/55/17999/6ec339ec-d59e-4b8a-9136-dc6e9a583328.pdf" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A4" target="_blank">
<img src="/content/images/icons/32/catalogue.png" alt="Catalogue Entry" class="icon" title="Full Catalogue Entry" /></a>
<a id="" title="Add to my shortlist" class="icon" Title="Add to shortlist" onclick="return o(this,900,650,1,1)" href="/clients/lots/shortlist.aspx?lotid=756425" target="shortlist"><img title="Add to my shortlist" src="/content/images/icons/32/shortlist.png" alt="" /></a>
</th>
</tr>
<tr>
<td colspan="2" style="background-color: #f5f5f5;">
<table style="width: 100%">
<tr>
<td style="background-color: #f1f1f1; width: 170px; text-align: center;">
<a href='/clients/lots/details.aspx?lotid=756425&hb=1' target='756425' onclick="window.open(this.href,this.target,'width=900,height=650,resizable=yes,scrollbars=yes');return false" title="Auction property in Pimlico, London, SW1">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_Image1" src="https://www.eigroup.co.uk/files/55/17999/de591a4f-7da1-4bcd-a42c-76731bd72a23.jpg" alt="Pimlico, London, SW1" style="border-color:Black;border-width:2px;border-style:Solid;width:150px;" />
</a>
</td>
<td style="padding-left: 10px; width: 50%;">
<p>
<b>Description</b><br />
Leasehold 2nd Floor Studio Flat Unmodernised Vacant
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P1">
<b>Guide Price</b><br />
£450,000 Plus
</p>
<p>
<b>Lot Number</b><br />
2
</p>
<p>
<b> </b>
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auctioneer</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctioneers/details.aspx?auctioneerid=55" target="_blank">Savills (London - National)</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P3">
<b>Vendor</b><br />
Housing Association
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auction Date</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank">28 October 2014</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P7">
<b>Lease Details</b><br />
125 Yr, commencing 01/01/2013 (GR.£250.PA)
</p>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
and I get the following:
In [86]: parsed_page[1][0][1]
Out[86]: u'Description Leasehold 2nd Floor Studio Flat Unmodernised Vacant Guide Price \xa3450,000 Plus Lot Number 2 Auctioneer Savills (London - National) Vendor Housing Association Auction Date 28 October 2014 Lease Details 125 Yr, commencing 01/01/2013 (GR.\xa3250.PA)'
The problem is, I want to be able to extract the description, guide price etc, but there aren't any delimiters and the number of characters afterwards is variable. Am I missing a keyword when I'm parsing?
How can I then split them into new columns?
Using beautifulSoup as I recommended in an answer to your last question, you can split the text and make a dict :
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
s = soup.find_all("p")
details = (ele.text.strip().split("\n") for ele in s)
d = {}
for det in details:
if len(det) == 2:
d[det[0].strip()] = det[1].strip()
{u'Vendor': u'Housing Association', u'Description': u'Leasehold 2nd Floor Studio Flat Unmodernised Vacant', u'Auction Date': u'28 October 2014', u'Auctioneer': u'Savills (London - National)', u'Lot Number': u'2', u'Guide Price': u'\xc2\u0141450,000 Plus', u'Lease Details': u'125 Yr, commencing 01/01/2013 (GR.\xc2\u0141250.PA)'}
Related
I have this HTML code for example:
<table class="nested4">
<tr>
<td colspan="1"></td>
<td colspan="2">
<h2 class="zeroMargin" id="govtMsg" visible="false"></h2>
</td>
<td colspan="2">
<h2 class="zeroMargin "> Net Metering Conn. </h2>
</td>
<td colspan="2">
<h2 class="zeroMargin" hidden> Life Line Consumer</h2>
</td>
</tr>
<tr>
<td colspan="2">
<p style="margin: 0; text-align: left; padding-left: 5px">
<span>NAME & ADDRESS</span>
<br />
<span>MUHAMMAD AMIN </span>
<br />
<span>S/O MUHAMMAD KHAN </span>
<br />
<span>H-NO.38 MARGALLA ROAD </span>
<br />
<span>F-6/3 ISLAMABAD3 </span>
<br />
<span></span>
</p>
</td>
<td colspan="3" style="text-align: left">
<h2 class="color-red">Say No To Corruption</h2>
<span style="font-size: 8pt; color: #78578e"> MCO Date : 10-Aug-2018</span>
<br />
</td>
<td>
<h3 style="font-size: 14pt;"> </h3>
<h2> <br /> </h2>
</td>
</tr>
<tr>
<td style="margin-top: 0;" class="border-b">
<br />
</td>
<td colspan="1" style="margin-top: 0;" class="border-b">
</td>
<td colspan="1" style="margin-top: 0;" class="border-b">
</td>
</tr>
<tr style="height: 7%;" class="border-tb">
<td style="width: 130px" class="border-r">
<h4>METER NO</h4>
</td>
<td style="width: 90px" class="border-r">
<h4>PREVIOUS READING</h4>
</td>
<td style="width: 90px" class="border-r">
<h4>PRESENT READING</h4>
</td>
<td style="width: 60px" class="border-r">
<h4>MF</h4>
</td>
<td style="width: 60px" class="border-r">
<h4>UNITS</h4>
</td>
<td>
<h4>STATUS</h4>
</td>
</tr>
<tr style="height: 30px" class="content">
<td class="border-r">
3-P I 3301539<br> I 3301539<br> E 3301539<br> E 3301539<br>
</td>
<td class="border-r">
78693<br>16823<br>19740<br>8<br>
</td>
<td class="border-r">
80086<br>17210<br>20139<br>8<br>
</td>
<td class="border-r">
1<br>1<br>1<br>1<br>
</td>
<td class="border-r">
1393<br>387<br>399<br>0<br>
</td>
<td>
</td>
</tr>
<tr id="roshniMsg" style="height: 30px" class="content">
<td colspan="6">
<div style="width: 452pt">
<img style="max-width: 100%; max-height: 35%" src="/images/companies/iesco/roshniMsg.jpg"
alt="Roshni Message" />
</div>
</td>
</tr>
</table>
From this table I want to extract the paragraph and from there I want to get all the span tags in that paragraph.
I used soup.find_all() to get the table but I don't know how to use this function iteratively to pass it back to the original soup object so that I could find the paragraph and, moreover the span tags in that paragraph.
This is the code Python code I wrote:
soup = BeautifulSoup(string, 'html.parser')
#Getting the table tag
results = soup.find_all('table', attrs={'class':'nested4'})
#Getting the paragragh tag
results = soup.find_all('p', attrs={'style':'margin: 0; text-align: left; padding-left: 5px'})
#Getting all the span tags
results = soup.find_all('span', attrs={})
I just want help on how to get the paragraphs within the table. And then how to get the spans within the paragraph as I am getting the spans in all of the original HTML code. I don't know how to pass the bs4 object list back to the soup object to use soup.find_all iteratively.
from bs4 import BeautifulSoup
html = '''
<table class="nested4">
<tr>
<td colspan="1"></td>
<td colspan="2">
<h2 class="zeroMargin" id="govtMsg" visible="false"></h2>
</td>
<td colspan="2">
<h2 class="zeroMargin "> Net Metering Conn. </h2>
</td>
<td colspan="2">
<h2 class="zeroMargin" hidden> Life Line Consumer</h2>
</td>
</tr>
<tr>
<td colspan="2">
<p style="margin: 0; text-align: left; padding-left: 5px">
<span>NAME & ADDRESS</span>
<br />
<span>MUHAMMAD AMIN </span>
<br />
<span>S/O MUHAMMAD KHAN </span>
<br />
<span>H-NO.38 MARGALLA ROAD </span>
<br />
<span>F-6/3 ISLAMABAD3 </span>
<br />
<span></span>
</p>
</td>
<td colspan="3" style="text-align: left">
<h2 class="color-red">Say No To Corruption</h2>
'''
soup = BeautifulSoup(html, 'html.parser')
spans = soup.select_one('table.nested4').select('span')
for span in spans:
print(span.text)
This returns:
NAME & ADDRESS
MUHAMMAD AMIN
S/O MUHAMMAD KHAN
H-NO.38 MARGALLA ROAD
F-6/3 ISLAMABAD3
if you have one table:
soup = BeautifulSoup(string, 'html.parser')
table = soup.find('table', attrs={'class': 'nested4'})
p = table.find('p', attrs={'style': 'margin: 0; text-align: left; padding-left: 5px'})
results = p.find_all('span')
for result in results:
print(result.get_text(strip=True))
if you have list of tables:
soup = BeautifulSoup(string, 'html.parser')
for table in soup.find_all('table', attrs={'class': 'nested4'}):
for p in table.find_all('p', attrs={'style': 'margin: 0; text-align: left; padding-left: 5px'}):
for span in p.find_all('span'):
print(span.get_text(strip=True))
I need to get the name and the price of each row in the sample html below, however when I'm using beatifulsoup to find_all('tr') it returns all the tr of the main table and the nested tables. what is the best way for extracting only the value and the price of each row?
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
Above is what I have and it returns "all" the tr including the nested tables.
What I need is to get all the names and the price of each item in front of it, and finally save them in a csv file
<table class="table table-hover table-responsive">
<tbody><tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Contact Energy Saver Plus</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$242.85 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.334</td><td>per kWh</td><tr><td>Daily</td><td>$0.333</td><td>per day</td><tr><td>EA Levy</td><td>$0.0013</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$63.14 (26%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>Provides fixed pricing until 31 June 2021 unless there are changes to taxes and levies.</li><li>24% Prompt Payment Discount when you pay on time. additional 1% discount for paying by direct debit (excl. credit card), and 1% discount for getting bills and correspondence by email. Up to 26% PPD available.</li><li>An early termination fee of $150 per contracted ICP if you terminate the contract before the end date�(31/06/2021). Fee may be waived if you are moving house and take Contact Energy to the new property.</li><li>Not available to prepay customers.</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w0" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form> <div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
<td style="text-align: center" class="hidden-xs">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
<td class="hidden-xs">
<form id="w1" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch">Switch Now!</button> </form> <div class="wannatalk">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
<tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Powershop Saver</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$183.40 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.2508</td><td>per kWh</td><tr><td>Daily</td><td>$0.30</td><td>per day</td><tr><td>EA Levy</td><td>$0.00</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$0.00 (0%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>The price estimate is based on forecast charges from Powershop for the next 12 months.</li><li>It assumes you purchase the Powershop Simple Saver powerpack once a month and special powerpacks that are made available from time to time.</li><li>This offer does not require a contract or a minimum supply period.</li><li>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, $10 on the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).�</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$183.40</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Get $150 off your bill over 12 months!<br /> <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="<div><div>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, then $10 for the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).</div><div>�</div></div><div><br></div><div><br></div>">More Info</a> </div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w2" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="53" /> <input type="hidden" name="plan_stage_id" value="273" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form><div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
so the output should be:
from td[3] and td[4] in first row:
Contact Energy Saver Plus
$179.71
and then the next row:
Powershop Saver
$183.40
and so on until the last row ( of the main table).
Similar process to that given in comments but different selectors
from bs4 import BeautifulSoup as bs
html = '''yourhtml'''
soup = bs(html, 'lxml')
names = [item.text for item in soup.select('.table h4 ')]
prices = [item.text for item in soup.select('[colspan="2"] > .total')]
results = list(zip(names, prices))
print(results)
I actually managed to solve this with using regex. I like the approach in the above answer much better specially using zip(), but I though pasting my solution here in case it becomes handy to some other readers.
deals=[]
prices=[]
results={}
with open("prices.html", "r") as f:
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
tbody = priceTable.find('tbody')
pplanPattern = '<td\ colspan="3"><h4>([^<]+)<\/h4><\/td>'
pricePatterns = '<h3 class="total">([^<]+)<\/h3>'
for rw in tbody:
plan = re.search(pplanPattern, rw)
price = re.search(pricePatterns, rw)
if plan:
deals.append(plan.group(1))
if price:
deals.append(price.group(1))
results[plan.group(1)] = price.group(1)
Currently trying to loop the following web scraping...
My current problem is that I can only get the first footballer from the table (I have the table HTML down below) and not the full 10 players, my immediate thoughts are that the loop isn't working and I'm unsure where I'm going wrong. I'm using the BeautifulSoup Method of gathering the data.
TD;DR My error is that only 1 player is appearing in my CSV file instead of the 10 players available from the HTML
Python Code
from urllib.request import urlopen as uReq
from urllib.request import Request
from bs4 import BeautifulSoup as soup
my_url = "https://www.fctables.com/teams/stoke-194901/"
#opening up connection , grabbing page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
#html parsing
page_soup = soup(page_html, "html.parser")
topScorers = page_soup.findAll("table",{"class":"table table-striped table-bordered table-hover stage-table table-condensed top_scores"})
filename = "stokeGoals.csv"
f = open(filename, "w")
headers = "player, goal_scored, average_goal"
f.write(headers)
for topScorer in topScorers:
#top 10 players who scored
player = topScorer.a["title"]
#top 10 goalscorers for the team
goalpp = topScorer.findAll("div", {"class": "progress"})
#average goal per game
avg = topScorer.findAll("div", {"class": "label label-primary"})
avgpp = avg[0].text.strip()
print("player: " + player)
print("goal_scored: " + goalpp)
print("AVG: "+ avgpp)
f.write(player + "," +goalpp.replace("," , "|")+ "," + avgpp +"\n")
f.close()
HTML Code for the table/website I'm scraping data from
<table class="table table-striped table-bordered table-hover stage-table table-condensed top_scores">
<thead>
<tr>
<th>#</th>
<th class="tl">Player</th>
<th data-toggle="tooltip" title="Goals scores by player / Goals scores by his team">goals</th>
<th data-toggle="tooltip" title="Average goals">
Avg
</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td class="tl psh" data-id="212996">
<img alt="Benik Afobe" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/s4/s4glg58a2350823d58/benik-afobe.png" width="20" /> Afobe
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 100%;">
<span class="goal_p">6</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>2</td>
<td class="tl psh" data-id="320050">
<img alt="Thomas Ince" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/t5/t5ni157c703a92110b/thomas-ince.jpg" width="20" /> Ince
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 83.333333333333%;">
<span class="goal_p">5</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.6</div>
</td>
</tr>
<tr>
<td>3</td>
<td class="tl psh" data-id="308648">
<img alt="Saido Berahino" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/po/poyhu58a234e0da106/saido-berahino.png" width="20" /> Berahino
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 66.666666666667%;">
<span class="goal_p">4</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>4</td>
<td class="tl psh" data-id="257340">
<img alt="Joe Allen" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/6w/6w45558a234deae78e/joe-allen.png" width="20" /> Allen
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
<span class="goal_p">3</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>5</td>
<td class="tl psh" data-id="234407">
<img alt="Erik Pieters" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/et/et08558a234dd63b68/erik-pieters.png" width="20" /> Pieters
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
<span class="goal_p">3</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>6</td>
<td class="tl psh" data-id="299368">
<img alt="Peter Crouch" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/qp/qptn558a234df86f1f/peter-crouch.png" width="20" /> Crouch
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
<span class="goal_p">2</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>7</td>
<td class="tl psh" data-id="214479">
<img alt="Bojan Krkic" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/pl/pleyv57eaedf0afeac/bojan-krkic.jpg" width="20" /> Krkic
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
<span class="goal_p">2</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>8</td>
<td class="tl psh" data-id="253114">
<img alt="James McClean" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/gb/gbjmm58a234f55a560/james-mcclean.png" width="20" /> McClean
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.1</div>
</td>
</tr>
<tr>
<td>9</td>
<td class="tl psh" data-id="309022">
<img alt="Sam Clucas" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/g7/g7dig58a234cb144a3/sam-clucas.png" width="20" /> Clucas
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>10</td>
<td class="tl psh" data-id="215724">
<img alt="Bruno Martins Indi" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/hk/hkung58a234de0dfaa/bruno-martins-indi.png" width="20" /> Indi
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.2</div>
</td>
</tr>
</tbody>
The webpage you specified, loads data via XMLHttpRequest
You can grab the html directly from:
https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10
Through the above url, you can get all the information you need without the extra html noise, i.e.:
my_url = "https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10"
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
_names = page_soup.findAll("img",{"class":"img-circle"})
_goals = page_soup.findAll("span",{"class":"goal_p"})
_avg = page_soup.findAll("div",{"class":"label label-primary"})
x = 0
for name in _names:
name = name['alt']
avg = _avg[x].get_text()
goals = _goals[x].get_text()
print(name, avg, goals)
x+=1
Benik Afobe 0.4 6
Thomas Ince 0.6 5
Saido Berahino 0.3 4
Joe Allen 0.4 3
Erik Pieters 0.4 3
Peter Crouch 0.3 2
Bojan Krkic 0.4 2
James McClean 0.1 1
Sam Clucas 0.3 1
Bruno Martins Indi 0.2 1
Note:
Adjust the url values as needed, you can change top_score, type, team_id, limit, etc...
Keeping the url as it is, you can try the following to fetch the required results:
import requests
from bs4 import BeautifulSoup
url = "https://www.fctables.com/teams/stoke-194901/"
res = requests.get(url)
soup = BeautifulSoup(res.text,"html.parser")
for items in soup.select(".top_scores tbody tr"):
name = items.select_one("td a[href^='/players/']").get("title")
goal = items.select_one("td .goal_p").text
avrg = items.select_one("td .label-primary").text
print(name, goal, avrg)
Output you should get:
Benik Afobe 6 0.4
Thomas Ince 5 0.6
Saido Berahino 4 0.3
Joe Allen 3 0.4
Erik Pieters 3 0.4
Peter Crouch 2 0.3
Bojan Krkic 2 0.4
James McClean 1 0.1
Sam Clucas 1 0.3
Bruno Martins Indi 1 0.2
I have created this for loop to find td items that start with 'td_threadtitle':
for item in posts:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
print(hello)
But I get this error:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
TypeError: slice indices must be integers or None or have an __index__ method
When I change the variable hello to this:
hello = item.find("td") , it works perfectly fine. Why does it throw that error when I try to specify the id?
EDIT:
This is how I created posts:
tableWithPosts = soup.find("body").find("div", attrs = {"align": "center"}).find("div", {"class" : "page"}).find("div", attrs = {"style" : "padding:0px 0px 0px 0px"}).find("center").find("form").find("table", {"id": "threadslist"})
posts = tableWithPosts.find("tbody", {"id": "threadbits_forum_75"}
Here is a portion of posts:
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=206824', '_self')" style="cursor:pointer">
thelavenhagen
</span>
</div>
</td>
<td class="alt2" title="Replies: 11, Views: 1,471">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
05:06:46 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=581132" rel="nofollow">
westopher
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660274#post1067660274">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=581132" onclick="who(581132); return false;">
11
</a>
</td>
<td align="center" class="alt2">
1,471
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_558556">
<img alt="" border="" id="thread_statusicon_558556" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_558556" title="1996 E36 M3 Lux Dakar Yellow, 87,800 miles, special order without sunroof. Second owner, owned...">
<div>
<span style="float:right">
<a href="#" onclick="attachments(558556); return false">
<img alt="4 Attachment(s)" border="0" class="inlineimg" src="images/misc/paperclip.gif"/>
</a>
</span>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556" id="thread_title_558556">
1996 E36 M3 - Dakar Lux Slicktop
</a>
<span class="smallfont" style="white-space:nowrap">
(
<img alt="Multi-page thread" border="0" class="inlineimg" src="images/misc/multipage.gif"/>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556">
1
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=2">
2
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=3">
3
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=95931', '_self')" style="cursor:pointer">
yellowbee
</span>
</div>
</td>
<td class="alt2" title="Replies: 23, Views: 5,147">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
04:04:07 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=558556" rel="nofollow">
mbausa
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660244#post1067660244">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=558556" onclick="who(558556); return false;">
23
</a>
</td>
<td align="center" class="alt2">
5,147
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_580693">
<img alt="" border="" id="thread_statusicon_580693" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_580693" title="Selling my wife's car. We have owned her for two years and have put over 20k hassle free miles on...">
<div>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=580693" id="thread_title_580693">
2011 BMW 740Li Alpine White M Package Dakota Brown Interior Weather-tech Mats
</a>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=128641', '_self')" style="cursor:pointer">
911-AL
</span>
</div>
</td>
Remove your for loop, try with this:
hello = posts.find_all("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
hello
It will find all td items that start with 'td_threadtitle'
hello will be a list which contains all td(objects <class 'bs4.element.Tag'> ) start with 'td_threadtitle', you can still access their div.
Can I filter-out all no data in id="time_xxxxxx" and id="odds_3_xxxxxx" from all tags which contain in <tr id="tr_xxxxxx">, and just display all has data in id="time_xxxxxx" and id="odds_3_xxxxxx"?
from this HTML code have one <tr> tag which contains id="tr_xxxxxxx
<tr> </tr>
in this tag contain two set of data
<td id="odds_3_xxxxxxx"></td>
<div id="time_xxxxx"> </div>
but I just want to get the tag which has completed set of the data which just first set of id="odds_3_xxxxxx" was ok.
In this html table, the data pattern are following:
first set (data complete-set)
second set (no time data)
third set (just has time data)
HTML:
<tr style="display: none;" id="tr_976952" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_1039509">
42
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_1039509" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(1039509,'3')">0.94</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(1039509,'3')">2.5/3</a>
<br>
0.86
</td>
<td width="90" id="odds_4_1234567" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(1039509,'3')">0.12</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(1039509,'3')">3</a>
<br>
0.41
</td>
</tr>
<tr style="display: none;" id="tr_654654" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_654654">
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_654654" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(654654,'3')">0.88</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(654654,'3')">1.5</a>
<br>
0.86
</td>
<td width="90" id="odds_4_1234567" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(654654,'3')">0.77</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(654654,'3')">2</a>
<br>
0.66
</td>
</tr>
<tr style="display: none;" id="tr_534589" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_534589">
50
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_534589" title=""></td>
<td width="90" id="odds_4_534589" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(534589,'3')">0.99</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(534589,'3')">6</a>
<br>
0.74
</td>
</tr>
Code: (for my code will display all data, but I just need to complete-set of data, if the data are not in complete-set, I don't want to display)
rows = table.findAll("tr", {"id" : re.compile('tr_*\d')})
cols = soup.find_all(["div", "td"], id=re.compile('^(odds_3|time)_\d+$'))
data = [t.strip()
for tag in cols
for t in tag.find_all(text=True) if t.strip()]
print '; '.join(data)
Output:
42; 0.94 ; 2.5/3 ; 0.86
0.88 ; 1.5 ; 0.86
50
Expected Output:
42; 0.94 ; 2.5/3 ; 0.86
Why don't you just filter what gets printed like this:
if len(data) == 4:
print '; '.join(data)