I have created this for loop to find td items that start with 'td_threadtitle':
for item in posts:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
print(hello)
But I get this error:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
TypeError: slice indices must be integers or None or have an __index__ method
When I change the variable hello to this:
hello = item.find("td") , it works perfectly fine. Why does it throw that error when I try to specify the id?
EDIT:
This is how I created posts:
tableWithPosts = soup.find("body").find("div", attrs = {"align": "center"}).find("div", {"class" : "page"}).find("div", attrs = {"style" : "padding:0px 0px 0px 0px"}).find("center").find("form").find("table", {"id": "threadslist"})
posts = tableWithPosts.find("tbody", {"id": "threadbits_forum_75"}
Here is a portion of posts:
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=206824', '_self')" style="cursor:pointer">
thelavenhagen
</span>
</div>
</td>
<td class="alt2" title="Replies: 11, Views: 1,471">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
05:06:46 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=581132" rel="nofollow">
westopher
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660274#post1067660274">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=581132" onclick="who(581132); return false;">
11
</a>
</td>
<td align="center" class="alt2">
1,471
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_558556">
<img alt="" border="" id="thread_statusicon_558556" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_558556" title="1996 E36 M3 Lux Dakar Yellow, 87,800 miles, special order without sunroof. Second owner, owned...">
<div>
<span style="float:right">
<a href="#" onclick="attachments(558556); return false">
<img alt="4 Attachment(s)" border="0" class="inlineimg" src="images/misc/paperclip.gif"/>
</a>
</span>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556" id="thread_title_558556">
1996 E36 M3 - Dakar Lux Slicktop
</a>
<span class="smallfont" style="white-space:nowrap">
(
<img alt="Multi-page thread" border="0" class="inlineimg" src="images/misc/multipage.gif"/>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556">
1
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=2">
2
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=3">
3
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=95931', '_self')" style="cursor:pointer">
yellowbee
</span>
</div>
</td>
<td class="alt2" title="Replies: 23, Views: 5,147">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
04:04:07 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=558556" rel="nofollow">
mbausa
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660244#post1067660244">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=558556" onclick="who(558556); return false;">
23
</a>
</td>
<td align="center" class="alt2">
5,147
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_580693">
<img alt="" border="" id="thread_statusicon_580693" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_580693" title="Selling my wife's car. We have owned her for two years and have put over 20k hassle free miles on...">
<div>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=580693" id="thread_title_580693">
2011 BMW 740Li Alpine White M Package Dakota Brown Interior Weather-tech Mats
</a>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=128641', '_self')" style="cursor:pointer">
911-AL
</span>
</div>
</td>
Remove your for loop, try with this:
hello = posts.find_all("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
hello
It will find all td items that start with 'td_threadtitle'
hello will be a list which contains all td(objects <class 'bs4.element.Tag'> ) start with 'td_threadtitle', you can still access their div.
Related
I have a piece of HTML code below:
<div class="user-tagline ">
<span class="username " data-avatar="aaaaaaa">player1</span>
<span class="user-rating">(1357)</span>
<span class="country-flag-small flag-113" tip="Portugal"></span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="bbbbbbb">player2</span>
<span class="user-rating">(1387)</span>
<span class="country-flag-small flag-70" tip="Indonesia"></span>
</div>
I want to extract "Portugal" from it, note the span class is a dynamic one, it is not always class="country-flag-small flag-113" but indeed changes per the value of country generated for this div block.
To get the player1 and 1357, I am using the following cumbersome code:
player1info = soup.findAll('div', attrs={'class':'user-tagline'})[0].text.split("\n")
player1 = player1info[1]
pscore1 = player1info[1].replace('(','').replace(')', '')
It would be appreciated if someone can share with your better solution here. Thank you in advance
UPDATE:
With the initial HTML div info extracted, now I would like to expand it to extract more for the entire row, here is the row:
<tr board-popover="" fen="r1bk2r1/1p2n3/pN6/1B1qQp2/P2Pp2p/1P6/2P2PPP/R3K1R1 b Q -" flip-board="1" highlight-squares="c4b6">
<td>
<a class="clickable-link td-user" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
<span class="time-control">
<i class="icon-rapid">
</i>
</span>
<div class="user-tagline ">
<span class="username " data-avatar="https://betacssjs.chesscomfiles.com/bundles/web/images/noavatar_l.1c5172d5.gif" data-country="Portugal" data-enabled="true" data-flag="113" data-joined="Joined Jun 19, 2016" data-logged="Online 6 hrs ago" data-membership="basic" data-name="Atikinounette" data-popup="hover" data-title="" data-username="Atikinounette">
Atikinounette
</span>
<span class="user-rating">
(1357)
</span>
<span class="country-flag-small flag-113" tip="Portugal">
</span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="https://images.chesscomfiles.com/uploads/v1/user/28196414.83e31ff1.50x50o.3a6f77e4aa44.jpeg" data-country="Indonesia" data-enabled="true" data-flag="70" data-joined="Joined May 15, 2016" data-logged="Online Nov 7, 2017" data-membership="basic" data-name="belemnarmada" data-popup="hover" data-title="" data-username="belemnarmada">
belemnarmada
</span>
<span class="user-rating">
(1387)
</span>
<span class="country-flag-small flag-70" tip="Indonesia">
</span>
</div>
</a>
</td>
<td>
<a class="clickable-link text-middle" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
<div class="pull-left">
<span class="game-result">
1
</span>
<span class="game-result">
0
</span>
</div>
<div class="result">
<i class="icon-square-minus loss" tip="Lost">
</i>
</div>
</a>
</td>
<td class="text-center">
<a class="clickable-link" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
30 min
</a>
</td>
<td class="text-right">
<a class="clickable-link text-middle moves" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
25
</a>
</td>
<td class="text-right miniboard">
<a class="clickable-link archive-date" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
Aug 9, 2017
</a>
</td>
<td class="text-center miniboard">
<input class="checkbox" game-checkbox="" game-id="2249663029" game-is-live="true" ng-model="model.gameIds[2249663029].checked" type="checkbox"/>
</td>
</tr>
Needed info are:
player's info (answer provided by #balderman already got that)
game-result (1, 0)
playing time (30 min in this row)
total moves (25)
playing date (Aug 9, 2017)
Thank you so much here.
How about the code below?
The idea that the user attributes are 3 spans under the div. So the code points to those spans and extract the data.
from bs4 import BeautifulSoup
html = '''<html><body> <div class="user-tagline ">
<span class="username " data-avatar="aaaaaaa">player1</span>
<span class="user-rating">(1357)</span>
<span class="country-flag-small flag-113" tip="Portugal"></span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="bbbbbbb">player2</span>
<span class="user-rating">(1387)</span>
<span class="country-flag-small flag-70" tip="Indonesia"></span>
</div><body></html>'''
soup = BeautifulSoup(html, 'html.parser')
users = soup.findAll('div', attrs={'class': 'user-tagline'})
for user in users:
user_properties = user.findAll('span')
for idx, prop in enumerate(user):
if idx == 1:
print('user name: {}'.format(prop.text))
elif idx == 3:
print('user rating: {}'.format(prop.text))
elif idx == 5:
print('user country: {}'.format(prop.attrs['tip']))
Output
user name: player1
user rating: (1357)
user country: Portugal
user name: player2
user rating: (1387)
user country: Indonesia
This is a more readable solution:
div1 = soup.select("div.user-tagline")[0]
player1 = div1.select_one("span.user-rating").text
pscore1 = div1.select_one("span.country-flag-small").text
To extract data of all divs, just use a loop. And replace "0" with "i".
If you are interested only in the first div, you can go with this:
res = bsobj.find('div', {'class':'user-tagline'}).findAll('span')
print(res[0].text, res[1].text, res[2]['tip'])
Currently trying to loop the following web scraping...
My current problem is that I can only get the first footballer from the table (I have the table HTML down below) and not the full 10 players, my immediate thoughts are that the loop isn't working and I'm unsure where I'm going wrong. I'm using the BeautifulSoup Method of gathering the data.
TD;DR My error is that only 1 player is appearing in my CSV file instead of the 10 players available from the HTML
Python Code
from urllib.request import urlopen as uReq
from urllib.request import Request
from bs4 import BeautifulSoup as soup
my_url = "https://www.fctables.com/teams/stoke-194901/"
#opening up connection , grabbing page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
#html parsing
page_soup = soup(page_html, "html.parser")
topScorers = page_soup.findAll("table",{"class":"table table-striped table-bordered table-hover stage-table table-condensed top_scores"})
filename = "stokeGoals.csv"
f = open(filename, "w")
headers = "player, goal_scored, average_goal"
f.write(headers)
for topScorer in topScorers:
#top 10 players who scored
player = topScorer.a["title"]
#top 10 goalscorers for the team
goalpp = topScorer.findAll("div", {"class": "progress"})
#average goal per game
avg = topScorer.findAll("div", {"class": "label label-primary"})
avgpp = avg[0].text.strip()
print("player: " + player)
print("goal_scored: " + goalpp)
print("AVG: "+ avgpp)
f.write(player + "," +goalpp.replace("," , "|")+ "," + avgpp +"\n")
f.close()
HTML Code for the table/website I'm scraping data from
<table class="table table-striped table-bordered table-hover stage-table table-condensed top_scores">
<thead>
<tr>
<th>#</th>
<th class="tl">Player</th>
<th data-toggle="tooltip" title="Goals scores by player / Goals scores by his team">goals</th>
<th data-toggle="tooltip" title="Average goals">
Avg
</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td class="tl psh" data-id="212996">
<img alt="Benik Afobe" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/s4/s4glg58a2350823d58/benik-afobe.png" width="20" /> Afobe
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 100%;">
<span class="goal_p">6</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>2</td>
<td class="tl psh" data-id="320050">
<img alt="Thomas Ince" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/t5/t5ni157c703a92110b/thomas-ince.jpg" width="20" /> Ince
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 83.333333333333%;">
<span class="goal_p">5</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.6</div>
</td>
</tr>
<tr>
<td>3</td>
<td class="tl psh" data-id="308648">
<img alt="Saido Berahino" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/po/poyhu58a234e0da106/saido-berahino.png" width="20" /> Berahino
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 66.666666666667%;">
<span class="goal_p">4</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>4</td>
<td class="tl psh" data-id="257340">
<img alt="Joe Allen" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/6w/6w45558a234deae78e/joe-allen.png" width="20" /> Allen
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
<span class="goal_p">3</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>5</td>
<td class="tl psh" data-id="234407">
<img alt="Erik Pieters" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/et/et08558a234dd63b68/erik-pieters.png" width="20" /> Pieters
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
<span class="goal_p">3</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>6</td>
<td class="tl psh" data-id="299368">
<img alt="Peter Crouch" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/qp/qptn558a234df86f1f/peter-crouch.png" width="20" /> Crouch
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
<span class="goal_p">2</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>7</td>
<td class="tl psh" data-id="214479">
<img alt="Bojan Krkic" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/pl/pleyv57eaedf0afeac/bojan-krkic.jpg" width="20" /> Krkic
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
<span class="goal_p">2</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.4</div>
</td>
</tr>
<tr>
<td>8</td>
<td class="tl psh" data-id="253114">
<img alt="James McClean" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/gb/gbjmm58a234f55a560/james-mcclean.png" width="20" /> McClean
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.1</div>
</td>
</tr>
<tr>
<td>9</td>
<td class="tl psh" data-id="309022">
<img alt="Sam Clucas" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/g7/g7dig58a234cb144a3/sam-clucas.png" width="20" /> Clucas
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.3</div>
</td>
</tr>
<tr>
<td>10</td>
<td class="tl psh" data-id="215724">
<img alt="Bruno Martins Indi" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/hk/hkung58a234de0dfaa/bruno-martins-indi.png" width="20" /> Indi
<div class="slider">
<div class="inner"></div>
</div>
</td>
<td width="30%">
<div class="progress">
<div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
<span class="goal_p">1</span>
</div>
</div>
</td>
<td>
<div class="label label-primary">0.2</div>
</td>
</tr>
</tbody>
The webpage you specified, loads data via XMLHttpRequest
You can grab the html directly from:
https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10
Through the above url, you can get all the information you need without the extra html noise, i.e.:
my_url = "https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10"
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
_names = page_soup.findAll("img",{"class":"img-circle"})
_goals = page_soup.findAll("span",{"class":"goal_p"})
_avg = page_soup.findAll("div",{"class":"label label-primary"})
x = 0
for name in _names:
name = name['alt']
avg = _avg[x].get_text()
goals = _goals[x].get_text()
print(name, avg, goals)
x+=1
Benik Afobe 0.4 6
Thomas Ince 0.6 5
Saido Berahino 0.3 4
Joe Allen 0.4 3
Erik Pieters 0.4 3
Peter Crouch 0.3 2
Bojan Krkic 0.4 2
James McClean 0.1 1
Sam Clucas 0.3 1
Bruno Martins Indi 0.2 1
Note:
Adjust the url values as needed, you can change top_score, type, team_id, limit, etc...
Keeping the url as it is, you can try the following to fetch the required results:
import requests
from bs4 import BeautifulSoup
url = "https://www.fctables.com/teams/stoke-194901/"
res = requests.get(url)
soup = BeautifulSoup(res.text,"html.parser")
for items in soup.select(".top_scores tbody tr"):
name = items.select_one("td a[href^='/players/']").get("title")
goal = items.select_one("td .goal_p").text
avrg = items.select_one("td .label-primary").text
print(name, goal, avrg)
Output you should get:
Benik Afobe 6 0.4
Thomas Ince 5 0.6
Saido Berahino 4 0.3
Joe Allen 3 0.4
Erik Pieters 3 0.4
Peter Crouch 2 0.3
Bojan Krkic 2 0.4
James McClean 1 0.1
Sam Clucas 1 0.3
Bruno Martins Indi 1 0.2
I've just parsed a web page via pandas:
r = requests.post("https://www.eigroup.co.uk/clients/auctions/fulldetails.aspx?auctionid=17999 ", params=payload)
parsed_page = pd.read_html(r.text, attrs={"class": "table-search-result"})
(an example of the HTML being parsed)
<table cellspacing="0" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1" style="width:100%;border-collapse:collapse;">
<tr>
<td colspan="2">
<table class="table-search-result">
<tr>
<th>66D Charlwood Street, Pimlico, London, SW1V 4PQ</th>
<th style="text-align: right; white-space: nowrap;">
<a href="http://www.englishhouseprices.com/results.aspx?postcode=SW1V 4PQ" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A2" class="icon" target="_blank">
<img src="/content/images/icons/32/houseprices.png" alt="Compare with Property Prices" title="Compare with Property Prices in this Postcode" /></a>
<a id="" title="View Auction Details" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank"><img title="View Auction Details" src="/content/images/icons/32/auctiondetails.png" alt="" /></a>
<a id="" title="Trend Analysis" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/lots/trend-analysis.aspx?lotid=756425" target="_blank"><img title="Trend Analysis" src="/content/images/icons/32/piechart.png" alt="" /></a>
<a href='http://maps.google.co.uk?q=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageLocationMap" title="Location Map" class="icon" src="/content/images/icons/32/compass.png" /></a>
<a href='http://www.multimap.com/map/photo.cgi?scale=5000&mapsize=big&pc=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageAerialPhoto" title="Aerial Photo" class="icon" src="/content/images/icons/32/camera.png" /></a>
<a href='/clients/search/search-results.aspx?searchtype=comparable&lotid=756425' title="Find similar properties like this one">
<img src="/content/images/icons/32/find.png" alt="Find other properties matching this tenant" title="Find similar properties like this one" class="icon" /></a>
<a href='/clients/search/search-results.aspx?searchtype=history&lotid=756425'>
<img src="/content/images/icons/32/history.png" alt="Find history of property in this street" title="Find history of property in this street" class="icon" /></a>
<a id="" title="Add to one of my portfolios" class="icon" Title="Add to portfolio" onclick="return o(this,650,500,1,1)" href="/clients/portfolios/lot.aspx?lotid=756425" target="_blank"><img title="Add to one of my portfolios" src="/content/images/icons/32/briefcase.png" alt="" /></a>
<a href="https://www.eigroup.co.uk/files/55/17999/6ec339ec-d59e-4b8a-9136-dc6e9a583328.pdf" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A4" target="_blank">
<img src="/content/images/icons/32/catalogue.png" alt="Catalogue Entry" class="icon" title="Full Catalogue Entry" /></a>
<a id="" title="Add to my shortlist" class="icon" Title="Add to shortlist" onclick="return o(this,900,650,1,1)" href="/clients/lots/shortlist.aspx?lotid=756425" target="shortlist"><img title="Add to my shortlist" src="/content/images/icons/32/shortlist.png" alt="" /></a>
</th>
</tr>
<tr>
<td colspan="2" style="background-color: #f5f5f5;">
<table style="width: 100%">
<tr>
<td style="background-color: #f1f1f1; width: 170px; text-align: center;">
<a href='/clients/lots/details.aspx?lotid=756425&hb=1' target='756425' onclick="window.open(this.href,this.target,'width=900,height=650,resizable=yes,scrollbars=yes');return false" title="Auction property in Pimlico, London, SW1">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_Image1" src="https://www.eigroup.co.uk/files/55/17999/de591a4f-7da1-4bcd-a42c-76731bd72a23.jpg" alt="Pimlico, London, SW1" style="border-color:Black;border-width:2px;border-style:Solid;width:150px;" />
</a>
</td>
<td style="padding-left: 10px; width: 50%;">
<p>
<b>Description</b><br />
Leasehold 2nd Floor Studio Flat Unmodernised Vacant
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P1">
<b>Guide Price</b><br />
£450,000 Plus
</p>
<p>
<b>Lot Number</b><br />
2
</p>
<p>
<b> </b>
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auctioneer</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctioneers/details.aspx?auctioneerid=55" target="_blank">Savills (London - National)</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P3">
<b>Vendor</b><br />
Housing Association
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auction Date</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank">28 October 2014</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P7">
<b>Lease Details</b><br />
125 Yr, commencing 01/01/2013 (GR.£250.PA)
</p>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
and I get the following:
In [86]: parsed_page[1][0][1]
Out[86]: u'Description Leasehold 2nd Floor Studio Flat Unmodernised Vacant Guide Price \xa3450,000 Plus Lot Number 2 Auctioneer Savills (London - National) Vendor Housing Association Auction Date 28 October 2014 Lease Details 125 Yr, commencing 01/01/2013 (GR.\xa3250.PA)'
The problem is, I want to be able to extract the description, guide price etc, but there aren't any delimiters and the number of characters afterwards is variable. Am I missing a keyword when I'm parsing?
How can I then split them into new columns?
Using beautifulSoup as I recommended in an answer to your last question, you can split the text and make a dict :
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
s = soup.find_all("p")
details = (ele.text.strip().split("\n") for ele in s)
d = {}
for det in details:
if len(det) == 2:
d[det[0].strip()] = det[1].strip()
{u'Vendor': u'Housing Association', u'Description': u'Leasehold 2nd Floor Studio Flat Unmodernised Vacant', u'Auction Date': u'28 October 2014', u'Auctioneer': u'Savills (London - National)', u'Lot Number': u'2', u'Guide Price': u'\xc2\u0141450,000 Plus', u'Lease Details': u'125 Yr, commencing 01/01/2013 (GR.\xc2\u0141250.PA)'}
Can I filter-out all no data in id="time_xxxxxx" and id="odds_3_xxxxxx" from all tags which contain in <tr id="tr_xxxxxx">, and just display all has data in id="time_xxxxxx" and id="odds_3_xxxxxx"?
from this HTML code have one <tr> tag which contains id="tr_xxxxxxx
<tr> </tr>
in this tag contain two set of data
<td id="odds_3_xxxxxxx"></td>
<div id="time_xxxxx"> </div>
but I just want to get the tag which has completed set of the data which just first set of id="odds_3_xxxxxx" was ok.
In this html table, the data pattern are following:
first set (data complete-set)
second set (no time data)
third set (just has time data)
HTML:
<tr style="display: none;" id="tr_976952" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_1039509">
42
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_1039509" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(1039509,'3')">0.94</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(1039509,'3')">2.5/3</a>
<br>
0.86
</td>
<td width="90" id="odds_4_1234567" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(1039509,'3')">0.12</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(1039509,'3')">3</a>
<br>
0.41
</td>
</tr>
<tr style="display: none;" id="tr_654654" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_654654">
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_654654" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(654654,'3')">0.88</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(654654,'3')">1.5</a>
<br>
0.86
</td>
<td width="90" id="odds_4_1234567" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(654654,'3')">0.77</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(654654,'3')">2</a>
<br>
0.66
</td>
</tr>
<tr style="display: none;" id="tr_534589" index="0" align="center" bgcolor="#F7F3F7" height="15">
<div id="time_534589">
50
<img src="images/in.gif" border="0">
</div>
<td width="90" id="odds_3_534589" title=""></td>
<td width="90" id="odds_4_534589" title="">
<a class="sb" href="javascript:" onclick="ChangeDetail3(534589,'3')">0.99</a>
<img src="images/t3.gif">
<br>
<a class="pk" href="javascript:" onclick="ChangeDetail3(534589,'3')">6</a>
<br>
0.74
</td>
</tr>
Code: (for my code will display all data, but I just need to complete-set of data, if the data are not in complete-set, I don't want to display)
rows = table.findAll("tr", {"id" : re.compile('tr_*\d')})
cols = soup.find_all(["div", "td"], id=re.compile('^(odds_3|time)_\d+$'))
data = [t.strip()
for tag in cols
for t in tag.find_all(text=True) if t.strip()]
print '; '.join(data)
Output:
42; 0.94 ; 2.5/3 ; 0.86
0.88 ; 1.5 ; 0.86
50
Expected Output:
42; 0.94 ; 2.5/3 ; 0.86
Why don't you just filter what gets printed like this:
if len(data) == 4:
print '; '.join(data)
I really couldn't think of a decent title to give a overview of what I'm trying to do, but the examples I have should explain it nicely, my company provides a schedule online, but they don't have any APIs or anything to extract it, so I'm using the Python framework Scrapy to scrape the data, and then adding it to my Google Calendar
A girl gave me a Regex line to handle the data because it was kicking my butt for days and she was feeling nice, but I've since realized that it doesn't handle split shifts (most likely because I was not scheduled for any so she didn't see the possibility of one)
My regex is
re.findall("""dow1'>(\w+)<\S+?>(\w+ \d+)</td>\s*<td class.*?tlHours'>(\d+).*?span>\s*(\d+)<span.*?ment'>(.*?)</spa.*?Meal: (.*?)</sp.*?start'>(\S+?)</spa.*?end'>(\S+?)<""", response.body)
Example data:
This is a normal 8 hour day with a meal break, which is handled fine:
<tr>
<td class='dt'>
<span class='dow1'>Sunday</span>Dec 09
</td>
<td class='ScheduledDetails'valign='top'>
<div style="position:relative;">
<span class='tlHours'>8<span class='spart'> hrs</span> 0<span class='spart'> mins</span></span><span class='department'>Cashier</span><span class='meal'>Meal: 2pm - 3pm</span>
</div>
</td>
<td>
</td>
<td class='Schedunderlay'>
<div class='Sched'>
<div class='schedbar' style='left: 143px; width: 234px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 226px;'>
<span class='start'>10am</span><span class='end'>7pm</span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 9px; width: 498px; display: none;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 490px;'>
<span class='start'><img src='/Images/Schedule/arrowLeft.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'></span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 508px; width: 216px; display: none;'>
<div class='schedbar_l_on'></div>
<div class='schedbar_m_on' style='width: 208px;'><span class='start'></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'><img src='/Images/Schedule/arrowRight.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
</div>
<div class='schedbar_r_on'></div>
</div>
</div>
</td>
<td> </td>
<td class='rightColDetails'>
<div class='AvailDetails' align='left' style='display: table-cell;'>
<span class='iefix'><b>Avail - All Day</b></span><br/>
<span style='font-size: 11px;'>Pref - All Day</span>
</div>
</td>
</tr>
And this is a split shift, two four hour shifts separated by a empty 1 hour slot (they do this to cheat the scoring system, two covered shifts instead of one):
<tr>
<td class='dt'>
<span class='dow1'>Thursday</span>Dec 13
</td>
<td class='ScheduledDetails' valign='top'>
<div style="position:relative;">
<span class='tlHours'>8<span class='spart'> hrs</span> 0<span class='spart'> mins</span></span><span class='department'>Cashier</span><span class='meal'>Meal: None</span>
</div>
</td>
<td> </td>
<td class='Schedunderlay'>
<div class='Sched'>
<div class='schedbar' style='left: 247px; width: 104px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 96px;'>
<span class='start'>2pm</span><span class='end'>6pm</span>
</div><div class='schedbar_r'></div>
</div>
<div class='schedbar' style='left: 377px; width: 104px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 96px;'>
<span class='start'>7pm</span> <span class='end'>11pm</span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 9px; width: 498px; display: none;'>
<div class='schedbar_l'></div><div class='schedbar_m' style='width: 490px;'>
<span class='start'><img src='/Images/Schedule/arrowLeft.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'></span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 508px; width: 216px; display: none;'>
<div class='schedbar_l_on'></div>
<div class='schedbar_m_on' style='width: 208px;'>
<span class='start'></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'><img src='/Images/Schedule/arrowRight.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
</div>
<div class='schedbar_r_on'></div>
</div>
</div>
</td>
<td> </td>
<td class='rightColDetails'>
<div class='AvailDetails' align='left' style='display: table-cell;'>
<span class='iefix'><b>Avail - All Day</b></span><br/><span style='font-size: 11px;'>Pref - All Day</span>
</div>
</td>
</tr>
The important difference is on the regular shift there's one start and one end time, with the split shift there's a start, and end, and start, and end....
I've been pounding my head against this for about five hours now... and making no headway, I suppose I'd have more luck if I understood Regex.. any help at all would be greatly appreciated...
Here is a solution using BeautifulSoup to parse the document and grab the info.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
for schedbar in soup.find_all('div', 'schedbar'):
print "start: " + schedbar.find('div', 'schedbar_m').find('span', 'start').string
print "end: " + schedbar.find('div', 'schedbar_m').find('span', 'end').string
Outputs:
start: 2pm
end: 6pm
start: 7pm
end: 11pm