The below HTML code has dynamic attributes for different individual series. Example, one series can have multiple units, like Million or Thousands.
<tr class="series-pager-title">
<td valign="top" colspan="2">
<div class="col-xs-12 col-sm-10">
Total Vehicle Sales
</div>
<div class="hidden-xs col-sm-2">
<span style="padding-left:49px;" class="popularity_bar"> </span> <span class="popularity_bar_background"> </span>
</div>
</td>
</tr>
<tr class="series-pager-attr">
<td colspan="2">
<div class="series-meta series-group-meta">
<span class="attributes">Monthly</span>
<br class="clear">
</div>
<div class="series-meta">
<input class="pager-item-checkbox" type="checkbox" name="sids[0]" value="TOTALSA">
<a href="/series/TOTALSA">
Millions of Units,
Seasonally Adjusted Annual Rate
</a>
<span class="series-meta-dates">
Jan 1976
to
Jul 2017
(4 days ago)
</span>
<br class="clear">
<input class="pager-item-checkbox" type="checkbox" name="sids[1]" value="TOTALNSA">
<a href="/series/TOTALNSA">
Thousands of Units,
Not Seasonally Adjusted
</a>
<span class="series-meta-dates">
Jan 1976
to
Jul 2017
(4 days ago)
</span>
</div>
</td>
</tr>
<tr><td colspan="2" style="font-size:9px"> </td></tr>
<tr class="series-pager-title">
<td valign="top" colspan="2">
<div class="col-xs-12 col-sm-10">
Light Weight Vehicle Sales: Autos and Light Trucks
</div>
<div class="hidden-xs col-sm-2">
<span style="padding-left:46px;" class="popularity_bar"> </span> <span class="popularity_bar_background"> </span>
</div>
</td>
</tr>
<tr class="series-pager-attr">
<td colspan="2">
<div class="series-meta series-group-single">
<input class="pager-item-checkbox" type="checkbox" name="sids[2]" value="ALTSALES">
<span class="attributes" style="width:350px;">Millions of Units, Monthly, Seasonally Adjusted Annual Rate</span><span class="series-meta-dates">Jan 1976 to Jul 2017 (4 days ago)</span>
<br class="clear">
</div>
<a href="/series/ALTSALES">
</a>
</td>
This gets me somewhat close, however it fails to obtain the 2nd frequency for the "Total Vehicle Sales," it only obtains the first "Millions of Units, Seasonally Adjusted Annual Rate." Aside from this issue, my assumption is that I would be mis-classifying things in general with my current query. Code I have created thus far:
browser=webdriver.Chrome(executable_path='F:\Anaconda\chromedriver\chromedriver_win32\chromedriver.exe')
browser.get('https://fred.stlouisfed.org/categories/32993')
soup=BeautifulSoup(browser.page_source,'lxml')
for l in soup.find_all('tbody'):
series_count=len(l.find_all('tr',attrs={'class':'series-pager-title'}))
series_data=l.find_all('tr',attrs={'class':'series-pager-title'})
attrs_data=l.find_all('tr',attrs={'class':'series-pager-attr'})
print(series_count)
print(len(attrs_data))
for m in range(0,series_count):
print(series_data[m].find('a',href=True).text+' | '+attrs_data[m].find('a',href=True).text.strip().replace(' ',' '))
In the above query, can someone please assist in creating the desired outcome:
If someone comes across this with a better solution I am all ears... In the interim, this seems to do the trick...
browser.get('https://fred.stlouisfed.org/categories/32993')
soup=BeautifulSoup(browser.page_source,'lxml')
test=soup.tbody
children=[child for child in test if child != '\n']
series_data=pd.DataFrame([],columns=['series_index','series_title','series_href'])
sub_series_data=pd.DataFrame([],columns=['series_index','frequency','sub_series_units','sub_series_href'])
series_index=0
for index,child in enumerate(children):
if child.find('a',attrs={'class':'series-title'}):
series_index+=1
series_title=child.text.strip()
series_link=child.find('a',href=True).attrs['href']
temp_series_df=({'series_index':series_index,
'series_title':series_title,
'series_href':series_link})
series_data=series_data.append([temp_series_df],ignore_index=True)
if child.find('div',attrs={'class':'series-meta'}):
frequency=child.find('span',attrs={'class':'attributes'}).text.strip()
for i in child.find_all('a',href=True):
temp_sub_series_df=({'series_index':series_index,
'frequency':frequency.strip(),
#'sub_series_units':i.text.strip(),
'sub_series_units':re.sub(' +',' ',re.sub('\n',' ',i.text)),
'sub_series_href':'https://fred.stlouisfed.org'+i.attrs['href']})
sub_series_data=sub_series_data.append([temp_sub_series_df],ignore_index=True)
print(series_data)
print(sub_series_data)
combine_series_data=pd.merge(series_data,sub_series_data,how='left',on=['series_index'])
Related
I need to get the name and the price of each row in the sample html below, however when I'm using beatifulsoup to find_all('tr') it returns all the tr of the main table and the nested tables. what is the best way for extracting only the value and the price of each row?
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
Above is what I have and it returns "all" the tr including the nested tables.
What I need is to get all the names and the price of each item in front of it, and finally save them in a csv file
<table class="table table-hover table-responsive">
<tbody><tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Contact Energy Saver Plus</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$242.85 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.334</td><td>per kWh</td><tr><td>Daily</td><td>$0.333</td><td>per day</td><tr><td>EA Levy</td><td>$0.0013</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$63.14 (26%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>Provides fixed pricing until 31 June 2021 unless there are changes to taxes and levies.</li><li>24% Prompt Payment Discount when you pay on time. additional 1% discount for paying by direct debit (excl. credit card), and 1% discount for getting bills and correspondence by email. Up to 26% PPD available.</li><li>An early termination fee of $150 per contracted ICP if you terminate the contract before the end date�(31/06/2021). Fee may be waived if you are moving house and take Contact Energy to the new property.</li><li>Not available to prepay customers.</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w0" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form> <div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
<td style="text-align: center" class="hidden-xs">
<h3 class="total">$179.71</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Special PPD & Fixed rates<br />
<a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
</div>
</td>
<td class="hidden-xs">
<form id="w1" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="54" /> <input type="hidden" name="plan_stage_id" value="367" /> <button type="submit" class="btn btn-block btn-switch">Switch Now!</button> </form> <div class="wannatalk">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
<tr>
<td style="vertical-align: middle; width: 20%;" class="hidden-xs">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
<td style="vertical-align: middle; width: 75px;" class="hidden-xs">
<img src="/images/result-arrow.png" />
</td>
<td>
<table style="width: 100%;">
<tbody><tr class="visible-xs">
<td class="text-center" colspan="2">
<img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
</td>
</tr>
<tr>
<td colspan="3"><h4>Powershop Saver</h4></td>
</tr>
<tr style="text-transform: uppercase">
<td width="150px">Electricity:</td>
<td>$183.40 <a class="plan-breakdown" data-placement="right" title="" data-original-title="<table><tr><td>Anytime</td><td>$0.2508</td><td>per kWh</td><tr><td>Daily</td><td>$0.30</td><td>per day</td><tr><td>EA Levy</td><td>$0.00</td><td>per kWh</td></table>"><i class="glyphicon glyphicon-info-sign"> </i></a>
</td>
</tr>
<tr style="text-transform: uppercase">
<td>Discount:</td>
<td>$0.00 (0%)
</td>
</tr>
<tr>
<td colspan="3">
<a class="plan-detail" data-placement="right" title="" data-original-title="<ul><li>The price estimate is based on forecast charges from Powershop for the next 12 months.</li><li>It assumes you purchase the Powershop Simple Saver powerpack once a month and special powerpacks that are made available from time to time.</li><li>This offer does not require a contract or a minimum supply period.</li><li>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, $10 on the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).�</li></ul>"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$183.40</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Get $150 off your bill over 12 months!<br /> <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="<div><div>New customers will get a $150 power credit applied over their first 12 months ($25 straight away, then $10 for the next 10�monthly account
review periods, and a final credit of $25 in the final account review period of
your first year as a Powershop customer).</div><div>�</div></div><div><br></div><div><br></div>">More Info</a> </div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<form id="w2" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" /> <input type="hidden" name="query_id" value="409884" /> <input type="hidden" name="plan_group_id" value="53" /> <input type="hidden" name="plan_stage_id" value="273" /> <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button> </form><div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>
so the output should be:
from td[3] and td[4] in first row:
Contact Energy Saver Plus
$179.71
and then the next row:
Powershop Saver
$183.40
and so on until the last row ( of the main table).
Similar process to that given in comments but different selectors
from bs4 import BeautifulSoup as bs
html = '''yourhtml'''
soup = bs(html, 'lxml')
names = [item.text for item in soup.select('.table h4 ')]
prices = [item.text for item in soup.select('[colspan="2"] > .total')]
results = list(zip(names, prices))
print(results)
I actually managed to solve this with using regex. I like the approach in the above answer much better specially using zip(), but I though pasting my solution here in case it becomes handy to some other readers.
deals=[]
prices=[]
results={}
with open("prices.html", "r") as f:
soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
tbody = priceTable.find('tbody')
pplanPattern = '<td\ colspan="3"><h4>([^<]+)<\/h4><\/td>'
pricePatterns = '<h3 class="total">([^<]+)<\/h3>'
for rw in tbody:
plan = re.search(pplanPattern, rw)
price = re.search(pricePatterns, rw)
if plan:
deals.append(plan.group(1))
if price:
deals.append(price.group(1))
results[plan.group(1)] = price.group(1)
I have a piece of HTML code below:
<div class="user-tagline ">
<span class="username " data-avatar="aaaaaaa">player1</span>
<span class="user-rating">(1357)</span>
<span class="country-flag-small flag-113" tip="Portugal"></span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="bbbbbbb">player2</span>
<span class="user-rating">(1387)</span>
<span class="country-flag-small flag-70" tip="Indonesia"></span>
</div>
I want to extract "Portugal" from it, note the span class is a dynamic one, it is not always class="country-flag-small flag-113" but indeed changes per the value of country generated for this div block.
To get the player1 and 1357, I am using the following cumbersome code:
player1info = soup.findAll('div', attrs={'class':'user-tagline'})[0].text.split("\n")
player1 = player1info[1]
pscore1 = player1info[1].replace('(','').replace(')', '')
It would be appreciated if someone can share with your better solution here. Thank you in advance
UPDATE:
With the initial HTML div info extracted, now I would like to expand it to extract more for the entire row, here is the row:
<tr board-popover="" fen="r1bk2r1/1p2n3/pN6/1B1qQp2/P2Pp2p/1P6/2P2PPP/R3K1R1 b Q -" flip-board="1" highlight-squares="c4b6">
<td>
<a class="clickable-link td-user" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
<span class="time-control">
<i class="icon-rapid">
</i>
</span>
<div class="user-tagline ">
<span class="username " data-avatar="https://betacssjs.chesscomfiles.com/bundles/web/images/noavatar_l.1c5172d5.gif" data-country="Portugal" data-enabled="true" data-flag="113" data-joined="Joined Jun 19, 2016" data-logged="Online 6 hrs ago" data-membership="basic" data-name="Atikinounette" data-popup="hover" data-title="" data-username="Atikinounette">
Atikinounette
</span>
<span class="user-rating">
(1357)
</span>
<span class="country-flag-small flag-113" tip="Portugal">
</span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="https://images.chesscomfiles.com/uploads/v1/user/28196414.83e31ff1.50x50o.3a6f77e4aa44.jpeg" data-country="Indonesia" data-enabled="true" data-flag="70" data-joined="Joined May 15, 2016" data-logged="Online Nov 7, 2017" data-membership="basic" data-name="belemnarmada" data-popup="hover" data-title="" data-username="belemnarmada">
belemnarmada
</span>
<span class="user-rating">
(1387)
</span>
<span class="country-flag-small flag-70" tip="Indonesia">
</span>
</div>
</a>
</td>
<td>
<a class="clickable-link text-middle" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
<div class="pull-left">
<span class="game-result">
1
</span>
<span class="game-result">
0
</span>
</div>
<div class="result">
<i class="icon-square-minus loss" tip="Lost">
</i>
</div>
</a>
</td>
<td class="text-center">
<a class="clickable-link" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
30 min
</a>
</td>
<td class="text-right">
<a class="clickable-link text-middle moves" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
25
</a>
</td>
<td class="text-right miniboard">
<a class="clickable-link archive-date" href="https://www.chess.com/live/game/2249663029?username=belemnarmada" target="_self">
Aug 9, 2017
</a>
</td>
<td class="text-center miniboard">
<input class="checkbox" game-checkbox="" game-id="2249663029" game-is-live="true" ng-model="model.gameIds[2249663029].checked" type="checkbox"/>
</td>
</tr>
Needed info are:
player's info (answer provided by #balderman already got that)
game-result (1, 0)
playing time (30 min in this row)
total moves (25)
playing date (Aug 9, 2017)
Thank you so much here.
How about the code below?
The idea that the user attributes are 3 spans under the div. So the code points to those spans and extract the data.
from bs4 import BeautifulSoup
html = '''<html><body> <div class="user-tagline ">
<span class="username " data-avatar="aaaaaaa">player1</span>
<span class="user-rating">(1357)</span>
<span class="country-flag-small flag-113" tip="Portugal"></span>
</div>
<div class="user-tagline ">
<span class="username " data-avatar="bbbbbbb">player2</span>
<span class="user-rating">(1387)</span>
<span class="country-flag-small flag-70" tip="Indonesia"></span>
</div><body></html>'''
soup = BeautifulSoup(html, 'html.parser')
users = soup.findAll('div', attrs={'class': 'user-tagline'})
for user in users:
user_properties = user.findAll('span')
for idx, prop in enumerate(user):
if idx == 1:
print('user name: {}'.format(prop.text))
elif idx == 3:
print('user rating: {}'.format(prop.text))
elif idx == 5:
print('user country: {}'.format(prop.attrs['tip']))
Output
user name: player1
user rating: (1357)
user country: Portugal
user name: player2
user rating: (1387)
user country: Indonesia
This is a more readable solution:
div1 = soup.select("div.user-tagline")[0]
player1 = div1.select_one("span.user-rating").text
pscore1 = div1.select_one("span.country-flag-small").text
To extract data of all divs, just use a loop. And replace "0" with "i".
If you are interested only in the first div, you can go with this:
res = bsobj.find('div', {'class':'user-tagline'}).findAll('span')
print(res[0].text, res[1].text, res[2]['tip'])
I have created this for loop to find td items that start with 'td_threadtitle':
for item in posts:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
print(hello)
But I get this error:
hello = item.find("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
TypeError: slice indices must be integers or None or have an __index__ method
When I change the variable hello to this:
hello = item.find("td") , it works perfectly fine. Why does it throw that error when I try to specify the id?
EDIT:
This is how I created posts:
tableWithPosts = soup.find("body").find("div", attrs = {"align": "center"}).find("div", {"class" : "page"}).find("div", attrs = {"style" : "padding:0px 0px 0px 0px"}).find("center").find("form").find("table", {"id": "threadslist"})
posts = tableWithPosts.find("tbody", {"id": "threadbits_forum_75"}
Here is a portion of posts:
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=206824', '_self')" style="cursor:pointer">
thelavenhagen
</span>
</div>
</td>
<td class="alt2" title="Replies: 11, Views: 1,471">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
05:06:46 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=581132" rel="nofollow">
westopher
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660274#post1067660274">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=581132" onclick="who(581132); return false;">
11
</a>
</td>
<td align="center" class="alt2">
1,471
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_558556">
<img alt="" border="" id="thread_statusicon_558556" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_558556" title="1996 E36 M3 Lux Dakar Yellow, 87,800 miles, special order without sunroof. Second owner, owned...">
<div>
<span style="float:right">
<a href="#" onclick="attachments(558556); return false">
<img alt="4 Attachment(s)" border="0" class="inlineimg" src="images/misc/paperclip.gif"/>
</a>
</span>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556" id="thread_title_558556">
1996 E36 M3 - Dakar Lux Slicktop
</a>
<span class="smallfont" style="white-space:nowrap">
(
<img alt="Multi-page thread" border="0" class="inlineimg" src="images/misc/multipage.gif"/>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556">
1
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=2">
2
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=558556&page=3">
3
</a>
)
</span>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=95931', '_self')" style="cursor:pointer">
yellowbee
</span>
</div>
</td>
<td class="alt2" title="Replies: 23, Views: 5,147">
<div class="smallfont" style="text-align:right; white-space:nowrap">
Thu, May-25-2017
<span class="time">
04:04:07 AM
</span>
<br/>
by
<a href="member.php?s=625e629b088a68126ca2d867c056b363&find=lastposter&t=558556" rel="nofollow">
mbausa
</a>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&p=1067660244#post1067660244">
<img alt="Go to last post" border="0" class="inlineimg" src="images/buttons/lastpost.gif"/>
</a>
</div>
</td>
<td align="center" class="alt1">
<a href="misc.php?do=whoposted&t=558556" onclick="who(558556); return false;">
23
</a>
</td>
<td align="center" class="alt2">
5,147
</td>
</tr>
<tr>
<td class="alt1" id="td_threadstatusicon_580693">
<img alt="" border="" id="thread_statusicon_580693" src="images/statusicon/thread_hot.gif"/>
</td>
<td class="alt2">
<img alt="" border="0" src="images/icons/icon1.gif"/>
</td>
<td class="alt1" id="td_threadtitle_580693" title="Selling my wife's car. We have owned her for two years and have put over 20k hassle free miles on...">
<div>
<span style="color: blue">
<b>
<u>
FS:
</u>
</b>
</span>
<a href="showthread.php?s=625e629b088a68126ca2d867c056b363&t=580693" id="thread_title_580693">
2011 BMW 740Li Alpine White M Package Dakota Brown Interior Weather-tech Mats
</a>
</div>
<div class="smallfont">
<span onclick="window.open('member.php?s=625e629b088a68126ca2d867c056b363&u=128641', '_self')" style="cursor:pointer">
911-AL
</span>
</div>
</td>
Remove your for loop, try with this:
hello = posts.find_all("td", {"id": lambda L: L and L.startswith('td_threadtitle')})
hello
It will find all td items that start with 'td_threadtitle'
hello will be a list which contains all td(objects <class 'bs4.element.Tag'> ) start with 'td_threadtitle', you can still access their div.
I've just parsed a web page via pandas:
r = requests.post("https://www.eigroup.co.uk/clients/auctions/fulldetails.aspx?auctionid=17999 ", params=payload)
parsed_page = pd.read_html(r.text, attrs={"class": "table-search-result"})
(an example of the HTML being parsed)
<table cellspacing="0" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1" style="width:100%;border-collapse:collapse;">
<tr>
<td colspan="2">
<table class="table-search-result">
<tr>
<th>66D Charlwood Street, Pimlico, London, SW1V 4PQ</th>
<th style="text-align: right; white-space: nowrap;">
<a href="http://www.englishhouseprices.com/results.aspx?postcode=SW1V 4PQ" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A2" class="icon" target="_blank">
<img src="/content/images/icons/32/houseprices.png" alt="Compare with Property Prices" title="Compare with Property Prices in this Postcode" /></a>
<a id="" title="View Auction Details" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank"><img title="View Auction Details" src="/content/images/icons/32/auctiondetails.png" alt="" /></a>
<a id="" title="Trend Analysis" class="icon" onclick="return o(this,900,650,1,1)" href="/clients/lots/trend-analysis.aspx?lotid=756425" target="_blank"><img title="Trend Analysis" src="/content/images/icons/32/piechart.png" alt="" /></a>
<a href='http://maps.google.co.uk?q=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageLocationMap" title="Location Map" class="icon" src="/content/images/icons/32/compass.png" /></a>
<a href='http://www.multimap.com/map/photo.cgi?scale=5000&mapsize=big&pc=SW1V 4PQ' target="_blank">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_ImageAerialPhoto" title="Aerial Photo" class="icon" src="/content/images/icons/32/camera.png" /></a>
<a href='/clients/search/search-results.aspx?searchtype=comparable&lotid=756425' title="Find similar properties like this one">
<img src="/content/images/icons/32/find.png" alt="Find other properties matching this tenant" title="Find similar properties like this one" class="icon" /></a>
<a href='/clients/search/search-results.aspx?searchtype=history&lotid=756425'>
<img src="/content/images/icons/32/history.png" alt="Find history of property in this street" title="Find history of property in this street" class="icon" /></a>
<a id="" title="Add to one of my portfolios" class="icon" Title="Add to portfolio" onclick="return o(this,650,500,1,1)" href="/clients/portfolios/lot.aspx?lotid=756425" target="_blank"><img title="Add to one of my portfolios" src="/content/images/icons/32/briefcase.png" alt="" /></a>
<a href="https://www.eigroup.co.uk/files/55/17999/6ec339ec-d59e-4b8a-9136-dc6e9a583328.pdf" id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_A4" target="_blank">
<img src="/content/images/icons/32/catalogue.png" alt="Catalogue Entry" class="icon" title="Full Catalogue Entry" /></a>
<a id="" title="Add to my shortlist" class="icon" Title="Add to shortlist" onclick="return o(this,900,650,1,1)" href="/clients/lots/shortlist.aspx?lotid=756425" target="shortlist"><img title="Add to my shortlist" src="/content/images/icons/32/shortlist.png" alt="" /></a>
</th>
</tr>
<tr>
<td colspan="2" style="background-color: #f5f5f5;">
<table style="width: 100%">
<tr>
<td style="background-color: #f1f1f1; width: 170px; text-align: center;">
<a href='/clients/lots/details.aspx?lotid=756425&hb=1' target='756425' onclick="window.open(this.href,this.target,'width=900,height=650,resizable=yes,scrollbars=yes');return false" title="Auction property in Pimlico, London, SW1">
<img id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_Image1" src="https://www.eigroup.co.uk/files/55/17999/de591a4f-7da1-4bcd-a42c-76731bd72a23.jpg" alt="Pimlico, London, SW1" style="border-color:Black;border-width:2px;border-style:Solid;width:150px;" />
</a>
</td>
<td style="padding-left: 10px; width: 50%;">
<p>
<b>Description</b><br />
Leasehold 2nd Floor Studio Flat Unmodernised Vacant
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P1">
<b>Guide Price</b><br />
£450,000 Plus
</p>
<p>
<b>Lot Number</b><br />
2
</p>
<p>
<b> </b>
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auctioneer</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctioneers/details.aspx?auctioneerid=55" target="_blank">Savills (London - National)</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P3">
<b>Vendor</b><br />
Housing Association
</p>
</td>
<td style="white-space: nowrap;">
<p>
<b>Auction Date</b><br />
<a id="" onclick="return o(this,900,650,1,1)" href="/clients/auctions/details.aspx?auctionid=17999" target="_blank">28 October 2014</a>
</p>
<p id="ListViewLots_ClientPropertyControl1_1_FormViewLot_1_P7">
<b>Lease Details</b><br />
125 Yr, commencing 01/01/2013 (GR.£250.PA)
</p>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
and I get the following:
In [86]: parsed_page[1][0][1]
Out[86]: u'Description Leasehold 2nd Floor Studio Flat Unmodernised Vacant Guide Price \xa3450,000 Plus Lot Number 2 Auctioneer Savills (London - National) Vendor Housing Association Auction Date 28 October 2014 Lease Details 125 Yr, commencing 01/01/2013 (GR.\xa3250.PA)'
The problem is, I want to be able to extract the description, guide price etc, but there aren't any delimiters and the number of characters afterwards is variable. Am I missing a keyword when I'm parsing?
How can I then split them into new columns?
Using beautifulSoup as I recommended in an answer to your last question, you can split the text and make a dict :
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
s = soup.find_all("p")
details = (ele.text.strip().split("\n") for ele in s)
d = {}
for det in details:
if len(det) == 2:
d[det[0].strip()] = det[1].strip()
{u'Vendor': u'Housing Association', u'Description': u'Leasehold 2nd Floor Studio Flat Unmodernised Vacant', u'Auction Date': u'28 October 2014', u'Auctioneer': u'Savills (London - National)', u'Lot Number': u'2', u'Guide Price': u'\xc2\u0141450,000 Plus', u'Lease Details': u'125 Yr, commencing 01/01/2013 (GR.\xc2\u0141250.PA)'}
I really couldn't think of a decent title to give a overview of what I'm trying to do, but the examples I have should explain it nicely, my company provides a schedule online, but they don't have any APIs or anything to extract it, so I'm using the Python framework Scrapy to scrape the data, and then adding it to my Google Calendar
A girl gave me a Regex line to handle the data because it was kicking my butt for days and she was feeling nice, but I've since realized that it doesn't handle split shifts (most likely because I was not scheduled for any so she didn't see the possibility of one)
My regex is
re.findall("""dow1'>(\w+)<\S+?>(\w+ \d+)</td>\s*<td class.*?tlHours'>(\d+).*?span>\s*(\d+)<span.*?ment'>(.*?)</spa.*?Meal: (.*?)</sp.*?start'>(\S+?)</spa.*?end'>(\S+?)<""", response.body)
Example data:
This is a normal 8 hour day with a meal break, which is handled fine:
<tr>
<td class='dt'>
<span class='dow1'>Sunday</span>Dec 09
</td>
<td class='ScheduledDetails'valign='top'>
<div style="position:relative;">
<span class='tlHours'>8<span class='spart'> hrs</span> 0<span class='spart'> mins</span></span><span class='department'>Cashier</span><span class='meal'>Meal: 2pm - 3pm</span>
</div>
</td>
<td>
</td>
<td class='Schedunderlay'>
<div class='Sched'>
<div class='schedbar' style='left: 143px; width: 234px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 226px;'>
<span class='start'>10am</span><span class='end'>7pm</span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 9px; width: 498px; display: none;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 490px;'>
<span class='start'><img src='/Images/Schedule/arrowLeft.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'></span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 508px; width: 216px; display: none;'>
<div class='schedbar_l_on'></div>
<div class='schedbar_m_on' style='width: 208px;'><span class='start'></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'><img src='/Images/Schedule/arrowRight.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
</div>
<div class='schedbar_r_on'></div>
</div>
</div>
</td>
<td> </td>
<td class='rightColDetails'>
<div class='AvailDetails' align='left' style='display: table-cell;'>
<span class='iefix'><b>Avail - All Day</b></span><br/>
<span style='font-size: 11px;'>Pref - All Day</span>
</div>
</td>
</tr>
And this is a split shift, two four hour shifts separated by a empty 1 hour slot (they do this to cheat the scoring system, two covered shifts instead of one):
<tr>
<td class='dt'>
<span class='dow1'>Thursday</span>Dec 13
</td>
<td class='ScheduledDetails' valign='top'>
<div style="position:relative;">
<span class='tlHours'>8<span class='spart'> hrs</span> 0<span class='spart'> mins</span></span><span class='department'>Cashier</span><span class='meal'>Meal: None</span>
</div>
</td>
<td> </td>
<td class='Schedunderlay'>
<div class='Sched'>
<div class='schedbar' style='left: 247px; width: 104px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 96px;'>
<span class='start'>2pm</span><span class='end'>6pm</span>
</div><div class='schedbar_r'></div>
</div>
<div class='schedbar' style='left: 377px; width: 104px;'>
<div class='schedbar_l'></div>
<div class='schedbar_m' style='width: 96px;'>
<span class='start'>7pm</span> <span class='end'>11pm</span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 9px; width: 498px; display: none;'>
<div class='schedbar_l'></div><div class='schedbar_m' style='width: 490px;'>
<span class='start'><img src='/Images/Schedule/arrowLeft.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'></span>
</div>
<div class='schedbar_r'></div>
</div>
<div class='availbar' style='left: 508px; width: 216px; display: none;'>
<div class='schedbar_l_on'></div>
<div class='schedbar_m_on' style='width: 208px;'>
<span class='start'></span>
<div class='OTtext' align='center'>All Day</div>
<span class='end'><img src='/Images/Schedule/arrowRight.gif' alt='' style='margin-left:5px; margin-top:2px;' /></span>
</div>
<div class='schedbar_r_on'></div>
</div>
</div>
</td>
<td> </td>
<td class='rightColDetails'>
<div class='AvailDetails' align='left' style='display: table-cell;'>
<span class='iefix'><b>Avail - All Day</b></span><br/><span style='font-size: 11px;'>Pref - All Day</span>
</div>
</td>
</tr>
The important difference is on the regular shift there's one start and one end time, with the split shift there's a start, and end, and start, and end....
I've been pounding my head against this for about five hours now... and making no headway, I suppose I'd have more luck if I understood Regex.. any help at all would be greatly appreciated...
Here is a solution using BeautifulSoup to parse the document and grab the info.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
for schedbar in soup.find_all('div', 'schedbar'):
print "start: " + schedbar.find('div', 'schedbar_m').find('span', 'start').string
print "end: " + schedbar.find('div', 'schedbar_m').find('span', 'end').string
Outputs:
start: 2pm
end: 6pm
start: 7pm
end: 11pm