Develop Reference

How to continue filtering beyond BeautifulSoup find_all ResultSet?

Imagine you're trying to parse something like this with bs4:
<table>
<tbody>
<tr>
<th attr="attr" class="title">
Title Text
</th>
<th attr="attr" class="title">
Title Text 2
</th>
<th attr="attr" class="title">
Title Text 3
</th>
</tr>
</tbody>
<a href"otherlink.com">Other link to throw you off</a>
</table>
Currently I am able to get to a list of all the th elements with
html_content = BeautifulSoup(requests.get("parsingwebsite.com").content, "html.parser")
res = html_content.find_all("th", {"attr": "attr"}, class_="title")
But I only want the title text inside <a>. (ideally ["Title Text", "Title Text 2", "Title Text 3"])
Is there a way to continue filtering down by html element or otherwise modify the original query to filter down to the text inside the link, without having to use regex?

You can use CSS selector for selecting <a> tags under specific <th> tags.
For example th[attr="attr"].title a will select all <a> tags under <th> tags with attr="attr" and class="title":
txt = '''<table>
<tbody>
<tr>
<th attr="attr" class="title">
Title Text
</th>
<th attr="attr" class="title">
Title Text 2
</th>
<th attr="attr" class="title">
Title Text 3
</th>
</tr>
</tbody>
<a href"otherlink.com">Other link to throw you off</a>
</table>'''
soup = BeautifulSoup(txt, 'html.parser')
print([a.text for a in soup.select('th[attr="attr"].title a')])
Prints:
['Title Text', 'Title Text 2', 'Title Text 3']
Or using BeautifulSoup's own API:
print( [th.a.text for th in soup.find_all("th", {"attr": "attr"}, class_="title") if th.a] )

You can try this:
import requests
from bs4 import BeautifulSoup
html = '''<table> <tbody>
<tr>
<th attr="attr" class="title">
Title Text
</th>
<th attr="attr" class="title">
Title Text 2
</th>
<th attr="attr" class="title">
Title Text 3
</th>
</tr>
</tbody>
</table>'''
html_code = BeautifulSoup(html, 'html.parser')
a = html_code.find_all('a')
text_a = [i.text for i in a]
print(text_a)

Beautifulsoup iterate to get either <td>sometext</td> or url

I Want to create a list that contains a key-value pair. With the <thead> items as the key. For the values I want to get the text for all <th>items except the <th> items where there is a <a href='url'>, then I want to get the url instead.
Currently I am only able to get the text from all items. But how do I do to get '/someurl' instead of Makulerad and Detaljer?
<table class="table table-bordered table-hover table-striped zero-margin-top">
<thead>
<tr>
<th>Volymsenhet</th>
<th>Pris</th>
<th>Valuta</th>
<th>Handelsplats</th>
<th>url1</th>
<th>url2</th>
</tr>
</thead>
<tbody>
<tr class="iprinactive">
<td>Antal</td>
<td>5,40</td>
<td>SEK</td>
<td>NASDAQ STOCKHOLM AB</td>
<td>Makulerad</td>
<td>
Detaljer
</td>
</tr>
</tbody>
</table>
My code:
raw_html = simple_get('https://example.com/')
soup = BeautifulSoup(raw_html, 'html.parser')
table = soup.find("table", attrs={"class":"table"})
head = [th.get_text() for th in table.find("tr").find_all("th")]
datasets = []
for row in table.find_all("tr")[1:]:
dataset = dict(zip(head,(td.get_text() for td in row.find_all("td"))))
datasets.append(dataset)

Try this:
simply get the text data of <td> if it doesn't have an <a>. Otherwise get the href value.
from bs4 import BeautifulSoup
raw_html = '''<table class="table table-bordered table-hover table-striped zero-margin-top">
<thead>
<tr>
<th>Volymsenhet</th>
<th>Pris</th>
<th>Valuta</th>
<th>Handelsplats</th>
<th>url1</th>
<th>url2</th>
</tr>
</thead>
<tbody>
<tr class="iprinactive">
<td>Antal</td>
<td>5,40</td>
<td>SEK</td>
<td>NASDAQ STOCKHOLM AB</td>
<td>Makulerad</td>
<td>
Detaljer
</td>
</tr>
</tbody>
</table>'''
soup = BeautifulSoup(raw_html, 'html.parser')
table = soup.find("table", attrs={"class":"table"})
head = [th.get_text() for th in table.find("tr").find_all("th")]
datasets = []
for row in table.find_all("tr")[1:]:
dataset = dict(zip(head, [td.get_text() if not td.a else td.a['href'] for td in row.find_all("td")]))
datasets.append(dataset)
print(datasets)
OUTPUT:
[{'Volymsenhet': 'Antal', 'Pris': '5,40', 'Valuta': 'SEK', 'Handelsplats': 'NASDAQ STOCKHOLM AB', 'url1': '/someurl', 'url2': '/someurl'}]

How to use BeauifulSoup for parsing data in following example?

I am a beginner in Python and BeautifulSoup and I am trying to make a web scraper. However, I am facing some issues and can't figure out a way out. Here is my issue:
This is part of the HTML from where I want to scrap:
<tr>
<td class="num cell-icon-string" data-sort-value="6">
<td class="cell-icon-string"><a class="ent-name" href="/pokedex/charizard" title="View pokedex for #006 Charizard">Charizard</a></td>
</tr>
<tr>
<td class="num cell-icon-string" data-sort-value="6">
<td class="cell-icon-string"><a class="ent-name" href="/pokedex/charizard" title="View pokedex for #006 Charizard">Charizard</a><br>
<small class="aside">Mega Charizard X</small></td>
</tr>
Now, I want to extract "Charizard" from 1st table row and "Mega Charizard X" from the second row. Right now, I am able to extract "Charizard" from both rows.
Here is my code:
#!/usr/bin/env python3
from bs4 import BeautifulSoup
soup = BeautifulSoup(open("data.html"), "lxml")
poke_boxes = soup.findAll('a', attrs = {'class': 'ent-name'})
for poke_box in poke_boxes:
poke_name = poke_box.text.strip()
print(poke_name)

import bs4
html = '''<tr>
<td class="num cell-icon-string" data-sort-value="6">
<td class="cell-icon-string"><a class="ent-name" href="/pokedex/charizard" title="View pokedex for #006 Charizard">Charizard</a></td>
</tr>
<tr>
<td class="num cell-icon-string" data-sort-value="6">
<td class="cell-icon-string"><a class="ent-name" href="/pokedex/charizard" title="View pokedex for #006 Charizard">Charizard</a><br>
<small class="aside">Mega Charizard X</small></td>
</tr>'''
soup = bs4.BeautifulSoup(html, 'lxml')
in:
[tr.get_text(strip=True) for tr in soup('tr')]
out:
['Charizard', 'CharizardMega Charizard X']
you can use get_text() to concatenate all the text in the tag, strip=Ture will strip all the space in the string

You'll need to change your logic to go through the rows and check to see if the small element exists, if it does print out that text, otherwise print out the anchor text as you are now.
soup = BeautifulSoup(html, 'lxml')
trs = soup.findAll('tr')
for tr in trs:
smalls = tr.findAll('small')
if smalls:
print(smalls[0].text)
else:
poke_box = tr.findAll('a')
print(poke_box[0].text)

Using Python + BeautifulSoup to pick up text in a table on webpage

I want to pick up a date on a webpage.
The original webpage source code looks like:
<TR class=odd>
<TD>
<TABLE class=zp>
<TBODY>
<TR>
<TD><SPAN>Expiry Date</SPAN>2016</TD></TR></TBODY></TABLE></TD>
<TD> </TD>
<TD> </TD></TR>
I want to pick up the ‘2016’ but I fail. The most I can do is:
page = urllib2.urlopen('http://www.thewebpage.com')
soup = BeautifulSoup(page.read())
a = soup.find_all(text=re.compile("Expiry Date"))
And I tried:
b = a[0].findNext('').text
print b
and
b = a[0].find_next('td').select('td:nth-of-type(1)')
print b
neither of them works out.
Any help? Thanks.

There are multiple options.
Option #1 (using CSS selector, being very explicit about the path to the element):
from bs4 import BeautifulSoup
data = """
<TR class="odd">
<TD>
<TABLE class="zp">
<TBODY>
<TR>
<TD>
<SPAN>
Expiry Date
</SPAN>
2016
</TD>
</TR>
</TBODY>
</TABLE>
</TD>
<TD> </TD>
<TD> </TD>
</TR>
"""
soup = BeautifulSoup(data)
span = soup.select('tr.odd table.zp > tbody > tr > td > span')[0]
print span.next_sibling.strip() # prints 2016
We are basically saying: get me the span tag that is directly inside the td that is directly inside the tr that is directly inside tbody that is directly inside the table tag with zp class that is inside the tr tag with odd class. Then, we are using next_sibling to get the text after the span tag.
Option #2 (find span by text; think it is more readable)
span = soup.find('span', text=re.compile('Expiry Date'))
print span.next_sibling.strip() # prints 2016
re.compile() is needed since there could be multi-lines and additional spaces around the text. Do not forget to import re module.

An alternative to the css selector is:
import bs4
data = """
<TR class="odd">
<TD>
<TABLE class="zp">
<TBODY>
<TR>
<TD>
<SPAN>
Expiry Date
</SPAN>
2016
</TD>
</TR>
</TBODY>
</TABLE>
</TD>
<TD> </TD>
<TD> </TD>
</TR>
"""
soup = bs4.BeautifulSoup(data)
exp_date = soup.find('table', class_='zp').tbody.tr.td.span.next_sibling
print exp_date # 2016
To learn about BeautifulSoup, I recommend you read the documentation.

Issue in creating a html file from beautiful soup

Here is the my python code using BeautifulSoup. The main issue is with the attributes. What I am looking for is, each element of the th should be separated but for some reason it keep generating inside only one individual tag.
from BeautifulSoup import BeautifulSoup, Tag
soup=BeautifulSoup()
mem_attr=['Description','PhysicalID','Slot','Size','Width']
tag1 = Tag(soup, "html")
tag2 = Tag(soup, "table")
tag3 = Tag(soup, "tr")
tag4 = Tag(soup, "th")
tag5 = Tag(soup, "td")
soup.insert(0, tag1)
tag1.insert(0, tag2)
tag2.insert(0, tag3)
for i in range(0,len(mem_attr)):
tag3.insert(0,tag4)
tag4.insert(i,mem_attr[i])
print soup.prettify()
Here is its output:
<html>
<table>
<tr>
<th>
Description
PhysicalID
Slot
Size
Width
</th>
</tr>
</table>
</html>
What I am looking for is this one.
<html>
<table>
<tr>
<th>
Description
</th>
<th>
PhysicalID
</th>
<th>
Slot
</th>
<th>
Size
</th>
<th>
Width
</th>
</tr>
</table>
</html>
Can anyone tell me what is missing in the code?.

You're putting it in the same th. You never told it to create more than one.
Here is code more like what you are wanting:
from BeautifulSoup import BeautifulSoup, Tag
soup = BeautifulSoup()
mem_attr = ['Description', 'PhysicalID', 'Slot', 'Size', 'Width']
html = Tag(soup, "html")
table = Tag(soup, "table")
tr = Tag(soup, "tr")
soup.append(html)
html.append(table)
table.append(tr)
for attr in mem_attr:
th = Tag(soup, "th")
tr.append(th)
th.append(attr)
print soup.prettify()