Develop Reference

Retrieving table values from HTML with the same tag names using Beautiful Soup in Python

I am trying to retrieve all the td text for the below table using Beautiful Soup, unfortunately the tag names are the same and I am either only able to retrieve the first element or some elements are repeatedly printing. Hence not really sure of how to go about it.
Below is HTML table snippet:
<div>Table</div>
<table class="Auto" width="100%">
<tr>
<td class="Auto_head">Address</td>
<td class="Auto_head">Name</td>
<td class="Auto_head">Type</td>
<td class="Auto_head">Value IN</td>
<td class="Auto_head">AUTO Statement</td>
<td class="Auto_head">Value OUT</td>
<td class="Auto_head">RESULT</td>
<td class="Auto_head"></td>
</tr>
<tr>
<td class="Auto_body">1</td>
<td class="Auto_body">abc</td>
<td class="Auto_body">yes</td>
<td class="Auto_body">abc123</td>
<td class="Auto_body">jar</td>
<td class="Auto_body">123abc</td>
<td class="Auto_body">PASS</td>
<td class="Auto_body">na</td>
</tr>
What I want is all the text content inside these tags for example the first auto_head corresponds to first auto_body i.e. Address = 1 similarly all the values should be retrieved.
I have used find,findall,findNext and next_sibling but no luck. Here is my current code in python:
self.table = self.soup_file.findAll(class_="Table")
self.headers = [tab.find(class_="Auto_head").findNext('td',class_="Auto_head").contents[0] for tab in self.table]
self.data = [data.find(class_="Auto_body").findNext('td').contents[0] for data in self.table]

Get the headers first, then use zip(...) to combine
from bs4 import BeautifulSoup
data = '''\
<table class="Auto" width="100%">
<tr>
<td class="Auto_head">Address</td>
<td class="Auto_head">Name</td>
<td class="Auto_head">Type</td>
</tr>
<tr>
<td class="Auto_body">1</td>
<td class="Auto_body">abc</td>
<td class="Auto_body">yes</td>
</tr>
<tr>
<td class="Auto_body">2</td>
<td class="Auto_body">def</td>
<td class="Auto_body">no</td>
</tr>
<tr>
<td class="Auto_body">3</td>
<td class="Auto_body">ghi</td>
<td class="Auto_body">maybe</td>
</tr>
</table>
'''
soup = BeautifulSoup(data, 'html.parser')
for table in soup.select('table.Auto'):
# get rows
rows = table.select('tr')
# get headers
headers = [td.text for td in rows[0].select('td.Auto_head')]
# get details
for row in rows[1:]:
values = [td.text for td in row.select('td.Auto_body')]
print(dict(zip(headers, values)))
My output:
{'Address': '1', 'Name': 'abc', 'Type': 'yes'}
{'Address': '2', 'Name': 'def', 'Type': 'no'}
{'Address': '3', 'Name': 'ghi', 'Type': 'maybe'}

Get each category first then iterate using zip
s = '''<div>Table</div>
<table class="Auto" width="100%">
<tr>
<td class="Auto_head">Address</td>
<td class="Auto_head">Name</td>
<td class="Auto_head">Type</td>
<td class="Auto_head">Value IN</td>
<td class="Auto_head">AUTO Statement</td>
<td class="Auto_head">Value OUT</td>
<td class="Auto_head">RESULT</td>
<td class="Auto_head"></td>
</tr>
<tr>
<td class="Auto_body">1</td>
<td class="Auto_body">abc</td>
<td class="Auto_body">yes</td>
<td class="Auto_body">abc123</td>
<td class="Auto_body">jar</td>
<td class="Auto_body">123abc</td>
<td class="Auto_body">PASS</td>
<td class="Auto_body">na</td>
</tr></table>'''
soup = BeautifulSoup(s,features='html')
head = soup.find_all(name='td',class_='Auto_head')
body = soup.find_all(name='td',class_='Auto_body')
for one,two in zip(head,body):
print(f'{one.text}={two.text}')
Address=1
Name=abc
Type=yes
Value IN=abc123
AUTO Statement=jar
Value OUT=123abc
RESULT=PASS
=na
Searching by CSS class

The easiest solution is to add the find_all method at the end of the find
so your code will be
source = requests.get('YOUR URL')
soup=BeautifulSoup(source.text,'html.parser')
data = soup.find('tr').find_all('td')[0]
data = soup.find('tr').find_all('td')[1]
and so on just change the last list number 0,1,2... or else use for loop for the same

Python text out of TD error 'NoneType' object has no attribute 'text' WHY?

i have a piece of code that I run through different urls. Now I get an error when I try to extract the text or a td. What am I doing wrong?
ERROR
year = container.find('td', attrs={"class" : "label"}).text
AttributeError: 'NoneType' object has no attribute 'text'
CODE
soup = BeautifulSoup(results.text, "html.parser")
autos = soup.find('div', {"class" : "specleft"})
titles = []
years = []
time = []
movie_div = autos.find_all('tr')
#our loop through each container
for container in movie_div:
#year
year = container.find('td', attrs={"class" : "label"}).text
years.append(year)
time1 = container.find('td', attrs={"class" : "data"})
time.append(time1)
print(years)
print(time)
OUTPUT
[<td class="label">Original Base Price</td>, <td class="label">No. Produced</td>, None, <td class="label">Body Maker</td>, <td class="label">No. Doors</td>, <td class="label">Passengers</td>, <td class="label">Model Number</td>, <td class="label">Weight</td>, None, <td class="label">Wheelbase</td>, <td class="label">Length</td>, <td class="label">Width</td>, <td class="label">Height</td>, <td class="label">Front Tread</td>, <td class="label">Rear Tread</td>, None, <td class="label">Type</td>, <td class="label">Displacement</td>, <td class="label">Cylinders</td>, <td class="label">Bore & Stroke</td>, <td class="label">Compression Ratio-Std</td>, <td class="label">Compression Ratio-Opt</td>, <td class="label">Brake Horsepower</td>, <td class="label">Rated Horsepower</td>, <td class="label">Torque</td>, <td class="label">Main Bearings</td>, <td class="label">Valve Lifters</td>, <td class="label">Block Material</td>, <td class="label">Engine Numbers</td>, <td class="label">Engine No. Location</td>, <td class="label">Lubrication</td>, None, <td class="label">Type</td>, <td class="label">Make</td>, None, <td class="label">Type</td>, <td class="label">Drive</td>, <td class="label">No. Of Gears</td>, <td class="label">Gear Ratios</td>, <td class="label">1st</td>, <td class="label">2nd</td>, <td class="label">3rd</td>, <td class="label">4th</td>, <td class="label">5th</td>, <td class="label">Reverse</td>]

Looks like the find method returned None at some point. You can first check for None before getting the text. Something like this.
td_label = container.find('td', attrs={"class" : "label"})
text = "no text"
if td_label:
text = td_label.text

Scraping table with BeautifulSoup4

I am trying to scrape some particulars rows inside a table but I don't know how to access the information properly. Here is the html:
<tr class="even">
<td style="background: #F5645C; color: #F5645C;">1 </td>
<td>Michael</td>
<td class="right">57</td>
<td class="right">0</td>
<td class="right">5</td>
</tr>
<tr class="odd">
<td style="background: #8FB9B0; color: #8FB9B0;">1 </td>
<td>Clara</td>
<td class="right">48</td>
<td class="right">0</td>
<td class="right">5</td>
</tr>
<tr class="even">
<td style="background: #F5645C; color: #F5645C;">1 </td>
<td>Lisa</td>
<td class="right">44</td>
<td class="right">2</td>
<td class="right">5</td>
</tr>
<tr class="odd">
<td style="background: #8FB9B0; color: #8FB9B0;">0 </td>
<td>Joe</td>
<td class="right">43</td>
<td class="right">0</td>
<td class="right">13</td>
</tr>
<tr class="even">
<td style="background: #F5645C; color: #F5645C;">1 </td>
<td>John</td>
<td class="right">38</td>
<td class="right">3</td>
<td class="right">4</td>
</tr>
<tr class="odd">
<td style="background: #F5645C; color: #F5645C;">1 </td>
<td>Francesca</td>
<td class="right">35</td>
<td class="right">2</td>
<td class="right">5</td>
</tr>
<tr class="even">
<td style="background: #8FB9B0; color: #8FB9B0;">0 </td>
<td>Carlos</td>
<td class="right">27</td>
<td class="right">1</td>
<td class="right">2</td>
</tr>
What I try to obtain, is the text on the next td that comes after every td with the style of color F5645C, but unfortunately I am running into problems.
This is what I want the script to return:
Michael
Lisa
John
Francesca
Here is the code I currently have:
table = soup.find('table')
table_rows = table.find_all('tr')
for tr in table_rows:
td = tr.find('td', style='background: #F5645C; color: #F5645C;').find_next_sibling('td').get_text()
print(td)
On running the script: AttributeError: 'NoneType' object has no attribute 'find_next_sibling'

You can use CSS selector to select all <td> tags that contain attribute style with string color: #F5645C and then apply method find_next():
for td in soup.select('td[style*="color: #F5645C"]'):
print(td.find_next('td').text)
This prints:
Michael
Lisa
John
Francesca

data = BeautifulSoup(html)
for tr in data.find_all('tr'):
td = tr.find_all('td')
print(td[1].text)
Now you can take it further i think..

Use .findNext("td").text
Ex:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
for tr in soup.find_all("tr"):
print(tr.td.findNext("td").text)
Output:
Michael
Clara
Lisa
Joe
John
Francesca
Carlos

Use can use find_all and a filter for the style atribute:
bs = BeautifulSoup(htmlcontent)
bs.find_all('td', attrs={'style':'background-color: #F5645C, color: #F5645C'})

Customizing HTML snippet in Python

If I have a HTML Snippet as below, how do I get the required output as below in python.
Sample HTML snippet:
<td width="10" class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">></a></td>
<td class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">002396653</a></td>
<td class="data1">IMPORT EXPRESS RECYCLE</td>
<td class="data1">961879066</td>
<td class="data1">11/23/2016</td>
<td class="data1"></td> <!--SARA-->
<td class="data1" align="center">CN</td>
<td class="data1" align="center">PVG</td>
Output:
961879066|CN
My Code so far:
def reading():
with open("C:\\Users\\John\\Desktop\\test.txt") as f:
for lines in f.readlines():
line = lines.replace("\t","").strip()
print (line)
f.close()
reading()
Thanks,

You can try below code to get required output:
import lxml.html
html = lxml.html.fromstring("""<td width="10" class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">></a></td>
<td class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">002396653</a></td>
<td class="data1">IMPORT EXPRESS RECYCLE</td>
<td class="data1">961879066</td>
<td class="data1">11/23/2016</td>
<td class="data1"></td> <!--SARA-->
<td class="data1" align="center">CN</td>
<td class="data1" align="center">PVG</td>""")
output = html.xpath('concat(//td[4], "|", //td[7])')
print(output) # '961879066|CN'
Pass original HTML code to html variable

Iterating Through Table Rows in Selenium (Python)

I have a webpage with a table that only appears when I click 'Inspect Element' and is not visible through the View Source page. The table contains only two rows with several cells each and looks similar to this:
<table class="datadisplaytable">
<tbody>
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</tbody>
</table>
What I'm trying to do is to iterate through the rows and return the text contained in each cell. I can't really seem to do it with Selenium. The elements contain no IDs and I'm not sure how else to get them. I'm not very familiar with using xpaths and such.
Here is a debugging attempt that returns a TypeError:
def check_grades(self):
table = []
for i in self.driver.find_element_by_class_name("dddefault"):
table.append(i)
print(table)
What is an easy way to get the text from the rows?

XPath is fragile. It's better to use CSS selectors or classes:
mytable = find_element_by_css_selector('table.datadisplaytable')
for row in mytable.find_elements_by_css_selector('tr'):
for cell in row.find_elements_by_tag_name('td'):
print(cell.text)

If you want to go row by row using an xpath, you can use the following:
h = """<table class="datadisplaytable">
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</table>"""
from lxml import html
xml = html.fromstring(h)
# gets the table
table = xml.xpath("//table[#class='datadisplaytable']")[0]
# iterate over all the rows
for row in table.xpath(".//tr"):
# get the text from all the td's from each row
print([td.text for td in row.xpath(".//td[#class='dddefault'][text()])
Which outputs:
['16759', 'MATH', '123', '001', 'Calculus']
['16449', 'PHY', '456', '002', 'Physics']
Using td[text()] will avoid getting any Nones returned for the td's that hold no text.
So to do the same using selenium you would:
table = driver.find_element_by_xpath("//table[#class='datadisplaytable']")
for row in table.find_elements_by_xpath(".//tr"):
print([td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault'][1]"])
For multiple tables:
def get_row_data(table):
for row in table.find_elements_by_xpath(".//tr"):
yield [td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault'][text()]"])
for table in driver.find_elements_by_xpath("//table[#class='datadisplaytable']"):
for data in get_row_data(table):
# use the data

Correction of the Selenium part of #Padraic Cunningham's answer:
table = driver.find_element_by_xpath("//table[#class='datadisplaytable']")
for row in table.find_elements_by_xpath(".//tr"):
print([td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault']")])
Note: there was one missing round bracket at the end; also removed the [1] index, to match the first XML example.
Another note: Though, the example with the index [1] should also be preserved, to show how to extract individual elements.

Another Version (modified and corrected post by Padraic Cunningham):
Tested with Python 3.x
#!/usr/bin/python
h = """<table class="datadisplaytable">
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</table>"""
from lxml import html
xml = html.fromstring(h)
# gets the table
table = xml.xpath("//table[#class='datadisplaytable']")[0]
# iterate over all the rows
for row in table.xpath(".//tr"):
# get the text from all the td's from each row
print([td.text for td in row.xpath(".//td[#class='dddefault']")])