If I have a HTML Snippet as below, how do I get the required output as below in python.
Sample HTML snippet:
<td width="10" class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">></a></td>
<td class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">002396653</a></td>
<td class="data1">IMPORT EXPRESS RECYCLE</td>
<td class="data1">961879066</td>
<td class="data1">11/23/2016</td>
<td class="data1"></td> <!--SARA-->
<td class="data1" align="center">CN</td>
<td class="data1" align="center">PVG</td>
Output:
961879066|CN
My Code so far:
def reading():
with open("C:\\Users\\John\\Desktop\\test.txt") as f:
for lines in f.readlines():
line = lines.replace("\t","").strip()
print (line)
f.close()
reading()
Thanks,
You can try below code to get required output:
import lxml.html
html = lxml.html.fromstring("""<td width="10" class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">></a></td>
<td class="data1"><a class="datalink" href="m01_detail.asp?key=002396653&itemNumber=0">002396653</a></td>
<td class="data1">IMPORT EXPRESS RECYCLE</td>
<td class="data1">961879066</td>
<td class="data1">11/23/2016</td>
<td class="data1"></td> <!--SARA-->
<td class="data1" align="center">CN</td>
<td class="data1" align="center">PVG</td>""")
output = html.xpath('concat(//td[4], "|", //td[7])')
print(output) # '961879066|CN'
Pass original HTML code to html variable
Related
i have a piece of code that I run through different urls. Now I get an error when I try to extract the text or a td. What am I doing wrong?
ERROR
year = container.find('td', attrs={"class" : "label"}).text
AttributeError: 'NoneType' object has no attribute 'text'
CODE
soup = BeautifulSoup(results.text, "html.parser")
autos = soup.find('div', {"class" : "specleft"})
titles = []
years = []
time = []
movie_div = autos.find_all('tr')
#our loop through each container
for container in movie_div:
#year
year = container.find('td', attrs={"class" : "label"}).text
years.append(year)
time1 = container.find('td', attrs={"class" : "data"})
time.append(time1)
print(years)
print(time)
OUTPUT
[<td class="label">Original Base Price</td>, <td class="label">No. Produced</td>, None, <td class="label">Body Maker</td>, <td class="label">No. Doors</td>, <td class="label">Passengers</td>, <td class="label">Model Number</td>, <td class="label">Weight</td>, None, <td class="label">Wheelbase</td>, <td class="label">Length</td>, <td class="label">Width</td>, <td class="label">Height</td>, <td class="label">Front Tread</td>, <td class="label">Rear Tread</td>, None, <td class="label">Type</td>, <td class="label">Displacement</td>, <td class="label">Cylinders</td>, <td class="label">Bore & Stroke</td>, <td class="label">Compression Ratio-Std</td>, <td class="label">Compression Ratio-Opt</td>, <td class="label">Brake Horsepower</td>, <td class="label">Rated Horsepower</td>, <td class="label">Torque</td>, <td class="label">Main Bearings</td>, <td class="label">Valve Lifters</td>, <td class="label">Block Material</td>, <td class="label">Engine Numbers</td>, <td class="label">Engine No. Location</td>, <td class="label">Lubrication</td>, None, <td class="label">Type</td>, <td class="label">Make</td>, None, <td class="label">Type</td>, <td class="label">Drive</td>, <td class="label">No. Of Gears</td>, <td class="label">Gear Ratios</td>, <td class="label">1st</td>, <td class="label">2nd</td>, <td class="label">3rd</td>, <td class="label">4th</td>, <td class="label">5th</td>, <td class="label">Reverse</td>]
Looks like the find method returned None at some point. You can first check for None before getting the text. Something like this.
td_label = container.find('td', attrs={"class" : "label"})
text = "no text"
if td_label:
text = td_label.text
Im having trouble writing the contents of this soup function to the my ide.
I have the following soup function:
row = soup.find_all('td', attrs = {'class': 'Table__TD'})
here is the a subset of what it returns:
[<td class="Table__TD">Sat 11/9</td>,
<td class="Table__TD"><span class="flex"><span class="pr2">vs</span><span class="pr2 TeamLink__Logo"><a class="AnchorLink v-mid" data-clubhouse-uid="s:40~l:46~t:6" href="/nba/team/_/name/dal/dallas-mavericks" title="Team - Dallas Mavericks"><img alt="DAL" class="v-mid" data-clubhouse-uid="s:40~l:46~t:6" height="20" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" title="DAL" width="20"/></a></span><span><a class="AnchorLink v-mid" data-clubhouse-uid="s:40~l:46~t:6" href="/nba/team/_/name/dal/dallas-mavericks" title="Team - Dallas Mavericks">DAL</a></span></span></td>,
<td class="Table__TD"><a class="AnchorLink" data-game-link="true" href="http://www.espn.com/nba/game?gameId=401160772"><span class="flex tl"><span class="pr2"><div class="ResultCell tl loss-stat">L</div></span><span>138-122</span></span></a></td>,
<td class="Table__TD">31</td>,
<td class="Table__TD">6-12</td>,
<td class="Table__TD">50.0</td>,
<td class="Table__TD">4-9</td>,
<td class="Table__TD">44.4</td>,
<td class="Table__TD">2-2</td>,
<td class="Table__TD">100.0</td>,
<td class="Table__TD">4</td>,
<td class="Table__TD">4</td>,
<td class="Table__TD">2</td>,
<td class="Table__TD">3</td>,
<td class="Table__TD">2</td>,
<td class="Table__TD">1</td>,
<td class="Table__TD">18</td>,
<td class="Table__TD">Fri 11/8</td>,
I am trying to use a for loop to write these out but my console is not returning anything.
for data in row[0].find_all('td'):
print(data.get_text())
Can anyone tell me what I am doing wrong? Thanks.
With the initial search, you don't need to re-find_all on the tag name.
Just do something like:
for data in row:
print(data.get_text())
Let's say, i have an HTML Table like this:
<tr>
<td class="Klasse gerade">12A<br></td>
<td class="Stunde gerade">4<br></td>
<td class="Fach gerade">GEO statt GE<br></td>
<td class="Lehrer gerade"><br></td>
<td class="Vertretung gerade">Herr Grieger<br></td>
<td class="Raum gerade">603<br></td>
<td class="Anmerkung gerade"><br></td>
</tr>
<tr>
<td class="Klasse gerade">10A<br></td>
<td class="Stunde gerade">2<br></td>
<td class="Fach gerade">MA statt GE<br></td>
<td class="Lehrer gerade"><br></td>
<td class="Vertretung gerade">Herr Grieger<br></td>
<td class="Raum gerade">406<br></td>
<td class="Anmerkung gerade"><br></td>
</tr>
if phrase the HTML to python(2.7) with:
link = "http://www.test.com/vplan.html"
f = urllib.urlopen(link)
vplan = f.read()
print vplan
how can i do this?: if td=10A then print the complete tr of 10A
Sorry for the bad formulation but this is in my opinion the easiest was to explain my question and don't wonder about the German word's (I'm a German)
You need an HTML parser like Beautifulsoup. Assuming the table in question is the only one or the first one in the document, the program may look like this:
#!/usr/bin/env python
import urllib
from bs4 import BeautifulSoup
def main():
link = 'http://www.test.com/vplan.html'
soup = BeautifulSoup(urllib.urlopen(link), 'lxml')
table = soup.find('table')
rows = [x.find_parent('tr') for x in table.find_all(text='10A')]
for row in rows:
for cell in row.find_all('td'):
print cell.text
print '-' * 10
I have a webpage with a table that only appears when I click 'Inspect Element' and is not visible through the View Source page. The table contains only two rows with several cells each and looks similar to this:
<table class="datadisplaytable">
<tbody>
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</tbody>
</table>
What I'm trying to do is to iterate through the rows and return the text contained in each cell. I can't really seem to do it with Selenium. The elements contain no IDs and I'm not sure how else to get them. I'm not very familiar with using xpaths and such.
Here is a debugging attempt that returns a TypeError:
def check_grades(self):
table = []
for i in self.driver.find_element_by_class_name("dddefault"):
table.append(i)
print(table)
What is an easy way to get the text from the rows?
XPath is fragile. It's better to use CSS selectors or classes:
mytable = find_element_by_css_selector('table.datadisplaytable')
for row in mytable.find_elements_by_css_selector('tr'):
for cell in row.find_elements_by_tag_name('td'):
print(cell.text)
If you want to go row by row using an xpath, you can use the following:
h = """<table class="datadisplaytable">
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</table>"""
from lxml import html
xml = html.fromstring(h)
# gets the table
table = xml.xpath("//table[#class='datadisplaytable']")[0]
# iterate over all the rows
for row in table.xpath(".//tr"):
# get the text from all the td's from each row
print([td.text for td in row.xpath(".//td[#class='dddefault'][text()])
Which outputs:
['16759', 'MATH', '123', '001', 'Calculus']
['16449', 'PHY', '456', '002', 'Physics']
Using td[text()] will avoid getting any Nones returned for the td's that hold no text.
So to do the same using selenium you would:
table = driver.find_element_by_xpath("//table[#class='datadisplaytable']")
for row in table.find_elements_by_xpath(".//tr"):
print([td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault'][1]"])
For multiple tables:
def get_row_data(table):
for row in table.find_elements_by_xpath(".//tr"):
yield [td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault'][text()]"])
for table in driver.find_elements_by_xpath("//table[#class='datadisplaytable']"):
for data in get_row_data(table):
# use the data
Correction of the Selenium part of #Padraic Cunningham's answer:
table = driver.find_element_by_xpath("//table[#class='datadisplaytable']")
for row in table.find_elements_by_xpath(".//tr"):
print([td.text for td in row.find_elements_by_xpath(".//td[#class='dddefault']")])
Note: there was one missing round bracket at the end; also removed the [1] index, to match the first XML example.
Another note: Though, the example with the index [1] should also be preserved, to show how to extract individual elements.
Another Version (modified and corrected post by Padraic Cunningham):
Tested with Python 3.x
#!/usr/bin/python
h = """<table class="datadisplaytable">
<tr>
<td class="dddefault">16759</td>
<td class="dddefault">MATH</td>
<td class="dddefault">123</td>
<td class="dddefault">001</td>
<td class="dddefault">Calculus</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
<tr>
<td class="dddefault">16449</td>
<td class="dddefault">PHY</td>
<td class="dddefault">456</td>
<td class="dddefault">002</td>
<td class="dddefault">Physics</td>
<td class="dddefault"></td>
<td class="dddead"></td>
<td class="dddead"></td>
</tr>
</table>"""
from lxml import html
xml = html.fromstring(h)
# gets the table
table = xml.xpath("//table[#class='datadisplaytable']")[0]
# iterate over all the rows
for row in table.xpath(".//tr"):
# get the text from all the td's from each row
print([td.text for td in row.xpath(".//td[#class='dddefault']")])
I have this html table:
<table>
<tr>
<td class="datax">a</td>
<td class="datax">b</td>
<td class="datax">c</td>
<td class="datax">d</td>
</tr>
<tr>
<td class="datax">e</td>
<td class="datax">f</td>
<td class="datax">g</td>
<td class="datax">h</td>
</tr>
</table>
How to get the second and the fourth value of each <tr> ?
If i do:
bs.findAll('td', {'class':'datax'})
I get:
<td class="datax">a</td>
<td class="datax">b</td>
<td class="datax">c</td>
<td class="datax">d</td>
<td class="datax">e</td>
<td class="datax">f</td>
<td class="datax">g</td>
<td class="datax">h</td>
it's correct! but I would like to have this result:
<td class="datax">b</td>
<td class="datax">d</td>
<td class="datax">f</td>
<td class="datax">h</td>
so, the values I want are -> b - d - f - h
(the second and the forth <td> of each <tr>)
Is it possible with BeautifulSoup module?
Thank you very much!
This should do it~
final_values=[td.string for td in bs.findAll('td', {'class':'datax'})[1::2]]
(after comment clarification) for your specific case it would be:
final_values=[td.b.a.string for td in bs.findAll('td', {'class':'datax'})[1::2]]
I know using HTQL, it is simple:
<tr>.<td>2,4
--
HTQL only has COM support thought. Here is a complete example in javascript:
<html>
<body>
<script language=JavaScript>
var a= new ActiveXObject("HtqlCom.HtqlControl");
a.setUrl("C:\\test_table.html");
a.setQuery("<tr>.<td>2,4");
for (a.moveFirst(); !a.isEOF(); a.moveNext()){
document.write(a.getValueByIndex(1));
}
</script>
</body>
</html>