I am building HTML table from the list through lxml.builder and striving to make a link in one of the table cells
List is generated in a following way:
with open('some_file.html', 'r') as f:
table = etree.parse(f)
p_list = list()
rows = table.iter('div')
p_list.append([c.text for c in rows])
rows = table.xpath("body/table")[0].findall("tr")
for row in rows[2:]:
p_list.append([c.text for c in row.getchildren()])
HTML file which I parse is the same that is generated further by lxml, i.e. I set up some sort of recursion for testing purposes.
And here is how I build table
from lxml.builder import E
page = (
E.html(
E.head(
E.title("title")
),
E.body(
....
*[E.tr(
*[
E.td(E.a(E.img(src=str(col)))) if ind == 8 else
E.td(E.a(str(col), href=str(col))) if ind == 9 else
E.td(str(col)) for ind, col in enumerate(row)
]
) for row in p_list ]
When I specify link via literals all is going fine.
E.td(E.a("link", href="url_address"))
However, when I try to output list element value (which is https://blahblahblah.com) as a link
E.td(E.a(str(col), href=str(col)))
cell is empty, just nothing is showed in the cell.
If I specify link text as a literal and put str (col) into href, the link is showed normally, but instead of real href it contains the name of the generated html file.
If I output just that col value as a string
E.td(str(col))
it is showed normally, i.e. it is not empty. What is wrong with E.a and E.img elements?
Just noticed that this happens only if I build list from html file. When I build list manually, like this, all is output fine.
p_list = []
p_element = ['id']
p_element.append('value')
p_element.append('value2')
p_list.append(p_element)
Current output (pay attention to <a> and <href> tags)
<html>
<head>
<title>page</title>
</head>
<body>
<style type="text/css">
th {
background-color: DeepSkyBlue;
text-align: center;
vertical-align: bottom;
height: 150px;
padding-bottom: 3px;
padding-left: 5px;
padding-right: 5px;
}
.vertical {
text-align: center;
vertical-align: middle;
width: 20px;
margin: 0px;
padding: 0px;
padding-left: 3px;
padding-right: 3px;
padding-top: 10px;
white-space: nowrap;
-webkit-transform: rotate(-90deg);
-moz-transform: rotate(-90deg);
}</style>
<h1>title</h1>
<p>This is another paragraph, with a</p>
<table border="2">
<tr>
<th>
<div class="vertical">ID</div>
</th>
...
<th>
<div class="vertical">I blacklisted him</div>
</th>
</tr>
<tr>
<td>1020</td>
<td>ТаисияСтрахолет</td>
<td>No</td>
<td>Female</td>
<td>None</td>
<td>Санкт-Петербург</td>
<td>Росiя</td>
<td>None</td>
<td>
<a>
<img src="
"/>
</a>
</td>
<td>
<a href="
">
</a>
</td>
...
</tr>
</table>
</body>
</html>
Desired output
<html>
<head>
<title>page</title>
</head>
<body>
<style type="text/css">
th {
background-color: DeepSkyBlue;
text-align: center;
vertical-align: bottom;
height: 150px;
padding-bottom: 3px;
padding-left: 5px;
padding-right: 5px;
}
.vertical {
text-align: center;
vertical-align: middle;
width: 20px;
margin: 0px;
padding: 0px;
padding-left: 3px;
padding-right: 3px;
padding-top: 10px;
white-space: nowrap;
-webkit-transform: rotate(-90deg);
-moz-transform: rotate(-90deg);
}</style>
<h1>title</h1>
<p>This is another paragraph, with a</p>
<table border="2">
<tr>
<th>
<div class="vertical">ID</div>
</th>
...
<th>
<div class="vertical">I blacklisted him</div>
</th>
</tr>
<tr>
<td>1019</td>
<td>МихаилПавлов</td>
<td>No</td>
<td>Male</td>
<td>None</td>
<td>Санкт-Петербург</td>
<td>Росiя</td>
<td>C.-Петербург</td>
<td>
<a>
<img src="http://i.imgur.com/rejChZW.jpg"/>
</a>
</td>
<td>
link
</td>
...
</tr>
</table>
</body>
</html>
Got it myself. The problem was not in generating but in parsing HTML. Parsing function didn't fetch IMG and A tags nested in TD and these elements of the list were empty. Due to the rigorous logic of the program (fetching from file + fetching from site API) I wasn't able to detect the cause of the issue.
The correct parsing logic should be:
for row in rows[1:]:
data.append([
c.find("a").text if c.find("a") is not None else
c.find("img").attrib['src'] if c.find("img") is not None else
c.text
for c in row.getchildren()
])
Just a guess but it looks like 'str()' could be escaping it. Try E.td(E.a(col, href=col))
Related
I’m trying to create pdf file for payment receipt, but I’m not able to figure out how I should set border for it.
As border I want to use this image:
But while converting it to pdf, next page gets like this:
How can I make it constant border for all pages?
Python + Django code:
from weasyprint import HTML
html_string = render_to_string('receipt.html', DATA)
html = HTML(string=html_string)
result = html.write_pdf()
f = open(str(os.path.join(MEDIA_URL + "invoice_receipt/", 'temp.pdf')), 'wb')
f.write(result)
file_obj = File(open(MEDIA_URL + "invoice_receipt/" + "temp.pdf", 'rb'))
transaction.receipt_file = file_obj
transaction.save()
receipt.html template:
<style>
table tbody tr td{
border-top: unset !important;
}
table tbody tr:nth-child(7) td,
table tbody tr:nth-child(8) td,
table tbody tr:nth-child(9) td,
table tbody tr:nth-child(10) td,
table tbody tr:nth-child(11) td,
table tbody tr:nth-child(12) td
{
padding-top: 0;
padding-bottom: 0;
}
.amount-in-words{
border-bottom:3px solid black;
}
.table thead th {
vertical-align: bottom;
border-bottom: 4px solid black;
}
/* .invoice-template{
padding: 20px;
border: 20px solid transparent;
border-image: linear-gradient(to right,#633363 50%,#f3c53d 50%);
border-image-slice: 1;
} */
.logo{
margin-top: 2rem;
}
.logo2{
margin-top: 2rem;
height: 160px;
width:200px;
}
.invoice-template{
padding: 20px;
background-image: url('https://dev-api.test.com/files/files/DumpData/Frame.png');
background-repeat: no-repeat;
background-size: contain;
break-inside: auto;
}
.main-container{
border: 1px solid black;
padding: 20px 10px;
background: white;
}
p {
font-weight: 500;
}
</style>
</head>
<body>
<div class="container invoice-template">
<!-- <div class="main-container"> -->
<div class="row justify-content-center">
<div class="col-md-5 logo"><img src={{ logo }} class="logo2"></div>
<div class="col-md-5 text-right">
<ul style="list-style: none; color: purple; margin-top: 2rem;">
<li>{{ phone }}<span></span></li>
<li><p>{{ email }}<br>{{ website }}</p><span></span></li>
<li>Resource Factory Pvt. Ltd.<br>{{ shop_address|linebreaksbr }}<span></span></li>
</ul>
</div>
</div>
<div class="row text-center">
<div class="col-md-12"><h6>INVOICE</h6></div>
</div>
<div class="row justify-content-center">
<div class="col-md-5">
<p>
To,<br>
{{ user_name }}<br>
{{ user_address|linebreaksbr }}
</p>
<p>Client GST Number.:</p>
</div>
<div class="col-md-5 text-center">
<p>Date: {{ order_date|date:"d-m-Y" }}</p>
<p>Invoice No. {{ invoice }}</p>
</div>
</div>
I’m giving a short version of my html code. If needed full code please mention.
The behavior of box decorations when a box is split (like your main <div> here) is controller by box-decoration-break. Default is slice which breaks the borders after rendering them. clone will compute the borders on each part of the box:
.invoice-template {
box-decoration-break: clone;
}
I'm trying to parse an HTML email from python code to extract various details and would appreciate a regular expression or two to help achieve this as it is too complex for my limited regex understanding. e.g. look for 'Travel Date' and extract 'October 30 2018 (Tue)'.
In all cases there is a field name contained within <td> tags followed by the field value contained within another set of <td> tags. Sometimes the name and value are contained within the same row <tr> tags (Case 1) and other times they are in separate row tags (Case 2). Other items like <span> and <img> need to be skipped over as well.
Case 1
<tr>
<td colspan="2"> </td></tr>
<tr><td style="vertical-align: top; font-size: 13px; font-family: Arial; color: #777777;">Travel Date</td>
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color: #444444;">October 30 2018 (Tue)</td>
</tr>
Case 2
<tr><td style="vertical-align: top;">
<span style="font-size: 10px; font-family: Arial; color: #999999; font-weight: bold; line-height: 19px; text-transform: uppercase;">Drop-off to Address</span>
</td></tr>
<tr><td style="vertical-align: top;">
<span style="font-size: 13px; font-family: Arial; color: #444444;"><img style="vertical-align:text-bottom;" src="https://d1lk4k9zl9klra.cloudfront.net/Email/Common/address_icon.png" alt="" width="14" height="14" /> 200 George St, Sydney NSW 2000, Australia</span>
</td></tr>
Instead of using regex, I would use Beautiful Soup. It makes it easier to go through HTML elements and scrape what you need. If you know the relationship between the key and value, then you could use that to extract information. Here's an example for case 1:
In [8]: from bs4 import BeautifulSoup
In [9]: text = """
...: <tr>
...: <td colspan="2"> </td></tr>
...: <tr><td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#777777;">Travel Date</td>
...: <td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#444444;">October 30 2018 (Tue)</td>
...: </tr>"""
In [11]: soup = BeautifulSoup(text, 'lxml')
In [13]: soup.find_all('td')
Out[13]:
[<td colspan="2"> </td>,
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#777777;">Travel Date</td>,
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#444444;">October 30 2018 (Tue)</td>]
In [15]: for tag in soup.find_all('td'):
...: if tag.text == "Travel Date":
...: print tag.find_next().text
...:
October 30 2018 (Tue)
Beautiful Soup gives a lot of flexibility when scraping HTML from the web.
I was given some great code by 'tuomastik' on this site, and have tweaked it slightly to work for me. However, I have spent several hours trying to edit it so it prints to one PDF rather than multiple (one PDF, but each report starting on a fresh page), but my HTML isn't as good as I want to be, and I am getting stuck.
The code is:
HTML
<html>
<head>
<style type="text/css">
html, body {
width: 500px;
font-size: 12px;
background: #fff;
padding: 0px;
}
#my-custom-table {
width: 500px;
border: 0;
margin-top: 20px;
}
#my-custom-table td {
padding: 5px 0px 1px 5px;
text-align: left;
}
</style>
</head>
<body>
<table cellspacing="0" border="0" style="width:500px; border:0; font-size: 14px;">
<tr>
<td style="text-align:left;">
<b><span>Title of the PDF report - Row {{ row_ix + 1 }}</span></b>
</td>
<td style="text-align:right;">
<b><span>{{ date }}</span></b>
</td>
</tr>
</table>
<table cellspacing="0" border="0.1" id="my-custom-table">
{% for variable_name, variable_value in df.iteritems() %}
{% if loop.index0 == 0 %}
<tr style="border-top: 1px solid black;
border-bottom: 1px solid black;
font-weight: bold;">
<td>Variable name</td>
<td>Variable value</td>
</tr>
{% else %}
<tr>
<td>{{ variable_name }}</td>
<td>{{ variable_value }}</td>
</tr>
{% endif %}
{% endfor %}
</table>
</body>
</html>
Python
from datetime import date
import jinja2
import pandas as pd
from xhtml2pdf import pisa
df = pd.read_csv('data.csv', encoding='cp1252')
for row_ix, row in df.iterrows():
html = jinja2.Environment( # Pandas DataFrame to HTML
loader=jinja2.FileSystemLoader(searchpath='')).get_template(
'report_template.html').render(date=date.today().strftime('%d, %b %Y'),
row_ix=row_ix, df=row)
# Convert HTML to PDF
with open('report_row_%s.pdf' % (row_ix+1), "w+b") as out_pdf_file_handle:
pisa.CreatePDF(
src=html, # HTML to convert
dest=out_pdf_file_handle) # File handle to receive result
Could anyone help? I know I should remove the for loop from the python file, but not sure what to do with the html file.
I hate asking, but I have given this a good attempt but I can't get it to run when I mess with the loops in html.
Many thanks
report_template.html
<html>
<head>
<style type="text/css">
html, body {
width: 500px;
font-size: 12px;
background: #fff;
padding: 0px;
}
#my-custom-table {
width: 500px;
border: 0;
margin-top: 20px;
}
#my-custom-table td {
padding: 5px 0px 1px 5px;
text-align: left;
}
</style>
</head>
<body>
{% for row_ix, row in df.iterrows() %}
<table cellspacing="0" border="0" style="width:500px; border:0; font-size: 14px;">
<tr>
<td style="text-align:left;">
<b><span>Title of the PDF report - Row {{ row_ix + 1 }}</span></b>
</td>
<td style="text-align:right;">
<b><span>{{ date }}</span></b>
</td>
</tr>
</table>
<table cellspacing="0" border="0.1" id="my-custom-table">
{% for variable_name, variable_value in row.iteritems() %}
{% if loop.index0 == 0 %}
<tr style="border-top: 1px solid black;
border-bottom: 1px solid black;
font-weight: bold;">
<td>Variable name</td>
<td>Variable value</td>
</tr>
{% else %}
<tr>
<td>{{ variable_name }}</td>
<td>{{ variable_value }}</td>
</tr>
{% endif %}
{% endfor %}
</table>
<!-- Page break (the syntax is for xhtml2pdf) -->
<pdf:nextpage />
{% endfor %}
</body>
</html>
Python
from datetime import date
import jinja2
import pandas as pd
from xhtml2pdf import pisa
df = pd.DataFrame({
"Average Introducer Score": [9, 9.1, 9.2],
"Reviewer Scores": ["Academic: 6, 6, 6", "Something", "Content"],
"Average Academic Score": [5.7, 5.8, 5.9],
"Average User Score": [1.2, 1.3, 1.4],
"Applied for (RC)": [9.2, 9.3, 9.4],
"Applied for (FEC)": [5.5, 5.6, 5.7],
"Duration (Months)": [36, 37, 38]})
html = jinja2.Environment( # Pandas DataFrame to HTML
loader=jinja2.FileSystemLoader(searchpath='')).get_template(
'report_template.html').render(date=date.today().strftime('%d, %b %Y'),
df=df)
# Convert HTML to PDF
with open('report.pdf', "w+b") as out_pdf_file_handle:
pisa.CreatePDF(
src=html, # HTML to convert
dest=out_pdf_file_handle) # File handle to receive result
I'm getting this error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2661' in position 1409: ordinal not in range(128)
I'm very green to programming still, so have mercy on me and my ignorance. But I understand the error to be that it's not able to handle unicode characters. There's that at least one unicode char, but there could be countless others that'll perk up in that feed.
I've done some looking for others who've had similar problems, but I can't can't find a solution I understand or can make work.
#import library to do http requests:
import urllib
from xml.dom.minidom import parseString, parse
f = open('games.html', 'w')
document = urllib.urlopen('https://itunes.apple.com/us/rss/topfreemacapps/limit=300/genre=12006/xml')
dom = parse(document)
image = dom.getElementsByTagName('im:image')
title = dom.getElementsByTagName('title')
price = dom.getElementsByTagName('im:price')
address = dom.getElementsByTagName('id')
imglist = []
titlist = []
pricelist = []
addlist = []
i = 0
j = 20
k = 40
f.write('''\
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
<!--
A:link {text-decoration: none; color: #246DA8;}
A:visited {text-decoration: none; color: #246DA8;}
A:active {text-decoration: none; color: #40A9E3;}
A:hover {text-decoration: none; color: #40A9E3;}
.box {
vertical-align:middle;
width: 180px;
height: 120px;
border: 1px solid #99c;
padding: 5px;
margin: 0px;
margin-left: auto;
margin-right: auto;
-moz-border-radius: 5px;
border-radius: 5px;
-webkit-border-radius: 5px;
background-color:#ffffff;
font-family: Arial, Helvetica, sans-serif; color: black;
font-size: small;
font-weight: bold;
}
-->
</style>
</head>
<body>
''')
for i in range(0,len(image)):
if image[i].getAttribute('height') == '53':
imglist.append(image[i].firstChild.nodeValue)
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue)
for i in range(0,len(price)):
pricelist.append(price[i].firstChild.nodeValue)
for i in range(1,len(address)):
addlist.append(address[i].firstChild.nodeValue)
for i in range(0,20):
f.write('''
<div style="width: 600px;">
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i] + '''<br>
<img src="''' + imglist[i] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i] + '''</span>
</div>
</div>
</div>
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i+j] + '''<br>
<img src="''' + imglist[i+j] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i+j] + '''</span>
</div>
</div>
</div>
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i+k] + '''<br>
<img src="''' + imglist[i+k] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i+k] + '''</span>
</div>
</div>
</div>
<br style="clear: left;" />
</div>
<br>
''')
f.write('''</body>''')
f.close()
The basic problem is that you're concatenating the Unicode strings with ordinary byte-strings without converting them using a proper encoding; in these cases, ASCII is used by default (which, clearly, can't handle extended characters).
The line in your script that does this is too long to quote, but another practical example which displays the same problem could look like this:
parameter = u"foo \u2661"
sys.stdout.write(parameter + " bar\n")
You will need to instead encode the Unicode strings with an explicitly specified encoding, e.g. like this:
parameter = u"foo \u2661"
sys.stdout.write(parameter.encode("utf8") + " bar\n")
In your case, you can do this in your loops so as to not have to specify it on every concatenation:
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue.encode("utf8"))
--
Also, while we're at it, you can improve your code by not iterating through the elements using an integer index. For instance, instead of this:
title = dom.getElementsByTagName('title')
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue.encode("utf8"))
... you can do this instead:
for title in dom.getElementsByTagName('title')
titlist.append(title.firstChild.nodeValue.encode("utf8"))
To start off here's my current code in its entirety:
import urllib
from BeautifulSoup import BeautifulSoup
import sgmllib
import re
page = 'http://www.sec.gov/Archives/edgar/data/\
8177/000114036111018563/form10k.htm'
sock = urllib.urlopen(page)
raw = sock.read()
soup = BeautifulSoup(raw)
tablelist = soup.findAll('table')
class MyParser(sgmllib.SGMLParser):
def parse(self, segment):
self.feed(segment)
self.close()
def __init__(self, verbose=0):
sgmllib.SGMLParser.__init__(self, verbose)
self.descriptions = []
self.inside_td_element = 0
self.starting_description = 0
def start_td(self, attributes):
for name, value in attributes:
if name == "valign":
self.inside_td_element = 1
self.starting_description = 1
else:
self.inside_td_element = 1
self.starting_description = 1
def end_td(self):
self.inside_td_element = 0
def handle_data(self, data):
if self.inside_td_element:
if self.starting_description:
self.descriptions.append(data)
self.starting_description = 0
else:
self.descriptions[-1] += data
def get_descriptions(self):
return self.descriptions
counter = 0
trlist = []
dtablelist = []
while counter < len(tablelist):
trsegment = tablelist[counter].findAll('tr')
trlist.append(trsegment)
strsegment = str(trsegment)
myparser = MyParser()
myparser.parse(strsegment)
sub = myparser.get_descriptions()
dtablelist.append(sub)
counter = counter + 1
ex = []
dtablelist = [s for s in dtablelist if s != ex]
So what I want to accomplish is take all the tables from an html document, then reprint them onto an Excel spreadsheet. So when I create trlist the output looks like this:
print trlist[1]
[<tr>
<td valign="top" width="25%"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT- SIZE: 10pt; FONT-FAMILY: times new roman; TEXT-DECORATION: underline"> </font></td>
<td valign="top" width="25%">
<div style="DISPLAY: block; MARGIN-LEFT: 0pt; TEXT-INDENT: 0pt; MARGIN-RIGHT: 0pt" align="center"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: times new roman; TEXT-DECORATION: underline">Title of each class</font></div>
</td>
<td valign="top" width="25%" style="TEXT-ALIGN: center"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: times new roman; TEXT-DECORATION: underline">Name of exchange</font></td>
<td valign="top" width="25%" style="TEXT-ALIGN: center"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: times new roman; TEXT-DECORATION: underline"> </font></td>
</tr>, <tr>
<td valign="top" width="25%"><font style="DISPLAY: inline; FONT-SIZE: 10pt; FONT-FAMILY: times new roman"> </font></td>
<td valign="top" width="25%">
<div style="DISPLAY: block; MARGIN-LEFT: 0pt; TEXT-INDENT: 0pt; MARGIN-RIGHT: 0pt" align="center"><font style="DISPLAY: inline; FONT-SIZE: 10pt; FONT-FAMILY: times new roman"><font style="DISPLAY: inline; FONT-WEIGHT: bold">Common Stock, par value</font> </font></div>
</td>
<td valign="top" width="25%">
<div style="DISPLAY: block; MARGIN-LEFT: 0pt; TEXT-INDENT: 0pt; MARGIN-RIGHT: 0pt" align="center">
<div style="DISPLAY: block; MARGIN-LEFT: 0pt; TEXT-INDENT: 0pt; MARGIN-RIGHT: 0pt" align="center"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: times new roman"><font style="FONT-WEIGHT: bold"><font style="FONT-WEIGHT: bold">< <font style="FONT-WEIGHT: bold">NASDAQ Global Market</font></font></font></font></div>
</div>
</td>
<td valign="top" width="25%"><font style="DISPLAY: inline; FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: times new roman"> </font></td>
</tr>,...
As you can see each item in trlist is each individual row ( . . . ) of the table which is what I want. But when I run each trlist item through my sgmllib parser to retrieve the contents between the tags I get this output:
print dtablelist[1]
['\nTitle of each class\n', 'Name of exchange', '\nCommon Stock, par value\n', '\n\nNASDAQ Global Market\n\n', '\n$1.00 per share\n']
As you can see, the output is each of the contents as their own individual string, instead of a list of the contents of each table row (). So essentially I want the output:
[['\nTitle of each class\n', 'Name of exchange'], ['\nCommon Stock, par value\n', '\n\nNASDAQ Global Market\n\n'], ['\n$1.00 per share\n']]
Is it because I have to turn trlist into a string before I parse it with MyParser? Does anyone know any way around this, allowing me to parse lists within lists (aka Inception shit)?
Using lxml.html:
>>> import lxml.html
>>> data = ["<tr><td>test</td><td>help</td></tr>", "<tr><td>data1</td><td>data2</td></tr>"]
>>> [lxml.html.fromstring(tr).xpath(".//text()") for tr in data]
[['test', 'help'], ['data1', 'data2']]
And here is some more complete code. It stores the text in a list containing a list of tables, and each table has a list of tr's, and each tr has a list of all the text.
import urllib
import lxml.html
data = urllib.urlopen('http://www.sec.gov/Archives/edgar/data/8177/000114036111018563/form10k.htm').read()
tree = lxml.html.fromstring(data)
tables = []
for tbl in tree.iterfind('.//table'):
tele = []
tables.append(tele)
for tr in tbl.iterfind('.//tr'):
text = [e.strip() for e in tr.xpath('.//text()') if len(e.strip()) > 0]
tele.append(text)
print tables
Hope this helps, cheers!
If somebody is searching for a solution of the same problem but is using python 3:
You don't have to use an external library for parsing an HTML table even if you are using python 3. There the SGMLParser class was replaced by HTMLParser from html.parser. I've written code for a simple derived HTMLParser class. It is here in a github repo. It simply does remember the current scope of a <td>, <tr> or <table> tag. The advantages over using etree are that it runs correctly on non-xml-compliant html and that it doesn't use external libraries.
You can use that class (here named HTMLTableParser) the following way:
import urllib.request
from html_table_parser import HTMLTableParser
target = 'http://www.twitter.com'
# get website content
req = urllib.request.Request(url=target)
f = urllib.request.urlopen(req)
xhtml = f.read().decode('utf-8')
# instantiate the parser and feed it
p = HTMLTableParser()
p.feed(xhtml)
print(p.tables)
The output of this is a list of 2D-lists representing tables. It looks maybe like this:
[[[' ', ' Anmelden ']],
[['Land', 'Code', 'Für Kunden von'],
['Vereinigte Staaten', '40404', '(beliebig)'],
['Kanada', '21212', '(beliebig)'],
...
['3424486444', 'Vodafone'],
[' Zeige SMS-Kurzwahlen für andere Länder ']]]