Develop Reference

Python XML Parser: junk after document element

I'm learning Python at work. I've got a large XML file with data similar to this:
testData3.xml File
<r><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c></c><c></c><c>something1</c><c>something1</c></r>
<r><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c></c><c></c><c>something2</c><c>something2</c></r>
I have copied an XML parser out of one of my Python books that works in gathering the data when the data file contains only one line. As soon as I add a second line of data, the script fails when it runs.
Python script that I'm running (xmlReader.py):
from xml.dom.minidom import parse, Node
xmltree = parse('testData3.xml')
for node1 in xmltree.getElementsByTagName('c'):
for node2 in node1.childNodes:
if node2.nodeType == Node.TEXT_NODE:
print(node2.data)
I'm looking for some help on how to write the loop so that my xmlReader.py continues through the entire file instead of just one line. I get the following errors when I run this script:
Errors during execution:
xxxx#xxxx:~/xxxx/xxxx> python xmlReader.py
Traceback (most recent call last):
File "xmlReader.py", line 2, in <module>
xmltree = parse('testData3.xml')
File "/usr/lib64/python2.6/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
return expatbuilder.parse(file)
File "/usr/lib64/python2.6/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
result = builder.parseFile(fp)
File "/usr/lib64/python2.6/site-packages/_xmlplus/dom/expatbuilder.py", line 207, in parseFile
parser.Parse(buffer, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0
xxxx#xxxx:~/xxxx/xxxx>

The problem is that your example data is not valid XML. A valid XML document should have a single root element; this is true for a single line of the file, where <r> is the root element, but not true when you add a second line, because each line is contained within a separate <r> element, but there is no global parent element in the file.
Either construct valid XML, for example:
<root>
<r><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c>something1</c><c></c><c></c><c>something1</c><c>something1</c></r>
<r><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c>something2</c><c></c><c></c><c>something2</c><c>something2</c></r>
</root>
or parse the file line by line:
from xml.dom.minidom import parseString
f = open('testData3.xml'):
for line in f:
xmltree = parseString(line)
...
f.close()

Why doesn't Python lxml take my xml?

I'm using the Python lxml library to parse my xml, but I'm having a hard time parsing one specific text. Checkout the following code:
>>> print type(raw_text_xml)
<type 'unicode'>
>>> from lxml import etree
>>> article_xml_root = etree.fromstring(raw_text_xml, parser)
Traceback (most recent call last):
File "<input>", line 1, in <module>
article_xml_root = etree.fromstring(raw_text_xml, parser)
File "lxml.etree.pyx", line 3032, in lxml.etree.fromstring (src/lxml/lxml.etree.c:68121)
File "parser.pxi", line 1786, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:102470)
File "parser.pxi", line 1667, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:101229)
File "parser.pxi", line 1035, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:96139)
File "parser.pxi", line 582, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:91290)
File "parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:92476)
File "parser.pxi", line 622, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:91772)
XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1
so it says the first character is not a <, which by inspection is true:
>>> print raw_text_xml[:20]
ďťż<?xml version="1.
it has 3 weird characters in front of the xml. So to clean these I tried the following:
>>> article_xml_root = etree.fromstring(raw_text_xml[3:], parser)
Traceback (most recent call last):
File "<input>", line 1, in <module>
article_xml_root = etree.fromstring(raw_text_xml[3:], parser)
File "lxml.etree.pyx", line 3032, in lxml.etree.fromstring (src/lxml/lxml.etree.c:68121)
File "parser.pxi", line 1781, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:102435)
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
And now it suddenly complains about it being a unicode string with encoding declaration, while if you look all the way up to my first line of code, it was Unicode all along.
Does anybody know why after slicing it suddenly gives a whole different error? And most importantly, does anybody know how I can solve this?

why after slicing it suddenly gives a whole different error?
Because after the slicing the first error vanishes and the parsing can progress until the second one is found.
And most importantly, does anybody know how I can solve this?
Maybe the error message is right (it happens) and you can solve it by converting the unicode to bytes. I guess that's better than removing the encoding declaration.
raw_text_xml.encode('utf8')
Or instead of 'utf8' whatever encoding is declared in the xml fragment.

The first error was caused by wrong characters. Once you have fixed it, you fall in second which is that your raw_text_xml is unicode.
You can know what will be a proper encoding (ASCII, latin1, utf8, ...). I cannot without seeing the actual content.
Assuming it is the content of encoding variable, you should be able to do:
article_xml_root = etree.fromstring(raw_text_xml.encode(encoding), parser)
(but I strongly advice you to first control what shows print raw_text_xml[3:160] ...)

Where ever you decoded that original Unicode, it was done incorrectly. It looks like iso-8859-2, where it was originally UTF-8 with BOM signature. The following backs out the bad decoding and re-decodes correctly:
>>> s.encode('iso-8859-2').decode('utf-8-sig')
'<?xml version="1.'

Central way to filter invalid unicode chars in lxml?

It is common knowledge that certain character ranges aren't allowed in XML documents. I'm aware of solutions to filter those characters out (like [1], [2]).
Going with the Don't Repeat Yourself principle, I would prefer to implement one of these solutions in one central point – right now, I have to sanitize any potentially unsafe text before it is fed to lxml. Is there a way to achieve this, e.g. by subclassing a lxml filter class, catching some exceptions, or setting a configuration switch?
Edit: To hopefully clarify this question a bit, here a sample code:
from lxml import etree
root = etree.Element("root")
root.text = u'\uffff'
root.text += u'\ud800'
print(etree.tostring(root))
root.text += '\x02'.decode("utf-8")
Executing this gives the result
<root>￿?</root>
Traceback (most recent call last):
File "[…]", line 9, in <module>
root.text += u'\u0002'
File "lxml.etree.pyx", line 953, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:44956)
File "apihelpers.pxi", line 677, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:20273)
File "apihelpers.pxi", line 1395, in lxml.etree._utf8 (src/lxml/lxml.etree.c:26485)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
As you see, an exception is thrown for the 2 byte, but lxml happily escapes the other two out of range characters. The real trouble is that
s = "<root>￿?</root>"
root = etree.fromstring(s)
also throws an exception. This behavior is a bit unnerving in my opinion, especially because it produces invalid XML documents.
Turns out that this could be a 2 vs. 3 problem. With python3.4, the code above throws the exception
Traceback (most recent call last):
File "[…]", line 5, in <module>
root.text += u'\ud800'
File "lxml.etree.pyx", line 953, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:44971)
File "apihelpers.pxi", line 677, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:20273)
File "apihelpers.pxi", line 1387, in lxml.etree._utf8 (src/lxml/lxml.etree.c:26380)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed
The only remaining problem is the \uffff character, which lxml still happily accepts.

Just filter the string before you parse it in LXML: cleaning invalid characters from XML (gist by lawlesst).
I tried it with your code; it seems to work, save the fact that you need to change the gist to import re and sys!
from lxml import etree
from cleaner import invalid_xml_remove
root = etree.Element("root")
root.text = u'\uffff'
root.text += u'\ud800'
print(etree.tostring(root))
root.text += invalid_xml_remove('\x02'.decode("utf-8"))

Upper limit of fromstring function in ElementTree

I'm using Python 2.4 version on a Windows 32-bit PC. I'm trying to parse through a very large XML file using the ElementTree module. I downloaded version 1.2.6 of this module from effbot.org.
I followed the below code for my purpose:
import elementtree.ElementTree as ET
input = ''' 001 Chuck 009 Brent '''
stuff = ET.fromstring(input)
lst = stuff.findall("users/user")
print len(lst)
for item in lst:
print item.attrib["x"]
item = lst[0]
ET.dump(item)
item.get("x") # get works on attributes
item.find("id").text
item.find("id").tag
for user in stuff.getiterator('user'):
print "User" , user.attrib["x"]
ET.dump(user)
If the content of input is too large, more than 10,000 lines, the fromstring function raises an error (below). Can anyone help me out in rectifying this error?
This is the error generated:
Traceback (most recent call last): File "C:\Documents and Settings\hariprar\My Documents\My files\Python Try\xml_try1.py", line 16, in -toplevel- stuff = ET.fromstring(input) File "C:\Python24\Lib\site-packages\elementtree\ElementTree.py", line 1012, in XML return api.fromstring(text) File "C:\Python24\Lib\site-packages\elementtree\ElementTree.py", line 182, in fromstring parser.feed(text) File "C:\Python24\Lib\site-packages\elementtree\ElementTree.py", line 1292, in feed self._parser.Parse(data, 0) ExpatError: not well-formed (invalid token): line 2445, column 39

Take a look at the iterparse function. It will let you parse your input incrementally rather than reading it into memory as one big chunk.
It's described here: http://effbot.org/zone/element-iterparse.htm

Wikipedia with Python

I have this very simple python code to read xml for the wikipedia api:
import urllib
from xml.dom import minidom
usock = urllib.urlopen("http://en.wikipedia.org/w/api.php?action=query&titles=Fractal&prop=links&pllimit=500")
xmldoc=minidom.parse(usock)
usock.close()
print xmldoc.toxml()
But this code returns with these errors:
Traceback (most recent call last):
File "/home/user/workspace/wikipediafoundations/src/list.py", line 5, in <module><br>
xmldoc=minidom.parse(usock)<br>
File "/usr/lib/python2.6/xml/dom/minidom.py", line 1918, in parse<br>
return expatbuilder.parse(file)<br>
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 928, in parse<br>
result = builder.parseFile(file)<br>
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 207, in parseFile<br>
parser.Parse(buffer, 0)<br>
xml.parsers.expat.ExpatError: syntax error: line 1, column 62<br>
I have no clue as I just learning python. Is there a way to get an error with more detail? Does anyone know the solution? Also, please recommend a better language to do this in.
Thank You,
Venkat Rao

The URL you're requesting is an HTML representation of the XML that would be returned:
http://en.wikipedia.org/w/api.php?action=query&titles=Fractal&prop=links&pllimit=500
So the XML parser fails. You can see this by pasting the above in a browser. Try adding a format=xml at the end:
http://en.wikipedia.org/w/api.php?action=query&titles=Fractal&prop=links&pllimit=500&format=xml
as documented on the linked page:
http://en.wikipedia.org/w/api.php