Develop Reference

Python xml.etree.ElemenTree, getting HTML entities

I am trying to analyze xml data, and encountered an issue with regard to HTML entities when I use
import xml.etree.ElementTree as ET
tree = ET.parse(my_xml_file)
root = tree.getroot()
for regex_rule in root.findall('.//regex_rule'):
print(regex_rule.get('input')) #this ".get()" method turns < into <, but I want to get < as written
print(regex_rule.get('input') == "(?<!\S)hello(?!\S)") #prints out false because ElementTree's get method turns < into < , is that right?
And here is the xml file contents:
<rules>
<regex_rule input="(?<!\S)hello(?!\S)" output="world"/>
</rules>
I would appreciate if anybody can direct me to getting the string as is from the xml attribute for the input, without converting
<
into
<

xml.etree.ElementTree is doing exactly the standards-compliant thing, which is to decode XML character entities with the understanding that they do in fact encode the referenced character and should be interpreted as such.
The preferred course of action if you do need to encode the literal < is to change your input file to use &lt; instead (i.e. we XML-encode the &).
If you can't change your input file format then you'll probably need to use a different module, or write your own parser: xml.etree.ElementTree translates entities well before you can do anything meaningful with the output.

How to iterate through all XML Elements and apply logic to each Element's value with ElementTree for Python

I am currently trying to apply logic to Element values in a XML file. Specifically I am trying to encode all the values to UTF-8 while not touching any of the element names/attributes themselves.
Here is the sample XML:
<?xml version="1.0"?>
<sd_1>
<sd_2>
<sd_3>\311 is a fancy kind of E</sd_3>
</sd_2>
</sd_1>
Currently I have tried 3 methods to achieve this with no success:
First I tried the looping through each element retrieving the values with .text and using .parse:
import xml.etree.ElementTree as ET
et = ET.parse('xml/test.xml')
for child in et.getroot():
for core in child:
core_value = str(core.text)
core.text = core_value.encode('utf-8')
et.write('output.xml')
This results in an XML file that does not have the text \311 altered correctly, it just stays as it is.
Next I tried the .iterparse with cElementTree to no avail:
import xml.etree.cElementTree as etree
xml_file_path = 'xml/test.xml'
with open(xml_file_path) as xml_file:
tree = etree.iterparse(xml_file)
for items in tree:
for item in items:
print item.text
etree.write('output1.xml')
This results in:
"...print item.text\n', "AttributeError: 'str' object has no attribute 'text'..."
Not sure what I am doing wrong there, I have seen multiple examples with the same arrangement, but when I print through the elements without the .text I see the tuple with a string value of 'end' at the start and I think that is causing the issue with this method.
How do I properly iterate through my elements, and without specifying the element names e.g. .findall(), apply logic to the values housed in each Element so that when I write the xml to file it saves the changes made when the program was iterating through element values?

Is this what you are looking for?
import xml.etree.ElementTree as ET
et = ET.parse('xml/test.xml')
for child in et.getroot():
for core in child:
core_value = str(core.text)
core.text = core_value.decode('unicode-escape')
et.write('output.xml')

This is an interesting question. Let's focus on the first method you proposed, as that should be a totally fine way to approach this problem. When I print out the lines one by one, here is what I get:
>>> core_value
'\\311 is a fancy kind of E'
What happened for me is that the character was read as a literal '\', which must be escaped to be printed as such. If we change the escaped character (\\) to a non-escaped character (\), we get the following:
>>> cv = core_value.replace('\\311','\311')
'\xc9 is a fancy kind of E'
>>> print cv
É is a fancy kind of E
The weird piece here is that you don't know when in the original file \311 is "supposed to be" one character or four. If you know for a fact that those will all be one character, you can write some vile code based on this answer:
Python Unicode, have unicode number in normal string, want to print unicode
To transorm all of the things that come after a \ into the correct unicode characters and delete the \.

How to handle escaped string in lxml with Python

I'm trying to use lxml to help me parse some XML files and output it. However,there are some special characters in the XML file. I don't want to replace it because it is too complicated to escape it and unescape it. Also I can't force the others to produce a well-formed XML.
Is there any way Python can let me handle the non-well-formed XML with lxml?
I can read it in properly:
parser = etree.XMLParser(recover=True)
root = etree.parse(sys.argv[1],parser=parser)
But when I want to print the element text, it can only print the content until the special character occurs.
for element in root.iter("content"):
print("%s - %s attr - %s" % (element.tag, element.text, element.get("name")))

lxml does unescaping for you transparently. So you could try to fix invalid characters in the input first and then feed the result to lxml. For example, you could try a simple regex-based solution to escape invalid characters.

A popular option to deal with less-than-perfect markup language files in Python is to use Beautiful Soup. It can use many parsers, including lxml.
Can you post some of the XML that's giving you problems?

How can I disable 'output escaping' in minidom

I'm trying to build an xml document from scratch using xml.dom.minidom. Everything was going well until I tried to make a text node with a ® (Registered Trademark) symbol in. My objective is for when I finally hit print mydoc.toxml() this particular node will actually contain a ® symbol.
First I tried:
import xml.dom.minidom as mdom
data = '®'
which gives the rather obvious error of:
File "C:\src\python\HTMLGen\test2.py", line 3
SyntaxError: Non-ASCII character '\xae' in file C:\src\python\HTMLGen\test2.py on line 3, but no encoding declared; see http://www.python.or
g/peps/pep-0263.html for details
I have of course also tried changing the encoding of my python script to 'utf-8' using the opening line comment method, but this didn't help.
So I thought
import xml.dom.minidom as mdom
data = '®' #Both accepted xml encodings for registered trademark
data = '®'
text = mdom.Text()
text.data = data
print data
print text.toxml()
But because when I print text.toxml(), the ampersands are being escaped, I get this output:
®
®
My question is, does anybody know of a way that I can force the ampersands not to be escaped in the output, so that I can have my special character reference carry through to the XML document?
Basically, for this node, I want print text.toxml() to produce output of ® or ® in a happy and cooperative way!
EDIT 1:
By the way, if minidom actually doesn't have this capacity, I am perfectly happy using another module that you can recommend which does.
EDIT 2:
As Hugh suggested, I tried using data = u'®' (while also using data # -*- coding: utf-8 -*- Python source tags). This almost helped in the sense that it actually caused the ® symbol itself to be outputted to my xml. This is actually not the result I am looking for. As you may have guessed by now (and perhaps I should have specified earlier) this xml document happens to be an HTML page, which needs to work in a browser. So having ® in the document ends up causing rubbish in the browser (® to be precise!).
I also tried:
data = unichr(174)
text.data = data.encode('ascii','xmlcharrefreplace')
print text.toxml()
But of course this lead to the same origional problem where all that happens is the ampersand gets escaped by .toxml().
My ideal scenario would be some way of escaping the ampersand so that the XML printing function won't "escape" it on my behalf for the document (in other words, achieving my original goal of having ® or ® appear in the document).
Seems like soon I'm going to have to resort to regular expressions!
EDIT 2a:
Or perhaps not. Seems like getting my html meta information correct <META http-equiv="Content-Type" Content="text/html; charset=UTF-8"> could help, but I'm not sure yet how this fits in with the xml structure...

Two options that work, one with the escaping ® and the other without. It's not really obvious why you want escaping ... it's 6 bytes instead of the 2 or 3 bytes for non-CJK characters.
import xml.dom.minidom as mdom
text = mdom.Text()
# Start with unicode
text.data = u'\xae'
f = open('reg1.html', 'w')
f.write("header saying the file is ascii")
uxml = text.toxml()
bxml = uxml.encode('ascii', 'xmlcharrefreplace')
f.write(bxml)
f.close()
f = open('reg2.html', 'w')
f.write("header saying the file is UTF-8")
xml = text.toxml(encoding='UTF-8')
f.write(xml)
f.close()

If I understand correctly, what you really want is to be able to create a text node from a unicode object (e.g. u'®' or u'\u00ae') and then have toxml() output unicode characters encoded as entities (e.g. ®). Looking at the source of minidom.py, however, it seems that minidom doesn't support entity encoding on output except the special cases of &, ", < and >.
You also ask about alternative modules that could help, however. There are several possible candidates, but ElementTree (xml.etree) seems to do the appropriate encoding. For example, if you take the first example from this blog post by Doug Hellmann but replace:
child_with_tail.text = 'This child has regular text.'
... with:
child_with_tail.text = u'This child has regular text \u00ae.'
... and run the script, you should see the output contains:
This child has regular text®.
You could also use the lxml implementation of ElementTree in that example just by replacing the import statement with:
from lxml.etree import Element, SubElement, Comment, tostring
Update: the alternative answer from John Machin takes the nice approach of running .encode('ascii', 'xmlcharrefreplace') on the output from minidom's toxml(), which converts any non-ASCII characters to their equivalent XML numeric character references.

Default unescape:
from xml.sax.saxutils import unescape
unescape("< & >")
The result is,
'< & >'
And, unescape more:
unescape("&apos; "", {"&apos;": "'", """: '"'})
Check details here, https://wiki.python.org/moin/EscapingXml

Cleaning an XML file in Python before parsing

I'm using minidom to parse an xml file and it threw an error indicating that the data is not well formed. I figured out that some of the pages have characters like ไอเฟล &, causing the parser to hiccup. Is there an easy way to clean the file before I start parsing it? Right now I'm using a regular expressing to throw away anything that isn't an alpha numeric character and the </> characters, but it isn't quite working.

Try
xmltext = re.sub(u"[^\x20-\x7f]+",u"",xmltext)
It will get rid of everything except 0x20-0x7F range.
You may start from \x01, if you want want to keep control characters like tab, line breaks.
xmltext = re.sub(u"[^\x01-\x7f]+",u"",xmltext)

Take a look at µTidyLib, a Python wrapper to TidyLib.

If you do need the data with the strange characters you could, in stead of just stripping them, convert them to codes the XML parser can understand.
You could have a look at the unicodedata package, especially the normalize method.
I haven't used it myself, so I can't tell you all that much, but you could ask again here on SO if you decide you're going to convert and keep that data.
>>> import unicodedata
>>> unicodedata.normalize("NFKD" , u"ไภเฟล &")
u'a\u03001\u201ea\u0300 \u0327 a\u03001\u20aca\u0300 \u0327Y\u0308a\u0300 \u0327\xa5 &'

It looks like you're dealing with data which are saved with some kind of encoding "as if" they were ASCII. XML file should normally be UTF8, and SAX (the underlying parser used by minidom) should handle that, so it looks like something's wrong in that part of the processing chain. Instead of focusing on "cleaning up" I'd first try to make sure the encoding is correct and correctly recognized. Maybe a broken XML directive? Can you edit your Q to show the first few lines of the file, especially the <?xml ... directive at the very start?

I'd throw out all non-ASCII characters which can be identified by having the 8th bit (0x80) set (128 .. 255 respectively 0x80 .. 0xff).
You could read in the file into a Python string named old_str
Then perform a filter call in conjunction with a lambda statement:
new_str = filter(lambda x: x in string.ascii_letters, old_str)
Parse new_str
Many ways exist to accomplish stripping non-ASCII characters from a string.
This question might be related: How to check if a string in Python is in ASCII?