Develop Reference

Remove "encoding" attribute from XML in Python

I am using python to do some conditional changes to an XML document. The incoming document has <?xml version="1.0" ?> at the top.
I'm using xml.etree.ElementTree.
How I'm parsing the changed XMl:
filter_update_body = ET.tostring(root, encoding="utf8", method="xml")
The output has this at the top:
<?xml version='1.0' encoding='utf8'?>
The client wants the "encoding" tag removed but if I remove it then it either doesn't include the line at all or it puts in encoding= 'us-ascii'
Can this be done so the output matches: <?xml version="1.0" ?>?
(I don't know why it matters honestly but that's what I was told needed to happen)

As pointed out in this answer there is no way to make ElementTree omit the encoding attribute. However, as #James suggested in a comment, it can be stripped from the resulting output like this:
filter_update_body = ET.tostring(root, encoding="utf8", method="xml")
filter_update_body = filter_update_body.replace(b"encoding='utf8'", b"", 1)
The b prefixes are required because ET.tostring() will return a bytes object if encoding != "unicode". In turn, we need to call bytes.replace().
With encoding = "unicode" (note that this is the literal string "unicode"), it will return a regular str. In this case, the bs can be omitted. We use good old str.replace().
It's worth noting that the choice between bytes and str also affects how the XML will eventually be written to a file. A bytes object should be written in binary mode, a str in text mode.

parsing XML string with encoding block at the beginning in python with etree & LXML [duplicate]

This question already has answers here:
parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml)
(3 answers)
Closed 2 years ago.
I have to parse XML files that start as such:
xml_string = '''<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<annotationStandOffs xmlns="http://www.tei-c.org/ns/1.0">
<standOff>
...
</standOff>
</annotationStandOffs>
'''
The following code will only fly if I eliminate the first line of the above shown string:
import xml.etree.ElementTree as ET
from lxml import etree
parser = etree.XMLParser(resolve_entities=False,strip_cdata=False,recover=True)
XML_tree = etree.XML(xml_string,parser=parser)
Otherwise I get the error:
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

As the error indicates, the encoding part of the XML declaration is meant to provide the necessary information about how to convert bytes (e.g. as read from a file) into string. It doesn't make sense when the XML already is a string.
Some XML parsers silently ignore this information when parsing from a string. Some throw an error.
So since you're pasting XML into a string literal in Python source code, it would only make sense to remove the declaration yourself while you're editing the Python file.
The other, not so smart option would be to use a byte string literal b'''...''', or to encode the string into a single-byte encoding at run-time '''...'''.encode('windows-1252'). But this opens another can of worms. When your Python file encoding (e.g. UTF-8) clashes the alleged XML encoding from your copypasted XML (e.g. UTF-16), you'll get more interesting errors.
Long story short, don't do that. Don't copypaste XML into Python source code without taking the XML declaration out. And don't try to "fix" it by run-time string encode() tomfoolery.
The opposite is also true. If you have bytes (e.g. read from a file in binary mode, or from a network socket) then give those bytes to the XML parser. Don't manually decode() them into string first.

read XML file with Python

I'm trying to write a plugin to read data from XML file
inside test.xml there is:
<data>
<items>
<item test1="Arabic Words"></item>
<item test2="English Words"></item>
</items>
</data>
and the code is :
# coding: utf-8
from xml.dom import minidom
xmldoc = minidom.parse('test.xml')
itemlist = xmldoc.getElementsByTagName('item')
test1 = itemlist[0].attributes['test1'].value
test2 = itemlist[1].attributes['test2'].value
print(test1)
print(test2)
But I encounter a problem with coding: I can't set it to utf-8.
How can I make minidom interpret files with UTF-8 encoding?

Typically, valid XML begins with a XML pseudotag, containing the encoding:
<?xml version="1.0" encoding="UTF-8"?>
...
minidom should respect that; if your file has such a tag but isn't interpreted as UTF-8, you should file a bug against minidom; but I'd generally expect that your files simply don't contain this line.
You can use
minidom.parseString("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + open("file.xml","r").read())
to work around that (but I recommend fixing your XML files).

Either use encode/decode functions or the import codecs.
Example:
x = 'abcd'
y = x.encode('utf-8')
y.decode('utf-8')
Just use encoding/decoding and use minidom to parse a string instead of passing a file name.

XML parser, recover=True?

I'm trying to parse some XML, however I get an error message.
After looking around a little I suspect it is due to some kind of special character in the source text and a (recover=True) should be placed in my parser line.
However I do not know the exact location for this.
Could someone have a look?
for name in newlist:
tree = ET.parse(loc + name)
root = tree.getroot()
for post in root.findall('post'):
text = post.text
text = text.strip()
posts.append(text)
The error I get is:
ParseError: not well-formed (invalid token): line 103, column 225

im not familar with python but I've had issues like this using c#. It might be because the xml isnt formatted properly. Normally the first line of the xml file will contain something like
<?xml version="1.0" encoding="UTF-8" ?>
the version and encoding is important as it tells the parser what characters are allowed. UTF-8 is the default but sometimes the xml file will contain non ascii characters causing this to go crazy. Changing the encoding to UTF-16 sometimes fixes this.
Good luck

Forcing encoding on bad XML files with ElementTree

A big set of XML files have the wrong encoding defined. It should be utf-8 but the content has latin-1 characters all over the place. What's the best way to parse this content?
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
Edit: this is happening with Adobe InDesign IDML files, it seems the "Content" text has latin-1 but the rest could be utf-8. I'm favoring normal parsing with utf-8, then reencode the Unicode text chunks in Content to utf-8 and then re-parsing with latin-1. What a mess.
ಠ_ಠ

You can override the encoding specified in the XML when you parse it:
class xml.etree.ElementTree.XMLParser(html=0, target=None, encoding=None)
Element
structure builder for XML source data,
based on the expat parser. html are
predefined HTML entities. This flag is
not supported by the current
implementation. target is the target
object. If omitted, the builder uses
an instance of the standard
TreeBuilder class. encoding 1 is
optional. If given, the value
overrides the encoding specified in
the XML file.
docs

Don't try to deal with encoding problems during parse, but pre-process the offending file(s).