Develop Reference

Setting and accessing namespaces in python lxml

I am writing a script that processes a rdf:skos file with python3 and lxml:
I learnt that I need to pass to the findall procedure the namespaces that the XML mentions. (Ok, strange, since the XML files lists these in the header, so this seems like an unnecessary step but anyway).
When calling
for concept in root.findall('.//skos:Concept', namespaces=root.nsmap):
that works, because a root.nsmap is constructed by lxml.
But then later in my code I also need to perform a test on xml:lang
for pl in concept.findall(".//skos:prefLabel[#xml:lang='en']", namespaces=root.nsmap):
and here python tells me
SyntaxError: prefix 'xml' not found in prefix map
Ok, true, in my skos file there is no extra declaration for the xml namespace. So I try to add it to the root.nsmap dict
root.nsmap['xml'] = "http://www.w3.org/XML/1998/namespace"
but that too doesn't work
nsmap = {'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 'uneskos': 'http://purl.org/umu/uneskos#', 'iso-thes': 'http://purl.org/iso25964/skos-thes#', 'dcterms': 'http://purl.org/dc/terms/', 'skos': 'http://www.w3.org/2004/02/skos/core#', 'rdfs': 'http://www.w3.org/2000/01/rdf-schema#'}
Seems I am not allowed to modify the root.nsmap?
Anyone an idea how this is done? I have processed tons of XML in the past with Perl XML::Twig which is very very comfortable and I assmue, the Python community has (at least) similarly comfortable ways to do that ... but how?
Any hint appreciated.

Modifying root.nsmap has no effect. But you can create another dictionary and modify that one. Example:
from lxml import etree
doc = """
<root xmlns:skos="http://www.w3.org/2004/02/skos/core#">
<skos:prefLabel xml:lang='en'>FOO</skos:prefLabel>
<skos:prefLabel xml:lang='de'>BAR</skos:prefLabel>
</root>"""
root = etree.fromstring(doc)
nsmap = root.nsmap
nsmap["xml"] = "http://www.w3.org/XML/1998/namespace"
en = root.find(".//skos:prefLabel[#xml:lang='en']", namespaces=nsmap)
print(en.text)
Output:
FOO

XML Python Parsing Namespace [duplicate]

I have the following XML which I want to parse using Python's ElementTree:
<rdf:RDF xml:base="http://dbpedia.org/ontology/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns="http://dbpedia.org/ontology/">
<owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
<rdfs:label xml:lang="en">basketball league</rdfs:label>
<rdfs:comment xml:lang="en">
a group of sports teams that compete against each other
in Basketball
</rdfs:comment>
</owl:Class>
</rdf:RDF>
I want to find all owl:Class tags and then extract the value of all rdfs:label instances inside them. I am using the following code:
tree = ET.parse("filename")
root = tree.getroot()
root.findall('owl:Class')
Because of the namespace, I am getting the following error.
SyntaxError: prefix 'owl' not found in prefix map
I tried reading the document at http://effbot.org/zone/element-namespaces.htm but I am still not able to get this working since the above XML has multiple nested namespaces.
Kindly let me know how to change the code to find all the owl:Class tags.

You need to give the .find(), findall() and iterfind() methods an explicit namespace dictionary:
namespaces = {'owl': 'http://www.w3.org/2002/07/owl#'} # add more as needed
root.findall('owl:Class', namespaces)
Prefixes are only looked up in the namespaces parameter you pass in. This means you can use any namespace prefix you like; the API splits off the owl: part, looks up the corresponding namespace URL in the namespaces dictionary, then changes the search to look for the XPath expression {http://www.w3.org/2002/07/owl}Class instead. You can use the same syntax yourself too of course:
root.findall('{http://www.w3.org/2002/07/owl#}Class')
Also see the Parsing XML with Namespaces section of the ElementTree documentation.
If you can switch to the lxml library things are better; that library supports the same ElementTree API, but collects namespaces for you in .nsmap attribute on elements and generally has superior namespaces support.

Here's how to do this with lxml without having to hard-code the namespaces or scan the text for them (as Martijn Pieters mentions):
from lxml import etree
tree = etree.parse("filename")
root = tree.getroot()
root.findall('owl:Class', root.nsmap)
UPDATE:
5 years later I'm still running into variations of this issue. lxml helps as I showed above, but not in every case. The commenters may have a valid point regarding this technique when it comes merging documents, but I think most people are having difficulty simply searching documents.
Here's another case and how I handled it:
<?xml version="1.0" ?><Tag1 xmlns="http://www.mynamespace.com/prefix">
<Tag2>content</Tag2></Tag1>
xmlns without a prefix means that unprefixed tags get this default namespace. This means when you search for Tag2, you need to include the namespace to find it. However, lxml creates an nsmap entry with None as the key, and I couldn't find a way to search for it. So, I created a new namespace dictionary like this
namespaces = {}
# response uses a default namespace, and tags don't mention it
# create a new ns map using an identifier of our choice
for k,v in root.nsmap.iteritems():
if not k:
namespaces['myprefix'] = v
e = root.find('myprefix:Tag2', namespaces)

Note: This is an answer useful for Python's ElementTree standard library without using hardcoded namespaces.
To extract namespace's prefixes and URI from XML data you can use ElementTree.iterparse function, parsing only namespace start events (start-ns):
>>> from io import StringIO
>>> from xml.etree import ElementTree
>>> my_schema = u'''<rdf:RDF xml:base="http://dbpedia.org/ontology/"
... xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
... xmlns:owl="http://www.w3.org/2002/07/owl#"
... xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
... xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
... xmlns="http://dbpedia.org/ontology/">
...
... <owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
... <rdfs:label xml:lang="en">basketball league</rdfs:label>
... <rdfs:comment xml:lang="en">
... a group of sports teams that compete against each other
... in Basketball
... </rdfs:comment>
... </owl:Class>
...
... </rdf:RDF>'''
>>> my_namespaces = dict([
... node for _, node in ElementTree.iterparse(
... StringIO(my_schema), events=['start-ns']
... )
... ])
>>> from pprint import pprint
>>> pprint(my_namespaces)
{'': 'http://dbpedia.org/ontology/',
'owl': 'http://www.w3.org/2002/07/owl#',
'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
'rdfs': 'http://www.w3.org/2000/01/rdf-schema#',
'xsd': 'http://www.w3.org/2001/XMLSchema#'}
Then the dictionary can be passed as argument to the search functions:
root.findall('owl:Class', my_namespaces)

I've been using similar code to this and have found it's always worth reading the documentation... as usual!
findall() will only find elements which are direct children of the current tag. So, not really ALL.
It might be worth your while trying to get your code working with the following, especially if you're dealing with big and complex xml files so that that sub-sub-elements (etc.) are also included.
If you know yourself where elements are in your xml, then I suppose it'll be fine! Just thought this was worth remembering.
root.iter()
ref: https://docs.python.org/3/library/xml.etree.elementtree.html#finding-interesting-elements
"Element.findall() finds only elements with a tag which are direct children of the current element. Element.find() finds the first child with a particular tag, and Element.text accesses the element’s text content. Element.get() accesses the element’s attributes:"

To get the namespace in its namespace format, e.g. {myNameSpace}, you can do the following:
root = tree.getroot()
ns = re.match(r'{.*}', root.tag).group(0)
This way, you can use it later on in your code to find nodes, e.g using string interpolation (Python 3).
link = root.find(f"{ns}link")

This is basically Davide Brunato's answer however I found out that his answer had serious problems the default namespace being the empty string, at least on my python 3.6 installation. The function I distilled from his code and that worked for me is the following:
from io import StringIO
from xml.etree import ElementTree
def get_namespaces(xml_string):
namespaces = dict([
node for _, node in ElementTree.iterparse(
StringIO(xml_string), events=['start-ns']
)
])
namespaces["ns0"] = namespaces[""]
return namespaces
where ns0 is just a placeholder for the empty namespace and you can replace it by any random string you like.
If I then do:
my_namespaces = get_namespaces(my_schema)
root.findall('ns0:SomeTagWithDefaultNamespace', my_namespaces)
It also produces the correct answer for tags using the default namespace as well.

My solution is based on #Martijn Pieters' comment:
register_namespace only influences serialisation, not search.
So the trick here is to use different dictionaries for serialization and for searching.
namespaces = {
'': 'http://www.example.com/default-schema',
'spec': 'http://www.example.com/specialized-schema',
}
Now, register all namespaces for parsing and writing:
for name, value in namespaces.iteritems():
ET.register_namespace(name, value)
For searching (find(), findall(), iterfind()) we need a non-empty prefix. Pass these functions a modified dictionary (here I modify the original dictionary, but this must be made only after the namespaces are registered).
self.namespaces['default'] = self.namespaces['']
Now, the functions from the find() family can be used with the default prefix:
print root.find('default:myelem', namespaces)
but
tree.write(destination)
does not use any prefixes for elements in the default namespace.

Processing large XML file [duplicate]

I have the following XML which I want to parse using Python's ElementTree:
<rdf:RDF xml:base="http://dbpedia.org/ontology/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns="http://dbpedia.org/ontology/">
<owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
<rdfs:label xml:lang="en">basketball league</rdfs:label>
<rdfs:comment xml:lang="en">
a group of sports teams that compete against each other
in Basketball
</rdfs:comment>
</owl:Class>
</rdf:RDF>
I want to find all owl:Class tags and then extract the value of all rdfs:label instances inside them. I am using the following code:
tree = ET.parse("filename")
root = tree.getroot()
root.findall('owl:Class')
Because of the namespace, I am getting the following error.
SyntaxError: prefix 'owl' not found in prefix map
I tried reading the document at http://effbot.org/zone/element-namespaces.htm but I am still not able to get this working since the above XML has multiple nested namespaces.
Kindly let me know how to change the code to find all the owl:Class tags.

You need to give the .find(), findall() and iterfind() methods an explicit namespace dictionary:
namespaces = {'owl': 'http://www.w3.org/2002/07/owl#'} # add more as needed
root.findall('owl:Class', namespaces)
Prefixes are only looked up in the namespaces parameter you pass in. This means you can use any namespace prefix you like; the API splits off the owl: part, looks up the corresponding namespace URL in the namespaces dictionary, then changes the search to look for the XPath expression {http://www.w3.org/2002/07/owl}Class instead. You can use the same syntax yourself too of course:
root.findall('{http://www.w3.org/2002/07/owl#}Class')
Also see the Parsing XML with Namespaces section of the ElementTree documentation.
If you can switch to the lxml library things are better; that library supports the same ElementTree API, but collects namespaces for you in .nsmap attribute on elements and generally has superior namespaces support.

Here's how to do this with lxml without having to hard-code the namespaces or scan the text for them (as Martijn Pieters mentions):
from lxml import etree
tree = etree.parse("filename")
root = tree.getroot()
root.findall('owl:Class', root.nsmap)
UPDATE:
5 years later I'm still running into variations of this issue. lxml helps as I showed above, but not in every case. The commenters may have a valid point regarding this technique when it comes merging documents, but I think most people are having difficulty simply searching documents.
Here's another case and how I handled it:
<?xml version="1.0" ?><Tag1 xmlns="http://www.mynamespace.com/prefix">
<Tag2>content</Tag2></Tag1>
xmlns without a prefix means that unprefixed tags get this default namespace. This means when you search for Tag2, you need to include the namespace to find it. However, lxml creates an nsmap entry with None as the key, and I couldn't find a way to search for it. So, I created a new namespace dictionary like this
namespaces = {}
# response uses a default namespace, and tags don't mention it
# create a new ns map using an identifier of our choice
for k,v in root.nsmap.iteritems():
if not k:
namespaces['myprefix'] = v
e = root.find('myprefix:Tag2', namespaces)

Note: This is an answer useful for Python's ElementTree standard library without using hardcoded namespaces.
To extract namespace's prefixes and URI from XML data you can use ElementTree.iterparse function, parsing only namespace start events (start-ns):
>>> from io import StringIO
>>> from xml.etree import ElementTree
>>> my_schema = u'''<rdf:RDF xml:base="http://dbpedia.org/ontology/"
... xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
... xmlns:owl="http://www.w3.org/2002/07/owl#"
... xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
... xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
... xmlns="http://dbpedia.org/ontology/">
...
... <owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
... <rdfs:label xml:lang="en">basketball league</rdfs:label>
... <rdfs:comment xml:lang="en">
... a group of sports teams that compete against each other
... in Basketball
... </rdfs:comment>
... </owl:Class>
...
... </rdf:RDF>'''
>>> my_namespaces = dict([
... node for _, node in ElementTree.iterparse(
... StringIO(my_schema), events=['start-ns']
... )
... ])
>>> from pprint import pprint
>>> pprint(my_namespaces)
{'': 'http://dbpedia.org/ontology/',
'owl': 'http://www.w3.org/2002/07/owl#',
'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
'rdfs': 'http://www.w3.org/2000/01/rdf-schema#',
'xsd': 'http://www.w3.org/2001/XMLSchema#'}
Then the dictionary can be passed as argument to the search functions:
root.findall('owl:Class', my_namespaces)

I've been using similar code to this and have found it's always worth reading the documentation... as usual!
findall() will only find elements which are direct children of the current tag. So, not really ALL.
It might be worth your while trying to get your code working with the following, especially if you're dealing with big and complex xml files so that that sub-sub-elements (etc.) are also included.
If you know yourself where elements are in your xml, then I suppose it'll be fine! Just thought this was worth remembering.
root.iter()
ref: https://docs.python.org/3/library/xml.etree.elementtree.html#finding-interesting-elements
"Element.findall() finds only elements with a tag which are direct children of the current element. Element.find() finds the first child with a particular tag, and Element.text accesses the element’s text content. Element.get() accesses the element’s attributes:"

To get the namespace in its namespace format, e.g. {myNameSpace}, you can do the following:
root = tree.getroot()
ns = re.match(r'{.*}', root.tag).group(0)
This way, you can use it later on in your code to find nodes, e.g using string interpolation (Python 3).
link = root.find(f"{ns}link")

This is basically Davide Brunato's answer however I found out that his answer had serious problems the default namespace being the empty string, at least on my python 3.6 installation. The function I distilled from his code and that worked for me is the following:
from io import StringIO
from xml.etree import ElementTree
def get_namespaces(xml_string):
namespaces = dict([
node for _, node in ElementTree.iterparse(
StringIO(xml_string), events=['start-ns']
)
])
namespaces["ns0"] = namespaces[""]
return namespaces
where ns0 is just a placeholder for the empty namespace and you can replace it by any random string you like.
If I then do:
my_namespaces = get_namespaces(my_schema)
root.findall('ns0:SomeTagWithDefaultNamespace', my_namespaces)
It also produces the correct answer for tags using the default namespace as well.

My solution is based on #Martijn Pieters' comment:
register_namespace only influences serialisation, not search.
So the trick here is to use different dictionaries for serialization and for searching.
namespaces = {
'': 'http://www.example.com/default-schema',
'spec': 'http://www.example.com/specialized-schema',
}
Now, register all namespaces for parsing and writing:
for name, value in namespaces.iteritems():
ET.register_namespace(name, value)
For searching (find(), findall(), iterfind()) we need a non-empty prefix. Pass these functions a modified dictionary (here I modify the original dictionary, but this must be made only after the namespaces are registered).
self.namespaces['default'] = self.namespaces['']
Now, the functions from the find() family can be used with the default prefix:
print root.find('default:myelem', namespaces)
but
tree.write(destination)
does not use any prefixes for elements in the default namespace.

python xml xpath query using tag and attribute with ns

I must be doing something inherently wrong here, every example I've seen and search for on SO seems to suggest this would work.
I'm trying to use an XPath search with lxml etree library to parse a garmin tcx file:
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<TrainingCenterDatabase xmlns="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2 http://www.garmin.com/xmlschemas/TrainingCenterDatabasev2.xsd">
<Workouts>
<Workout Sport="Biking">
<Name>3P2 WK16 - 3</Name>
<Step xsi:type="Step_t">
<StepId>1</StepId>
<Name>[MP19]6:28-6:38</Name>
<Duration xsi:type="Distance_t">
<Meters>13000</Meters>
</Duration>
<Intensity>Active</Intensity>
<Target xsi:type="Speed_t">
<SpeedZone xsi:type="PredefinedSpeedZone_t">
<Number>2</Number>
</SpeedZone>
</Target>
</Step>
......
</Workout>
</Workouts>
</TrainingCenterDatabase>
I'd like to return the SpeedZone Element only where the type is PredefinedSpeedZone_t. I thought I'd be able to do:
root = ET.parse(open('file.tcx'))
xsi = {'xsi': 'http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2'}
for speed_zone in root.xpath(".//xsi:SpeedZone[#xsi:type='PredefinedSpeedZone_t']", namespaces=xsi):
print speed_zone
Though this doesn't seem to be the case. I've tried lots of combinations of removing/adding namespaces and to no avail. If I remove the attribute search and leave it as ".//xsi:SpeedZone" then this does return:
<Element {http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2}SpeedZone at 0x2595188>
as I'd expect.
I guess I could do it inside the for loop but it just feels like it should be possible on one line!

I'm a bit late, but the other answers are confusing IMHO.
In the Python code in the question and in the two other answers, the xsi prefix is bound to the http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2 URI. But in the XML document with the Garmin data, xsi is bound to http://www.w3.org/2001/XMLSchema-instance.
Since there are two namespaces at play here, I think the following code gives a clearer picture of what's going on. The namespace associated with the tcd prefix is the default namespace.
from lxml import etree
NSMAP = {"tcd": "http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2",
"xsi": "http://www.w3.org/2001/XMLSchema-instance"}
root = etree.parse('file.tcx')
for speed_zone in root.xpath(".//tcd:SpeedZone[#xsi:type='PredefinedSpeedZone_t']",
namespaces=NSMAP):
print speed_zone
Output:
<Element {http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2}SpeedZone at 0x25b7e18>

One way to workaround this is to avoid specifying the attribute name and use *:
.//xsi:SpeedZone[#*='PredefinedSpeedZone_t']
Another option (not that awesome as previous one) is to actually get all the SpeedZone tags and check for the attribute value in the loop:
attribute_name = '{%s}type' % root.nsmap['xsi']
for speed_zone in root.xpath(".//xsi:SpeedZone", namespaces=xsi):
if speed_zone.attrib.get(attribute_name) == 'PredefinedSpeedZone_t':
print speed_zone
Hope that helps.

If all else fails you can still use
".//xsi:SpeedZone[#*[name() = 'xsi:type' and . = 'PredefinedSpeedZone_t']]"
Using name() is not as nice as directly addressing the namespaced attribute, but at least etree understands it.

Is there any Python XML parser that was designed with humans in mind?

I like Python, but I don't want to write 10 lines just to get an attribute from an element. Maybe it's just me, but minidom isn't that mini. The code I have to write in order to parse something using it looks a lot like Java code.
Is there something that is more user-friendly ? Something with overloaded operators, and which maps elements to objects?
I'd like to be able to access this :
<root>
<node value="30">text</node>
</root>
as something like this :
obj = parse(xml_string)
print obj.node.value
and not using getChildren or some other methods like that.

You should take a look at ElementTree. It's not doing exactly what you want but it's a lot better then minidom. If I remember correctly, starting from python 2.4, it's included in the standard libraries. For more speed use cElementTree. For more more speed (and more features) you can use lxml (check the objectify API for your needs/approach).
I should add that BeautifulSoup do partly what you want. There's also Amara that have this approach.

I actually wrote a library that does things exactly the way you imagined it. The library is called "xe" and you can get it from: http://home.avvanta.com/~steveha/xe.html
xe can import XML to let you work with the data in an object-oriented way. It actually uses xml.dom.minidom to do the parsing, but then it walks over the resulting tree and packs the data into xe objects.
EDIT: Okay, I went ahead and implemented your example in xe, so you can see how it works. Here are classes to implement the XML you showed:
import xe
class Node(xe.TextElement):
def __init__(self, text="", value=None):
xe.TextElement.__init__(self, "node", text)
if value is not None:
self.attrs["value"] = value
class Root(xe.NestElement):
def __init__(self):
xe.NestElement.__init__(self, "root")
self.node = Node()
And here is an example of using the above. I put your sample XML into a file called "example.xml", but you could also just put it into a string and pass the string.
>>> root = Root()
>>> print root
<root/>
>>> root.import_xml("example.xml")
<Root object at 0xb7e0c52c>
>>> print root
<root>
<node value="30">text</node>
</root>
>>> print root.node.attrs["value"]
30
>>>
Note that in this example, the type of "value" will be a string. If you really need attributes of another type, that's possible too with a little bit of work, but I didn't bother for this example. (If you look at PyFeed, there is a class for OPML that has an attribute that isn't text.)

I had same need for simple xml parser and after a long time spent on checking different libraries I found xmltramp.
Based on your example xml:
import xmltramp
xml_string = """<root>
<node value="30">text</node>
</root>"""
obj = xmltramp.parse(xml_string)
print obj.node('value') # 30
print str(obj.node) # text
I didn't found anything more user-friendly.

I spent a fair bit of time going through the examples provided above and through the repositories listed on pip.
The easiest (and most Pythonic) way of parsing XML that I have found so far has been XMLToDict - https://github.com/martinblech/xmltodict
The example from the documentation available at GitHub above is copy-pasted below; It's made life VERY simple and EASY for me a LOT of times;
>>> doc = xmltodict.parse("""
... <mydocument has="an attribute">
... <and>
... <many>elements</many>
... <many>more elements</many>
... </and>
... <plus a="complex">
... element as well
... </plus>
... </mydocument>
... """)
>>>
>>> doc['mydocument']['#has']
u'an attribute'
>>> doc['mydocument']['and']['many']
[u'elements', u'more elements']
>>> doc['mydocument']['plus']['#a']
u'complex'
>>> doc['mydocument']['plus']['#text']
u'element as well'
It works really well and gave me just what I was looking for. However, if you're looking at reverse transformations, that is an entirely different matter altogether.