Develop Reference

Parse xsi:type in XML with ElementTree in Python

I'm trying to connect to a RESTful API and I'm hacing problems when building the XML request, for that I'm using Elementree library.
I have an example of the XML I have to send in the request. From that example a build a model and then write the different attributes by code. But the output XML is not exactly like the example I was given and I'm unable to connect to the API.
This is the example I have:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<GetLoc xmlns="http://abc/Getloc">
<request>
<Access>
<string xmlns="http://bcd/Arrays"></string>
</Access>
<Details xsi:type="Request">
<Postcode ></Postcode >
</Details>
<UserConsent>Yes</UserConsent>
</request>
</GetLoc>
</soap:Body>
</soap:Envelope>
This is my code:
tree = ET.parse('model.xml')
root = tree.getroot()
ns = {'loc':'http://abc/Getloc',\
'arr':http://bcd/Arrays',\
'soapenv':'http://schemas.xmlsoap.org/soap/envelope/', \
'xsi':"http://www.w3.org/2001/XMLSchema-instance", \
xsd': "http://www.w3.org/2001/XMLSchema"}
tree.find('.//arr:string', ns).text = 'THC'
tree.find('.//Postcode ', ns).text = '15478'
This is the output XML (SOAP):
<ns0:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://abc/Getloc" xmlns:ns2="http://bcd/Arrays" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns0:Body>
<ns1:GetLoc >
<ns1:request>
<ns1:Access>
<ns2:string>THC</ns2:string>
</ns1:Access>
<ns1:Details xsi:type="Request">
<ns1:Postcode >15478</ns1:Postcode >
</ns1:Details>
<ns1:UserConsent>Yes</ns1:UserConsent>
</ns1:request>
</ns1:GetLoc >
</ns0:Body>
</ns0:Envelope>
With the example (first above) I have no problem when connecting to the API. However with the second one I get and error:
" status="Service Not Found. The request may have been sent to an invalid URL, or intended for an unsupported operation." xmlns:l7="http://www.layer7tech.com/ws/policy/fault"/>"
Both XML are sent to the same URL with the same headers and auth. I see both XML equivalent so I was expecting same behavior. I don't understand why it isn't working.
EDIT: The output XML needs to be like
<ns0:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://abc/Getloc" xmlns:ns2="http://bcd/Arrays" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns0:Body>
<ns1:GetLoc >
<ns1:request>
<ns1:Access>
<ns2:string>THC</ns2:string>
</ns1:Access>
<ns1:Details xsi:type="ns1:Request">
<ns1:Postcode >15478</ns1:Postcode >
</ns1:Details>
<ns1:UserConsent>Yes</ns1:UserConsent>
</ns1:request>
</ns1:GetLoc >
</ns0:Body>
</ns0:Envelope>
But I don't know hoy to change the code to get: xsi:type="ns1:Request"

Finally I found the solution myself.
The solution is in here (an incredibly complete article), since I was already using ElementTree. You may find other solutions like using lxml library.
So, for ElementTree I just need to use my own parser instead of the standard ElementTree.parse('file.xml').
The xsi attribute name is handled by the parser, but the parser doesn’t know that the attribute happens to contain a qualified name as well, so it leaves it as is. To be able to handle such a format, you can use a custom parser that knows how to handle certain attributes and elements, or keep track of the prefix mapping for each element.
To do the latter, you can use the iterparse parser, and ask it to report “start-ns” and “end-ns” events. The following snippet adds an ns_map attribute to each element which contains the prefix/URI mapping that applies to that specific element:
def parse_map(file):
events = "start", "start-ns", "end-ns"
root = None
ns_map = []
for event, elem in ET.iterparse(file, events):
if event == "start-ns":
ns_map.append(elem)
elif event == "end-ns":
ns_map.pop()
elif event == "start":
if root is None:
root = elem
elem.ns_map = dict(ns_map)
return ET.ElementTree(root)

lxml: How do I search for fields without adding a xmlns (localhost) path to each search term?

I'm trying to locate fields in a SOAP xml file using lxml (3.6.0)
...
<soap:Body>
<Request xmlns="http://localhost/">
<Test>
<field1>hello</field1>
<field2>world</field2>
</Test>
</Request>
</soap:Body>
...
In this example I'm trying to find field1 and field2.
I need to add a path to the search term, to find the field:
print (myroot.find(".//{http://localhost/}field1").tag) # prints 'field1'
without it, I don't find anything
print (myroot.find("field1").tag) # finds 'None'
Is there any other way to search for the field tag (here field1) without giving path info?
Full example below:
from lxml import etree
example = """<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body><Request xmlns="http://localhost/">
<Test><field1>hello</field1><field2>world</field2></Test>
</Request></soap:Body></soap:Envelope>
"""
myroot = etree.fromstring(example)
# this works
print (myroot.find(".//{http://localhost/}field1").text)
print (myroot.find(".//{http://localhost/}field2").text)
# this fails
print (myroot.find(".//field1").text)
print (myroot.find("field1").text)
Comment: The input of the SOAP request is given, I can't change any of it in real live to make things easier.

There is a way to ignore namespace when selecting element using XPath, but that isn't a good practice. Namespace is there for a reason. Anyway, there is a cleaner way to reference element in namespace i.e by using namespace prefix that was mapped to the namespace uri, instead of using the actual namespace uri every time :
.....
>>> ns = {'d': 'http://localhost/'}
>>> print (myroot.find(".//d:field1", ns).text)
hello
>>> print (myroot.find(".//d:field2", ns).text)
world

Matching the XML Req and Res

I need advice on the below
Below are the request and response XML's. Request XML contains the words to be translated in the Foriegn language [String attribute inside Texts node] and the response XML contains the translation of these words in English [inside ].
REQUEST XML
<TranslateArrayRequest>
<AppId />
<From>ru</From>
<Options>
<Category xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" ></Category>
<ContentType xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2">text/plain</ContentType>
<ReservedFlags xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" />
<State xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" ></State>
<Uri xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" ></Uri>
<User xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" ></User>
</Options>
<Texts>
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/Arrays">вк азиза и ринат</string>
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/Arrays">скачать кайда кайдк кайрат нуртас бесплатно</string>
</Texts>
<To>en</To>
</TranslateArrayRequest>
RESPONSE XML
<ArrayOfTranslateArrayResponse xmlns="http://schemas.datacontract.org/2004/07/Microsoft.MT.Web.Service.V2" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<TranslateArrayResponse>
<From>ru</From>
<OriginalTextSentenceLengths xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:int>16</a:int>
</OriginalTextSentenceLengths>
<State/>
<TranslatedText>BK Aziza and Rinat</TranslatedText>
<TranslatedTextSentenceLengths xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:int>18</a:int>
</TranslatedTextSentenceLengths>
</TranslateArrayResponse>
<TranslateArrayResponse>
<From>ru</From>
<OriginalTextSentenceLengths xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:int>43</a:int> </OriginalTextSentenceLengths>
<State/>
<TranslatedText>Kairat kajdk Qaeda nurtas download free</TranslatedText>
<TranslatedTextSentenceLengths xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:int>39</a:int></TranslatedTextSentenceLengths>
</TranslateArrayResponse
</ArrayOfTranslateArrayResponse>

So there are two ways to relate the translated text to the original text:
Length of the original text; and
Order in the XML file
Relating by length being the probably unreliable because the probability of translating 2 or more phrases with the same number of characters is relatively significant.
So it comes down to order. I think it is relatively safe to assume that the files were processed and written in the same order. So I'll show you a way to relate the phrases using the order of the XML files.
This is relatively simple. We simply iterate through the trees and grab the words in the list. Also, for the translated XML due to its structure, we need to grab the root's namespace:
import re
import xml.etree.ElementTree as ElementTree
def map_translations(origin_file, translate_file):
origin_tree = ElementTree.parse(origin_file)
origin_root = origin_tree.getroot()
origin_text = [string.text for text_elem in origin_root.iter('Texts')
for string in text_elem]
translate_tree = ElementTree.parse(translate_file)
translate_root = translate_tree.getroot()
namespace = re.match('{.*}', translate_root.tag).group()
translate_text = [text.text for text in translate_root.findall(
'.//{}TranslatedText'.format(namespace))]
return dict(zip(origin_text, translate_text))
origin_file = 'some_file_path.xml'
translate_file = 'some_other_path.xml'
mapping = map_translations(origin_file, translate_file)
print(mapping)
Update
The above code is applicable for Python 2.7+. In Python 2.6 it changes slightly:
ElementTree objects do not have an iter function. Instead they have a getiterator function.
Change the appropriate line above to this:
origin_text = [string.text for text_elem in origin_root.iter('Texts')
for string in text_elem]
XPath syntax is (most likely) not supported. In order to get down to the TranslatedText nodes we need to use the same strategy as we do above:
Change the appropriate line above to this:
translate_text = [string.text for text in translate_root.getiterator(
'{0}TranslateArrayResponse'.format(namespace))
for string in text.getiterator(
'{0}TranslatedText'.format(namespace))]

Drop all namespaces in lxml?

I'm working with some of google's data APIs, using the lxml library in python. Namespaces are a huge hassle here. For a lot of the work I'm doing (xpath stuff, mainly), it would be nice to just plain ignore them.
Is there a simple way to ignore xml namespaces in python/lxml?
thanks!

If you'd like to remove all namespaces from elements and attributes, I suggest the code shown below.
Context: In my application I'm obtaining XML representations of SOAP response streams, but I'm not interested on building objects on the client side; I'm only interested on XML representations themselves. Moreover, I'm not interested on any namespace thing, which only makes things more complicated than they need to be, for my purposes. So, I simply remove namespaces from elements and I drop all attributes which contain namespaces.
def dropns(root):
for elem in root.iter():
parts = elem.tag.split(':')
if len(parts) > 1:
elem.tag = parts[-1]
entries = []
for attrib in elem.attrib:
if attrib.find(':') > -1:
entries.append(attrib)
for entry in entries:
del elem.attrib[entry]
# Test case
name = '~/tmp/mantisbt/test.xml'
f = open(name, 'rb')
import lxml.etree as etree
parser = etree.XMLParser(ns_clean=True, recover=True)
root = etree.parse(f, parser=parser)
print('=====================================================================')
print etree.tostring(root, pretty_print = True)
print('=====================================================================')
dropns(root)
print etree.tostring(root, pretty_print = True)
print('=====================================================================')
which prints:
=====================================================================
<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body>
<ns1:mc_issue_getResponse>
<return xsi:type="tns:IssueData">
<id xsi:type="xsd:integer">356</id>
<view_state xsi:type="tns:ObjectRef">
<id xsi:type="xsd:integer">10</id>
<name xsi:type="xsd:string">public</name>
</view_state>
</return>
</ns1:mc_issue_getResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
=====================================================================
<Envelope>
<Body>
<mc_issue_getResponse>
<return>
<id>356</id>
<view_state>
<id>10</id>
<name>public</name>
</view_state>
</return>
</mc_issue_getResponse>
</Body>
</Envelope>
=====================================================================

In lxml some_element.tag is a string like {namespace-uri}local-name if there is a namespace, just local-name otherwise. Beware that it is a non string value on non-element nodes (such as comments).
Try this:
for node in some_tree.iter():
startswith = getattr(node 'startswith', None)
if startswith and startswith('{'):
node.tag = node.tag.rsplit('}', 1)[-1]
On Python 2.x the tag can be either an ASCII byte-string or an Unicode string. The existence of a startswith method tests for either.

How can lxml validate some XML against both an XSD file while also loading an inline schema too?

I'm having problems getting lxml to successfully validate some xml. The XSD schema and XML file are both from Amazon documentation so should be compatible. But the XML itself refers to another schema that's not being loaded.
Here is my code, which is based on the lxml validation tutorial:
xsd_doc = etree.parse('ProductImage.xsd')
xsd = etree.XMLSchema(xsd_doc)
xml = etree.parse('ProductImage_sample.xml')
xsd.validate(xml)
print xsd.error_log
"ProductImage_sample.xml:2:0:ERROR:SCHEMASV:SCHEMAV_CVC_ELT_1: Element 'AmazonEnvelope': No matching global declaration available for the validation root."
I get no errors if I validate against amzn-envelope.xsd instead of ProductImage.xsd, but that defeats the point of seeing if a given Image feed is valid. All xsd & xml files mentioned are in my working directory along with my python script by the way.
Here is a snippet of the sample xml, which should definately be valid:
<?xml version="1.0"?>
<AmazonEnvelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="amzn-envelope.xsd">
<Header>
<DocumentVersion>1.01</DocumentVersion>
<MerchantIdentifier>Q_M_STORE_123</MerchantIdentifier>
</Header>
<MessageType>ProductImage</MessageType>
<Message>
<MessageID>1</MessageID>
<OperationType>Update</OperationType>
<ProductImage>
<SKU>1234</SKU>
Here is a snippet of the schema (this file is not public so I can't show all of it):
<?xml version="1.0"?>
<!-- Revision="$Revision: #5 $" -->
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
<xsd:include schemaLocation="amzn-base.xsd"/>
<xsd:element name="ProductImage">
<xsd:complexType>
<xsd:sequence>
<xsd:element ref="SKU"/>
I can say that following the include to amzn-base.xsd does not end up reaching a definition of the AmazonEnvelope tag. So my questions is: can lxml load schemas via a tag like <AmazonEnvelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="amzn-envelope.xsd">. And if not, how can I validate my Image feed?

The answer is I should validate by the parent schema file, which as mentioned at the top of the XML file is amzn-envelope.xsd as this contains the line:
<xsd:include schemaLocation="ProductImage.xsd"/>
In general then, lxml won't read such a declaration as xsi:noNamespaceSchemaLocation="amzn-envelope.xsd" but if you can find the parent schema to validate against then this should hopefully include the specific schema you're interested in.