Develop Reference

XML data extraction in python

I have an XML file like the following:
<AreaModel>
...
<RecipePhase>
<UniqueName>PHASE1</UniqueName>
...
<NumberOfParameterTags>7</NumberOfParameterTags>
...
<DefaultRecipeParameter>
<Name>PARAM1</Name>
----
</DefaultRecipeParameter>
<DefaultRecipeParameter>
<Name>PARAM2</Name>
----
</DefaultRecipeParameter>
<DefaultRecipeParameter>
<Name>PARAM3</Name>
----
</DefaultRecipeParameter>
</RecipePhase>
<RecipePhase>
....
</RecipePhase>
</AreaModel>
I would like to read this file in sequential order and generate different list. One for the texts of UniqueName TAGs and a list of lists containing for each list the set of texts for tag Name under each RecipePhase element.
For example, I might have 10 RecipePhase elements, each one with TAG UniqueName and each one containing a different set of children with tag DefaultRecipeParameter.
How can I take into account when I enter into RecipePhase and when I go out of the element during parsing?
I am trying ElementTree but I am not able to find a solution.
cheers,
m

You can use xml python module:
See my example:
from xml.dom import minidom as dom
import urllib2
def fetchPage(url):
a = urllib2.urlopen(url)
return ''.join(a.readlines())
def extract(page):
a = dom.parseString(page)
item = a.getElementsByTagName('Rate')
for i in item:
if i.hasChildNodes() == True:
print i.getAttribute('currency')+"-"+ i.firstChild.nodeValue
if __name__=='__main__':
page = fetchPage("http://www.bnro.ro/nbrfxrates.xml")
extract(page)

I solved partially my problem with the following code:
import xml.etree.ElementTree as ET
tree = ET.parse('control_strategies.axml')
root = tree.getroot()
phases=[]
for recipephase in root.findall('./RecipePhase/UniqueName'):
phases.append(recipephase.text)
n_elem = len(phases)
param=[[] for _ in range(n_elem)]
i = 0
for recipephase in root.findall('./RecipePhase'):
for defparam in recipephase.findall('./DefaultRecipeParameter'):
for paramname in defparam.findall('./Name'):
param[i].append(paramname.text)
i = i + 1

lxml (etree) - Pretty Print attributes of root tag

Is it possible in python to pretty print the root's attributes?
I used etree to extend the attributes of the child tag and then I had overwritten the existing file with the new content. However during the first generation of the XML, we were using a template where the attributes of the root tag were listed one per line and now with the etree I don't manage to achieve the same result.
I found similar questions but they were all referring to the tutorial of etree, which I find incomplete.
Hopefully someone has found a solution for this using etree.
EDIT: This is for custom XML so HTML Tidy (which was proposed in the comments), doesn't work for this.
Thanks!
generated_descriptors = list_generated_files(generated_descriptors_folder)
counter = 0
for g in generated_descriptors:
if counter % 20 == 0:
print "Extending Descriptor # %s out of %s" % (counter, len(descriptor_attributes))
with open(generated_descriptors_folder + "\\" + g, 'r+b') as descriptor:
root = etree.XML(descriptor.read(), parser=parser)
# Go through every ContextObject to check if the block is mandatory
for context_object in root.findall('ContextObject'):
for attribs in descriptor_attributes:
if attribs['descriptor_name'] == g[:-11] and context_object.attrib['name'] in attribs['attributes']['mandatoryobjects']:
context_object.set('allow-null', 'false')
elif attribs['descriptor_name'] == g[:-11] and context_object.attrib['name'] not in attribs['attributes']['mandatoryobjects']:
context_object.set('allow-null', 'true')
# Sort the ContextObjects based on allow-null and their name
context_objects = root.findall('ContextObject')
context_objects_sorted = sorted(context_objects, key=lambda c: (c.attrib['allow-null'], c.attrib['name']))
root[:] = context_objects_sorted
# Remove mandatoryobjects from Descriptor attributes and pretty print
root.attrib.pop("mandatoryobjects", None)
# paste new line here
# Convert to string in order to write the enhanced descriptor
xml = etree.tostring(root, pretty_print=True, encoding="UTF-8", xml_declaration=True)
# Write the enhanced descriptor
descriptor.seek(0) # Set cursor at beginning of the file
descriptor.truncate(0) # Make sure that file is empty
descriptor.write(xml)
descriptor.close()
counter+=1

python lxml etree namespaces on creation

I am using lxml etree to create xml or REST call. I have problem with namespaces since if not formulated correctly I get a syntax error from server.
As you can see in the following 2 examples I should be getting eg ns1, ns2, ns4, ns5 but the xml goes over with ns15, ns16 but at the end it has the e.g "" or " " - I know this explains it but for the nature of my REST call I need it as the example is.
How can I prevent that
I have to get the following xml
<ns5:prenosPodatkovRazporedaZahtevaSporocilo xmlns="http://xxx.yyy/sheme/pdr/skupno/v1" xmlns:ns2="http://xxx.yyy/sheme/pdr/v1" xmlns:ns3="http://xxx.yyy/sheme/kis/skupno/v2" xmlns:ns4="http://xxx.yyy/sheme/kis/v2" xmlns:ns5="http://xxx.yyy/sheme/pdr/sporocila/v1">
<ns5:podatkiRazporeda>
<ns2:podatkiRazporeda>
<ns2:delitvenaEnota>
<sifra>80</sifra>
</ns2:delitvenaEnota>
<ns2:vrstaRazporeda>
<sifra>4</sifra>
</ns2:vrstaRazporeda>
<ns2:tipRazporeda>
<sifra>D</sifra>
</ns2:tipRazporeda>
<ns2:obdobje>
<ns2:mesec>12</ns2:mesec>
<ns2:leto>2017</ns2:leto>
</ns2:obdobje>
<ns2:skupina>0</ns2:skupina>
<ns2:izvor>P_738</ns2:izvor>
<ns2:oznakeDelaZaDneve>
<ns2:oznakaDelaZaDan>
<ns2:dan>1</ns2:dan>
<ns2:oznakaDela>D4</ns2:oznakaDela>
</ns2:oznakaDelaZaDan>
....
</ns2:oznakeDelaZaDneve>
<ns2:organizacijskaEnota>
<sifra>738</sifra>
</ns2:organizacijskaEnota>
<ns2:zaposlenec>
<ns4:osebnaStevilka>10357</ns4:osebnaStevilka>
</ns2:zaposlenec>
</ns2:podatkiRazporeda>
</ns5:podatkiRazporeda>
Where I am getting this xml.
Mind the namespace marks.
<ns0:prenosPodatkovRazporedaOdgovorSporocilo xmlns:ns="http://rccirc.si/sheme/pdr/skupno/v1" xmlns:ns2="http://rccirc.si/sheme/pdr/v1" xmlns:ns3="http://rccirc.si/sheme/kis/skupno/v2" xmlns:ns4="http://rccirc.si/sheme/kis/v2" xmlns:ns5="http://rccirc.si/sheme/pdr/sporocila/v1" xmlns:ns0="ns5">
<ns0:podatkiRazporeda>
<ns1:podatkiRazporeda xmlns:ns1="ns2">
<ns1:vrstaRazporeda>
<sifra>647</sifra>
</ns1:vrstaRazporeda>
<ns1:tipRazporeda>
<sifra>D</sifra>
</ns1:tipRazporeda>
<ns1:obdobje>
<ns1:mesec>1</ns1:mesec>
<ns1:leto>2018</ns1:leto>
</ns1:obdobje>
<ns1:skupina>0</ns1:skupina>
<ns1:izvor>0</ns1:izvor>
<ns1:organizacijskaEnota>
<sifra>250</sifra>
</ns1:organizacijskaEnota>
<ns6:delitvenaenota xmlns:ns6="ns3">
<sifra>80</sifra>
</ns6:delitvenaenota>
<ns1:oznakeDelaZaDneve>
<oznakeDelaZaDneve>
<ns1:dan>29</ns1:dan>
<ns1:oznakaDela>1930-0730</ns1:oznakaDela>
</oznakeDelaZaDneve>
</ns1:oznakeDelaZaDneve>
<ns1:zaposlenec>
<ns7:osebnaStevilka xmlns:ns7="ns4">Z1</ns7:osebnaStevilka>
</ns1:zaposlenec>
</ns1:podatkiRazporeda>
.......
<ns11:podatkiRazporeda xmlns:ns11="ns2">
<ns11:vrstaRazporeda>
<sifra>647</sifra>
</ns11:vrstaRazporeda>
<ns11:tipRazporeda>
<sifra>D</sifra>
</ns11:tipRazporeda>
<ns11:obdobje>
<ns11:mesec>1</ns11:mesec>
<ns11:leto>2018</ns11:leto>
</ns11:obdobje>
<ns11:skupina>0</ns11:skupina>
<ns11:izvor>0</ns11:izvor>
<ns11:organizacijskaEnota>
<sifra>250</sifra>
</ns11:organizacijskaEnota>
<ns12:delitvenaenota xmlns:ns12="ns3">
<sifra>80</sifra>
</ns12:delitvenaenota>
<ns11:oznakeDelaZaDneve>
<oznakeDelaZaDneve>
<ns11:dan>3</ns11:dan>
<ns11:oznakaDela>0730-1530</ns11:oznakaDela>
</oznakeDelaZaDneve>
.....
</ns11:oznakeDelaZaDneve>
<ns11:zaposlenec>
<ns13:osebnaStevilka xmlns:ns13="ns4">Z1</ns13:osebnaStevilka>
</ns11:zaposlenec>
</ns11:podatkiRazporeda>
</ns0:podatkiRazporeda>
</ns0:prenosPodatkovRazporedaOdgovorSporocilo>
Here is my code.
root = etree.Element('{ns5}prenosPodatkovRazporedaOdgovorSporocilo', nsmap = {'ns': "http://xxx.yyy/sheme/pdr/skupno/v1",'ns2':"http://xxx.yyy/sheme/pdr/v1" ns3':"http://xxx.yyy/sheme/kis/skupno/v2",ns4': "http://xxx.yyy/sheme/kis/v2",ns5': "http://xxx.yyy/sheme/pdr/sporocila/v1"})
podatkiRazporedaMain = etree.SubElement(root, '{ns5}podatkiRazporeda')
#follwed by creating sub elements etc.
for rec in grouped_workers:
podatkiRazporeda = etree.SubElement(podatkiRazporedaMain, '{ns2}podatkiRazporeda')
vrstaRazporeda= etree.SubElement(podatkiRazporeda, '{ns2}vrstaRazporeda')
vrstaRazporedaSifra = etree.SubElement(vrstaRazporeda, 'sifra')
vrstaRazporedaSifra.text = "647"
tipRazporeda= etree.SubElement(podatkiRazporeda, '{ns2}tipRazporeda')
tipRazporedaSifra = etree.SubElement(tipRazporeda, 'sifra')
tipRazporedaSifra.text = 'D'
for rr in rec["data"]:
oznakaDelaZaDan = etree.SubElement(oznakeDelaZaDneve, 'oznakeDelaZaDneve')
dan= etree.SubElement(oznakaDelaZaDan, '{ns2}dan')
dan.text = str(rr["rw_date"].day)
oznakaDela = etree.SubElement(oznakaDelaZaDan, '{ns2}oznakaDela')
oznakaDela.text = str(rr["rw_shift"])
#print etree.tostring(root, pretty_print=True, xml_declaration=False, encoding='UTF-8')
fle = os.path.join(request.folder, 'private', str(647) + '.xml')
with open(fle, 'wb') as f:
f.write(etree.tostring(root, pretty_print=True, xml_declaration=False, encoding='UTF-8'))#,inclusive_ns_prefixes=None))
#etree..write(fle, pretty_print=True, xml_declaration=False, encoding='UTF-8')
print "Done"
So why are ns incremented?
Hope I was clear
Than you

So as it turns out when you are creating tags you should not write
vrstaRazporeda= etree.SubElement(podatkiRazporeda, '{ns2}vrstaRazporeda')
vrstaRazporedaSifra = etree.SubElement(vrstaRazporeda, 'sifra').text = "647"
But
vrstaRazporeda= etree.SubElement(podatkiRazporeda, '{http://xxx.yyy/sheme/pdr/v1}vrstaRazporeda')
vrstaRazporedaSifra = etree.SubElement(vrstaRazporeda, 'sifra').text = "647"
so the whole url - this seemed to solve the issue.

Creating XML file in python by iterating over a set of values

I have to create an xml file with root name 'structure' and it should consist of various subElements "packets", each of which contains a tree structure of elements. A single packet should look like this:
<L2>
<srcmac>value1</srcmac>
<dstmac>value2</dstmac>
</L2>
<L3>
<dscp>
<timetolive>value3</timetolive>
</dscp>
</L3>
<L4>
<protocol>value4</protocol>
</L4>
Th packet should be iterated 'n' number of times. It gives some error saying: unbound method write() must be called with ElementTree instance (got str instance instead)
This is the code I have written:
import xml.etree.cElementTree as et
import pcapy
import sys
from struct import*
def main():
count=1
root=et.Element("structure")
#creating xml layout
eachpacket=et.SubElement(root,"packet")
while count<n:
child1=et.SubElement(eachpacket,"L2")
subchild1=et.SubElement(child1,"smac")
subchild2=et.SubElement(child1,"dmac")
child2=et.SubElement(eachpacket,"L3")
sub1=et.SubElement(child2,"sip")
sub2=et.SubElement(child2,"dip")
sub3=et.SubElement(child2,"dscp")
s2=et.SubElement(sub3,"timetolive")
child3=et.SubElement(eachpacket,"L4")
schild1=et.SubElement(child3,"protocol")
try:
count=count+1
subchild1.text=str(value1)
subchild2.text=str(value2)
s2.text=str(value3)
schild1.text=str(protocol)
except:
break
else:
tree=et.ElementTree(root)
tree.write("FileNew.xml")
pass
if __name__== "__main__":
main()
Although I got an XML file as output, but the last entry seems to overwrite all the previous entries, and only the last iteration of 'packet' is stored in my generated xml file.
What do I need to modify to get a complete tree a iterations and to rectify the write() error?

I did some fixes, including style improvements, at your code and replaced your values like protocol to placeholders, since I have neither your libs, nor logic. As far I tested, it solves both issues.
from xml.etree import cElementTree as et
n = 5
value1 = 'value1'
value2 = 'value2'
value3 = 'value3'
protocol = 'protocol'
def main():
root = et.Element("structure")
# creating xml layout
eachpacket = et.SubElement(root, "packet")
for count in range(n):
child1 = et.SubElement(eachpacket, "L2")
subchild1 = et.SubElement(child1, "smac")
subchild2 = et.SubElement(child1, "dmac")
child2 = et.SubElement(eachpacket, "L3")
sub1 = et.SubElement(child2, "sip")
sub2 = et.SubElement(child2, "dip")
sub3 = et.SubElement(child2, "dscp")
s2 = et.SubElement(sub3, "timetolive")
child3 = et.SubElement(eachpacket, "L4")
schild1 = et.SubElement(child3, "protocol")
subchild1.text = str(value1)
subchild2.text = str(value2)
s2.text = str(value3)
schild1.text = str(protocol)
et.ElementTree.write(et.ElementTree(root), "FileNew.xml")
if __name__ == "__main__":
main()

How to parse XML using python

I am trying to parse an xml using python for create a result summary file. Below is my code and a snippet of xml, Like the below i have couple of sections with <test> and </test>
<test name="tst_case1">
<prolog time="2013-01-18T14:41:09+05:30"/>
<verification name="VP5" file="D:/Squish/HMI_testing/tst_case1/test.py" type="properties" line="6">
<result time="2013-01-18T14:41:10+05:30" type="PASS">
<description>VP5: Object propertycomparisonof ':_QMenu_3.enabled'passed</description> <description type="DETAILED">'false' and 'false' are equal</description>
<description type="object">:_QMenu_3</description>
<description type="property">enabled</description>
<description type="failedValue">false</description>
</result>
</verification>
<epilog time="2013-01-18T14:41:11+05:30"/>
</test>
What I want to get is,
in one <test> section how many PASS / FAIL is there.
With the below code its printing the total pass/Fail in the xml file.But i am interested in each section how many PASS/FAIL. can any boy tell me the procedure to fetchout this ?
import sys
import xml.dom.minidom as XY
file = open("result.txt", "w")
tree = XY.parse('D:\\Squish\\squish results\\Results-On-2013-01-18_0241 PM.xml')
Test_name = tree.getElementsByTagName('test')
Test_status = tree.getElementsByTagName('result')
count_testname =0
passcount = 0
failcount = 0
Test_name_array = []
for my_Test_name in Test_name:
count_testname = count_testname+1
passcount = 0
failcount = 0
my_Test_name_final = my_Test_name.getAttribute('name')
Test_name_array = my_Test_name_final
if(count_testname > 1):
print(my_Test_name_final)
for my_Test_status in Test_status:
my_Test_status_final = my_Test_status.getAttribute('type')
if(my_Test_status_final == 'PASS'):
passcount = passcount+1
if(my_Test_status_final == 'FAIL'):
failcount = failcount+1
print(str(my_Test_status_final))

I'd not use minidom for this task; the DOM API is very cumbersome, verbose, and not suited for searching and matching.
The Python library also includes the xml.etree.ElementTree API, I'd use that instead:
from xml.etree import ElementTree as ET
tree = ET.parse(r'D:\Squish\squish results\Results-On-2013-01-18_0241 PM.xml')
tests = dict()
# Find all <test> elements with a <verification> child:
for test in tree.findall('.//test[verification]'):
passed = len(test.findall(".//result[#type='PASS']"))
failed = len(test.findall(".//result[#type='FAIL']"))
tests[test.attrib['name']] = {'pass': passed, 'fail': failed}
The above piece of code counts the number of passed and failed tests per <test> element and stores them in a dictionary, keyed to the name attribute of the <test> element.
I've tested the above code with Python 3.2 and the full XML document from another question you posted, which results in:
{'tst_Setup_menu_2': {'fail': 0, 'pass': 8}}

Thanks for the posting. i got it working using minidon.
still wish to see how can be solved using xml.etree.ElementTree
import sys
import xml.dom.minidom as XY
file = open("Result_Summary.txt", "w")
#tree = XY.parse('D:\\Squish\\squish results\\Results-On-2013-01-18_0241 PM.xml')
#print (str(sys.argv[1]))
tree = XY.parse(sys.argv[1])
Test_name = tree.getElementsByTagName('test')
count_testname =0
file.write('Test Name \t\t\t No:PASS\t\t\t No:FAIL\t \n\n')
for my_Test_name in Test_name:
count_testname = count_testname+1
my_Test_name_final = my_Test_name.getAttribute('name')
if(count_testname > 1):
#print(my_Test_name_final)
file.write(my_Test_name_final)
file.write('\t\t\t\t')
my_Test_status = my_Test_name.getElementsByTagName('result')
passcount = 0
failcount = 0
for my_Test_status_1 in my_Test_status:
my_Test_status_final = my_Test_status_1.getAttribute('type')
if(my_Test_status_final == 'PASS'):
passcount = passcount+1
if(my_Test_status_final == 'FAIL'):
failcount = failcount+1
#print(str(my_Test_status_final))
file.write(str(passcount))
#print(passcount)
file.write('\t\t\t\t')
file.write(str(failcount))
# print(failcount)
file.write('\n')
#print ('loop count: %d' %count_testname)
#print('PASS count: %s' %passcount)
#print('FAIL count: %s' %failcount)
file.close()

Although not a standard module but well worth the effort of installing is lxml especially if you want to do fast Xml parsing etc IMHO.
Without a full example of your results I guessed at what they would look like.
from lxml import etree
tree = etree.parse("results.xml")
count_result_type = etree.XPath("count(.//result[#type = $name])")
for test in tree.xpath("//test"):
print test.attrib['name']
print "\t# FAILS ", count_result_type(test, name="FAIL")
print "\t# PASSES", count_result_type(test, name="PASS")
I generated the following running against my guess of your xml, which should give you an idea of what is happening.
tst_case1
# FAILS 1.0
# PASSES 1.0
tst_case0
# FAILS 0.0
# PASSES 1.0
tst_case2
# FAILS 0.0
# PASSES 1.0
tst_case3
# FAILS 0.0
# PASSES 1.0
What I like about lxml is how expressive it can be, YMMV.

I see you are using Squish. You should check your squish folder under \examples\regressiontesting. There you can find a file called xml2result2html.py. Here you can find an example of converting squish test results into html.