I'm trying to fetch particular parts of a XML file and move it into a pandas dataframe. Following some tutorials from xml.etree I'm still stuck at getting the output. So far, I've managed to find the child nodes, but I can't access them (i.e. can't get the actual data out of it). So, here is what I've got so far.
tree=ET.parse('data.xml')
root=tree_edu.getroot()
root.tag
#find all nodes within xml data
tree_edu.findall(".//")
#access the node
tree.findall(".//{http://someUrl.nl/schema/enterprise/program}programSummaryText")
What I want is to get the data from the node programDescriptions and specifically the child programDescriptionText xml:lang="nl", and of course a couple extra. But first focus on this one.
Some data to work with:
<?xml version="1.0" encoding="UTF-8"?>
<programs xmlns="http://someUrl.nl/schema/enterprise/program">
<program xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://someUrl.nl/schema/enterprise/program http://someUrl.nl/schema/enterprise/program.xsd">
<customizableOnRequest>true</customizableOnRequest>
<editor>webmaster#url</editor>
<expires>2019-04-21</expires>
<format>Edu-dex 1.0</format>
<generator>www.Url.com</generator>
<includeInCatalog>Catalogs</includeInCatalog>
<inPublication>true</inPublication>
<lastEdited>2019-04-12T20:03:09Z</lastEdited>
<programAdmission>
<applicationOpen>true</applicationOpen>
<applicationType>individual</applicationType>
<maxNumberOfParticipants>12</maxNumberOfParticipants>
<minNumberOfParticipants>8</minNumberOfParticipants>
<paymentDue>up-front</paymentDue>
<requiredLevel>academic bachelor</requiredLevel>
<startDateDetermination>fixed starting date</startDateDetermination>
</programAdmission>
<programCurriculum>
<instructionMode>training</instructionMode>
<teacher>
<id>{D83FFC12-0863-44A6-BDBB-ED618627F09D}</id>
<name>SomeName</name>
<summary xml:lang="nl">
Long text of the summary. Not needed.
</summary>
</teacher>
<studyLoad period="hour">26</studyLoad>
</programCurriculum>
<programDescriptions>
<programName xml:lang="nl">Program Course Name</programName>
<programSummaryText xml:lang="nl">short Program Course Name summary</programSummaryText>
<programSummaryHtml xml:lang="nl">short Program Course Name summary in HTML format</programSummaryHtml>
<programDescriptionText xml:lang="nl">This part is needed from the XML.
Big program description text. This part is needed to parse from the XML file.
</programDescriptionText>
<programDescriptionHtml xml:lang="nl">Not needed;
Not needed as well;
</programDescriptionHtml>
<subjectText>
<subject>curriculum</subject>
<header1 xml:lang="nl">Beschrijving</header1>
<descriptionHtml xml:lang="nl">Yet another HTML desscription;
Not necessarily needed;</descriptionHtml>
</subjectText>
<searchword xml:lang="nl">search word</searchword>
<webLink xml:lang="nl">website-url</webLink>
</programDescriptions>
<programSchedule>
<programRun>
<id>PR-019514</id>
<status>application opened</status>
<startDate isFinal="true">2019-06-26</startDate>
<endDate isFinal="true">2020-02-11</endDate>
</programRun>
</programSchedule>
</program>
</programs>
Try the code below: (55703748.xml contains the xml you have posted)
import xml.etree.ElementTree as ET
tree = ET.parse('55703748.xml')
root = tree.getroot()
nodes = root.findall(".//{http://someUrl.nl/schema/enterprise/program}programSummaryText")
for node in nodes:
print(node.text)
Output
short Program Course Name summary
Related
I have an xml file and I am trying to iterate through the tags to convert it to a pandas dataframe. My current process is to open the XML file with excel as an "XML table" but this takes forever. Trying to find a similar process in Python.
I am trying to follow along with the code presented on numerous other Stack Overflow questions and articles such as here here and here
I believe there are 2 problems I am facing:
Does having the namespace affect my xml?
I don't want to specify all of my tags as seen as a solution in 19.7.1.6. of the Element Tree documentation. I just want all of my tags to appear as a column for each "Security." If it doesn't have that tag it should be null. I also do not want to do a nasty if-else.
The problem is that when I run the code:
import xml.etree.ElementTree as et
etree = et.parse(xml_path)
test = etree.getroot()
and try and iterate as suggested in the above links, I am not able to easily access the child nodes.
Sample File:
<?xml version="1.0"?>
<SecurityInformation xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://tempuri.org/SecurityInformation.xsd">
<Security>
<Country>United States</Country>
</Security>
</SecurityInformation>
I've made a package for similar use case. It could work here too.
pip install pandas_read_xml
you can do something like
import pandas_read_xml as pdx
df = pdx.read_xml('filename.xml', ['SecurityInformation'])
To flatten, you could
df = pdx.flatten(df)
or
df = pdx.fully_flatten(df)
very new to anything xml related please bear with me - trying to build some code that converts rasters to KML files for google earth.
I've come across the lxml package which has made my life easier, but now am facing an issue.
Let's say I've created an element called kml with namespaces:
from lxml import etree
version = '2.2'
namespace_hdr = {'gx':f'http://www.google.com/kml/ext/{version}',
'kml':f'http://www.opengis.net/kml/{version}',
'atom':f'http://www.w3.org/2005/Atom'
}
kml = etree.Element('kml', nsmap=namespace_hdr)
And I've also created an element called Document:
Document = etree.SubElement(kml, 'Document')
Now..I have alot of data I want to write and am running into memory issues, so I figured the best approach would be to generate my data to write on the fly and write it as I go, hence the incremental writing.
The approach I'm using is:
out_file = 'test.kml'
with etree.xmlfile(out_file, encoding='utf-8') as xf:
xf.write_declaration()
with xf.element(kml):
xf.write(Document)
Which returns the error:
TypeError: Argument must be bytes or unicode, got '_Element'
If I change kml to 'kml' it works fine, but obviously does not write the namespaces to the file that I've defined in the kml element.
How is it possible to pass in the kml element instead of a string? Is there a way to do this? Or some other way of incrementally writing to the file?
Any thoughts would be appreciated.
FYI - output when using 'kml' is:
<?xml version='1.0' encoding='utf-8'?>
<kml><Document/>
</kml>
I'm trying to achieve the same but with the namespaces:
<?xml version="1.0" encoding="utf-8"?>
<kml xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">
<Document/>
</kml>
I'm using ElementTree to parse some XML retrieved from a website, but somehow I can't see to be able to use ".find" or ".findall". I tried to use ElementTree, and I tired lxml.etree and nothing is working with me. My goal is to retrieve //course from my XML file retrieved from a URL.
import requests
import xml.etree.ElementTree as ET
res = requests.get(COURSES_URL).text #Storing the XML into res
XML = ET.fromstring(res)
print(XML.findall('//COURSE'))
COURSES_URL is my own URL which I am retrieving the XML from, and yes it is working since I got the output XML that I want (sample):
<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by Oracle Reports version 11.1.2.1.0 -->
<SYRSPOS_REP>
<LIST_G_PROGRAM>
<G_PROGRAM>
<SPRIDEN_ID>U712214</SPRIDEN_ID>
<STUDENT_NAME>Mark Adam Johns</STUDENT_NAME>
<SMBPOGN_PIDM>98</SMBPOGN_PIDM>
<SMBPOGN_REQUEST_NO>46</SMBPOGN_REQUEST_NO>
<COURSE ID=1411001>PASS</COURSE>
<COURSE ID=1411023>PASS</COURSE>
<COURSE ID=1411136>PASS</COURSE>
</G_PROGRAM>
</LIST_G_PROGRAM>
</SYRSPOS_REP>
Solved:
Apparently I had 2 issues.
First of all I can't use findall in print since it returns a list, I had to do a for in loop for i in XML.findall(), then I print i.text().
Secondly, I had to add a dot after the quotation mark, as in ".//COURSES"
I have an xml document and I want to extract a subnode (boundedBy) and pretty_print it exactly as it looks in the original document (with exception to the pretty formatting).
<?xml version="1.0" encoding="UTF-8" ?>
<wfs:FeatureCollection
xmlns:sei="https://somedomain.com/namespace"
xmlns:wfs="http://www.opengis.net/wfs"
xmlns:gml="http://www.opengis.net/gml"
xmlns:ogc="http://www.opengis.net/ogc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd
https://somedomain.com/schemas/wfsnamespace some.xsd">
<gml:boundedBy>
<gml:Box srsName="EPSG:4326">
<gml:coordinates>-10.934396,-139.997120 77.396455,-53.627763</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<gml:featureMember>
<sei:HUB_HEIGHT_FCST>
<!--- This is the section I want --->
<gml:boundedBy>
<gml:Box srsName="EPSG:4326">
<gml:coordinates>14.574435,-139.997120 14.574435,-139.997120</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<!--- This is the section I want --->
<sei:geometry_4326>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>14.574435,-139.997120</gml:coordinates>
</gml:Point>
</sei:geometry_4326>
<sei:rundatetime>2017-09-26 00:00:00</sei:rundatetime>
<sei:validdatetime>2017-09-26 17:00:00</sei:validdatetime>
</sei:HUB_HEIGHT_FCST>
</gml:featureMember>
</wfs:FeatureCollection>
Here is how I'm extracting the subnode:
# parse the xml string
parser = etree.XMLParser(remove_blank_text=True, remove_comments=True, recover=False, strip_cdata=False)
root = etree.fromstring(xmlstr, parser=parser)
#find the subnode I want
subnodes = root.xpath("./gml:boundedBy", namespaces={'gml': 'http://www.opengis.net/gml'})
subnode = subnodes[0]
# make a pretty output
xmlstr = etree.tostring(subnode, xml_declaration=False, encoding="UTF-8", pretty_print=True)
print xmlstr
Which gives me this. Unfortunately lxml is adding the namespaces to the boundedBy node (which makes sense for the sake of completeness in xml).
<gml:boundedBy xmlns:gml="http://www.opengis.net/gml" xmlns:sei="https://somedomain.com/namespace" xmlns:wfs="http://www.opengis.net/wfs" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<gml:Box srsName="EPSG:4326">
<gml:coordinates>-10.934396,-139.997120 77.396455,-53.627763</gml:coordinates>
</gml:Box>
</gml:boundedBy>
I only want the subnode as it looked in the original document.
<gml:boundedBy>
<gml:Box srsName="EPSG:4326">
<gml:coordinates>14.574435,-139.997120 14.574435,-139.997120</gml:coordinates>
</gml:Box>
</gml:boundedBy>
I'm flexible with not using lxml, but either way I haven't found options on how to accomplish this.
edit:
Since it was pointed out that I should explain why I want to do this...
I'm trying to log the xml fragment without altering it's original structure. The automated test I'm building looks at certain nodes for correctness. In the process I'm logging the fragment and want to make it a bit more readable for the person reviewing. Some of the fragments can get fairly large which is why pretty_print is so nice.
I have several xml files that I want to parse in python. I am aware of the ElementTree package in python, however my xml files aren't stored in a tree like structure. Below is an example
<tag1 attribute1="at1" attribute2="at2">My files are text that I annotated with a tool
to create these xml files.</tag1>
Some parts of the text are enclosed in an xml tag, whereas others are not.
<tag1 attribute1="at1" attribute2="at2"><tag2 attribute3="at3" attribute4="at4">Some
are even enclosed in multiple tags.</tag1></tag2>
And some have overlapping tags:
<tag1 attribute1="at1" attribute2="at2">This is an example sentence
<tag3 attribute5="at5">containing a nested example sentence</tag3></tag1>
Whenever I use an ElementTree like function to parse the file, I can only access the very first tag. I am looking for a way to parse all the tags and don't want a tree like structure. Any help is greatly appreciated.
If you have one XML fragment per line, just parse each line individually.
for line in some_file:
# parse using ET and getroot.