Dictionary lookup of loaded yaml via a string representation in Python

Dictionary lookup of loaded yaml via a string representation in Python - python

So I am using ruamel to read a yaml file located in github and all goes well. I can see it loaded correctly. Now my scenario is that this load is in a function in a class that I am accessing. Now this function has a string variable "entry" which is what I want to search for. The goal is to search at varying depths and I know the locations so I am not hunting for it.
Sample Yaml file:
description: temp_file
image: beta1
group: dev
depends:
- image: base
basetag: 1.0.0
Entry string variable passing in
I want to keep this a string for the variable "entry" as if looking for a top level lookup using the .get will return my value just fine. Its just if I want to gather something like like the value of ["depends"][0]["image"], I cannot figure out how construct this so I can do the proper get.
entry = "image" # works fine
entry = '["depends"][0]["image"]' #never works
gho.get_entry_from_ivpbuild_yml(repo, commit, entry)
Code in Class
# imports
from ruamel.yaml import YAML
yaml = YAML()
yaml.preserve_quotes = True
def get_entry_from_loaded_yml(self, repo, commit, entry, failflag=True):
"""
:param repo: str (github repo)
:param commit: str (sha of commit to view)
:param entry: str (entry within yml you want to search for)
:param failflag: bool (Determines if script fails or not if entry is not found within yaml)
:return: str: (value of entry in yml you want to search for)
"""
yml_file = "sample.yaml"
try:
logger.debug("opening yaml for commit:{}".format(commit))
yml = self.gho.get_repo(repo).get_file_contents(yml_file, commit)
except Exception as e:
logger.error("Could not open yaml file:{} for repo:{} commit:{}:{}".format(yml_file, repo, commit, e))
sys.exit(1)
loaded = yaml.load(yml.decoded_content)
if not loaded.get(entry, default=None):
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None

As you already know, you can't pass something like the string '["depends"][0]["image"]' to dict.get and expect that to work. But there are a couple of options if you really need to specify a "path" to the object within a nested data structure like this.
The first is to do it explicitly, just passing a sequence of keys instead of a single key:
def get_entry_from_loaded_yml(self, repo, commit, entry_keys, failflag=True):
# ...
try:
entry = loaded
for key in entry_keys:
entry = entry[key]
except LookupError:
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None
else:
return entry
And now, you can do this:
gho.get_entry_from_ivpbuild_yml(repo, commit, ('depends', 0, 'image'))
Alternatively, you can use a library that handles "key paths", in a format like dpath (which is essentially a simplified version of XPath) or ObjectiveC's KVC. There are multiple libraries on PyPI for doing this (although some work on undecoded JSON strings rather than decoded nested objects, to allow searching huge JSON texts efficiently; those obviously won't work for you… and I don't know of any that work on YAML instead of JSON, but they might exist). Then your code would look something like this:
def get_entry_from_loaded_yml(self, repo, commit, entry, failflag=True):
# ...
result = dpath_search(loaded, entry, default=None):
if result is None:
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None
else:
return result
# ...
gho.get_entry_from_ivpbuild_yml(repo, commit, 'depends/0/image')
This has the advantage that if you ever need to look up a (possibly nested) sequence of multiple values, it can be as simple as this:
for result in dpath_search(loaded, 'depends/*/image'):

Related

How to efficiently detect an XML schema without having the entire file in python

I have a very large feed file that is sent as an XML document (5GB). What would be the fastest way to parse the structure of the main item node without previously knowing its structure? Is there a means in Python to do so 'on-the-fly' without having the complete xml loaded in memory? For example, what if I just saved the first 5MB of the file (by itself it would be invalid xml, as it wouldn't have ending tags) -- would there be a way to parse the schema from that?
Update: I've included an example XML fragment here: https://hastebin.com/uyalicihow.xml. I'm looking to extract something like a dataframe (or list or whatever other data structure you want to use) similar to the following:
Items/Item/Main/Platform Items/Item/Info/Name
iTunes Chuck Versus First Class
iTunes Chuck Versus Bo
How could this be done? I've added a bounty to encourage answers here.

Several people have misinterpreted this question, and re-reading it, it's really not at all clear. In fact there are several questions.
How to detect an XML schema
Some people have interpreted this as saying you think there might be a schema within the file, or referenced from the file. I interpreted it as meaning that you wanted to infer a schema from the content of the instance.
What would be the fastest way to parse the structure of the main item node without previously knowing its structure?
Just put it through a parser, e.g. a SAX parser. A parser doesn't need to know the structure of an XML file in order to split it up into elements and attributes. But I don't think you actually want the fastest parse possible (in fact, I don't think performance is that high on your requirements list at all). I think you want to do something useful with the information (you haven't told us what): that is, you want to process the information, rather than just parsing the XML.
Is there a python utility that can do so 'on-the-fly' without having
the complete xml loaded in memory?
Yes, according to this page which mentions 3 event-based XML parsers in the Python world: https://wiki.python.org/moin/PythonXml (I can't vouch for any of them)
what if I just saved the first 5MB of the file (by itself it would be invalid xml, as it wouldn't have ending tags) -- would there be a way to parse the schema from that?
I'm not sure you know what the verb "to parse" actually means. Your phrase certainly suggests that you expect the file to contain a schema, which you want to extract. But I'm not at all sure you really mean that. And in any case, if it did contain a schema in the first 5Mb, you could find it just be reading the file sequentially, there would be no need to "save" the first part of the file first.

Question: way to parse the structure of the main item node without previously knowing its structure
This class TopSequenceElement parse a XML File to find all Sequence Elements.
The default is, to break at the FIRST closing </...> of the topmost Element.
Therefore, it is independend of the file size or even by truncated files.
from lxml import etree
from collections import OrderedDict
class TopSequenceElement(etree.iterparse):
"""
Read XML File
results: .seq == OrderedDict of Sequence Element
.element == topmost closed </..> Element
.xpath == XPath to top_element
"""
class Element:
"""
Classify a Element
"""
SEQUENCE = (1, 'SEQUENCE')
VALUE = (2, 'VALUE')
def __init__(self, elem, event):
if len(elem):
self._type = self.SEQUENCE
else:
self._type = self.VALUE
self._state = [event]
self.count = 0
self.parent = None
self.element = None
#property
def state(self):
return self._state
#state.setter
def state(self, event):
self._state.append(event)
#property
def is_seq(self):
return self._type == self.SEQUENCE
def __str__(self):
return "Type:{}, Count:{}, Parent:{:10} Events:{}"\
.format(self._type[1], self.count, str(self.parent), self.state)
def __init__(self, fh, break_early=True):
"""
Initialize 'iterparse' only to callback at 'start'|'end' Events
:param fh: File Handle of the XML File
:param break_early: If True, break at FIRST closing </..> of the topmost Element
If False, run until EOF
"""
super().__init__(fh, events=('start', 'end'))
self.seq = OrderedDict()
self.xpath = []
self.element = None
self.parse(break_early)
def parse(self, break_early):
"""
Parse the XML Tree, doing
classify the Element, process only SEQUENCE Elements
record, count of end </...> Events,
parent from this Element
element Tree of this Element
:param break_early: If True, break at FIRST closing </..> of the topmost Element
:return: None
"""
parent = []
try:
for event, elem in self:
tag = elem.tag
_elem = self.Element(elem, event)
if _elem.is_seq:
if event == 'start':
parent.append(tag)
if tag in self.seq:
self.seq[tag].state = event
else:
self.seq[tag] = _elem
elif event == 'end':
parent.pop()
if parent:
self.seq[tag].parent = parent[-1]
self.seq[tag].count += 1
self.seq[tag].state = event
if self.seq[tag].count == 1:
self.seq[tag].element = elem
if break_early and len(parent) == 1:
break
except etree.XMLSyntaxError:
pass
finally:
"""
Find the topmost completed '<tag>...</tag>' Element
Build .seq.xpath
"""
for key in list(self.seq):
self.xpath.append(key)
if self.seq[key].count > 0:
self.element = self.seq[key].element
break
self.xpath = '/'.join(self.xpath)
def __str__(self):
"""
String Representation of the Result
:return: .xpath and list of .seq
"""
return "Top Sequence Element:{}\n{}"\
.format( self.xpath,
'\n'.join(["{:10}:{}"
.format(key, elem) for key, elem in self.seq.items()
])
)
if __name__ == "__main__":
with open('../test/uyalicihow.xml', 'rb') as xml_file:
tse = TopSequenceElement(xml_file)
print(tse)
Output:
Top Sequence Element:Items/Item
Items :Type:SEQUENCE, Count:0, Parent:None Events:['start']
Item :Type:SEQUENCE, Count:1, Parent:Items Events:['start', 'end', 'start']
Main :Type:SEQUENCE, Count:2, Parent:Item Events:['start', 'end', 'start', 'end']
Info :Type:SEQUENCE, Count:2, Parent:Item Events:['start', 'end', 'start', 'end']
Genres :Type:SEQUENCE, Count:2, Parent:Item Events:['start', 'end', 'start', 'end']
Products :Type:SEQUENCE, Count:1, Parent:Item Events:['start', 'end']
... (omitted for brevity)
Step 2: Now, you know there is a <Main> Tag, you can do:
print(etree.tostring(tse.element.find('Main'), pretty_print=True).decode())
<Main>
<Platform>iTunes</Platform>
<PlatformID>353736518</PlatformID>
<Type>TVEpisode</Type>
<TVSeriesID>262603760</TVSeriesID>
</Main>
Step 3: Now, you know there is a <Platform> Tag, you can do:
print(etree.tostring(tse.element.find('Main/Platform'), pretty_print=True).decode())
<Platform>iTunes</Platform>
Tested with Python:3.5.3 - lxml.etree:3.7.1

For very big files, reading is always a problem. I would suggest a simple algorithmic behavior for the reading of the file itself. The key point is always the xml tags inside the files. I would suggest you read the xml tags and sort them inside a heap and then validate the content of the heap accordingly.
Reading the file should also happen in chunks:
import xml.etree.ElementTree as etree
for event, elem in etree.iterparse(xmL, events=('start', 'end', 'start-ns', 'end-ns')):
store_in_heap(event, element)
This will parse the XML file in chunks at a time and give it to you at every step of the way. start will trigger when a tag is first encountered. At this point elem will be empty except for elem.attrib that contains the properties of the tag. end will trigger when the closing tag is encountered, and everything in-between has been read.
you can also benefit from the namespaces that are in start-ns and end-ns. ElementTree has provided this call to gather all the namespaces in the file.
Refer to this link for more information about namespaces

My interpretation of your needs is that you want to be able to parse the partial file and build up the structure of the document as you go. I've taken some assumptions from the file you uploaded:
Fundamentally you want to be parsing collections of things which have similar properties - I'm inferring this from the way you presented your desired output as a table with rows containing the values.
You expect these collections of things to have the same number of values.
You need to be able to parse partial files.
You don't worry about the properties of elements, just their contents.
I'm using xml.sax as this deals with arbitrarily large files and doesn't need to read the whole file into memory. Note that the strategy I'm following now doesn't actually scale that well as I'm storing all the elements in memory to build the dataframe, but you could just as well output the paths and contents.
In the sample file there is a problem with having one row per Item since there are multiples of the Genre tag and there are also multiple Product tags. I've handled the repeated Genre tags by appending them. This relies on the Genre tags appearing consecutively. It is not at all clear how the Product relationships can be handled in a single table.
import xml.sax
from collections import defaultdict
class StructureParser(xml.sax.handler.ContentHandler):
def __init__(self):
self.text = ''
self.path = []
self.datalist = defaultdict(list)
self.previouspath = ''
def startElement(self, name, attrs):
self.path.append(name)
def endElement(self, name):
strippedtext = self.text.strip()
path = '/'.join(self.path)
if strippedtext != '':
if path == self.previouspath:
# This handles the "Genre" tags in the sample file
self.datalist[path][-1] += f',{strippedtext}'
else:
self.datalist[path].append(strippedtext)
self.path.pop()
self.text = ''
self.previouspath = path
def characters(self, content):
self.text += content
You'd use this like this:
parser = StructureParser()
try:
xml.sax.parse('uyalicihow.xml', parser)
except xml.sax.SAXParseException:
print('File probably ended too soon')
This will read the example file just fine.
Once this has read and probably printed "File probably ended to soon", you have the parsed contents in parser.datalist.
You obviously want to have just the parts which read successfully, so you can figure out the shortest list and build a DataFrame with just those paths:
import pandas as pd
smallest_items = min(len(e) for e in parser.datalist.values())
df = pd.DataFrame({key: value for key, value in parser.datalist.items() if len(value) == smallest_items})
This gives something similar to your desired output:
Items/Item/Main/Platform Items/Item/Main/PlatformID Items/Item/Main/Type
0 iTunes 353736518 TVEpisode
1 iTunes 495275084 TVEpisode
The columns for the test file which are matched here are
>> df.columns
Index(['Items/Item/Main/Platform', 'Items/Item/Main/PlatformID',
'Items/Item/Main/Type', 'Items/Item/Main/TVSeriesID',
'Items/Item/Info/BaseURL', 'Items/Item/Info/EpisodeNumber',
'Items/Item/Info/HighestResolution',
'Items/Item/Info/LanguageOfMetadata', 'Items/Item/Info/LastModified',
'Items/Item/Info/Name', 'Items/Item/Info/ReleaseDate',
'Items/Item/Info/ReleaseYear', 'Items/Item/Info/RuntimeInMinutes',
'Items/Item/Info/SeasonNumber', 'Items/Item/Info/Studio',
'Items/Item/Info/Synopsis', 'Items/Item/Genres/Genre',
'Items/Item/Products/Product/URL'],
dtype='object')
Based on your comments, it appears as though it is more important to you to have all the elements represented, but perhaps just showing a preview, in which case you can perhaps use only the first elements from the data. Note that in this case the Products entries won't match the Item entries.
df = pd.DataFrame({key: value[:smallest_items] for key, value in parser.datalist.items()})
Now we get all the paths:
>> df.columns
Index(['Items/Item/Main/Platform', 'Items/Item/Main/PlatformID',
'Items/Item/Main/Type', 'Items/Item/Main/TVSeriesID',
'Items/Item/Info/BaseURL', 'Items/Item/Info/EpisodeNumber',
'Items/Item/Info/HighestResolution',
'Items/Item/Info/LanguageOfMetadata', 'Items/Item/Info/LastModified',
'Items/Item/Info/Name', 'Items/Item/Info/ReleaseDate',
'Items/Item/Info/ReleaseYear', 'Items/Item/Info/RuntimeInMinutes',
'Items/Item/Info/SeasonNumber', 'Items/Item/Info/Studio',
'Items/Item/Info/Synopsis', 'Items/Item/Genres/Genre',
'Items/Item/Products/Product/URL',
'Items/Item/Products/Product/Offers/Offer/Price',
'Items/Item/Products/Product/Offers/Offer/Currency'],
dtype='object')

There are a number of tools around that will generate a schema from a supplied instance document. I don't know how many of them will work on a 5Gb input file, and I don't know how many of them can be invoked from Python.
Many years ago I wrote a Java-based, fully streamable tool to generate a DTD from an instance document. It hasn't been touched in years but it should still run: https://sourceforge.net/projects/saxon/files/DTDGenerator/7.0/dtdgen7-0.zip/download?use_mirror=vorboss
There are other tools listed here: Any tools to generate an XSD schema from an XML instance document?

As I see it your question is very clear. I give it a plus one up vote for clearness. You are wanting to parse text.
Write a little text parser, we can call that EditorB, that reads in chunks of the file or at least line by line. Then edit or change it as you like and re-save that chunk or line.
It can be easy in Windows from 98SE on. It should be easy in other operating systems.
The process is (1) Adjust (manually or via program), as you currently do, we can call this EditorA, that is editing your XML document, and save it; (2) stop EditorA; (3) Run your parser or editor, EditorB, on the saved XML document either manually or automatically (started via detecting that the XML document has changed via date or time or size, etc.); (4) Using EditorB, save manually or automatically the edits from step 3; (5) Have your EditorA reload the XML document and go on from there; (6) do this as often as is necessary, making edits with EditorA and automatically adjusting them outside of EditorA by using EditorB.
Edit this way before you send the file.
It is a lot of typing to explain, but XML is just a glorified text document. It can be easily parsed and edited and saved, either character by character or by larger amounts line by line or in chunks.
As a further note, this can be applied via entire directory contained documents or system wide documents as I have done in the past.
Make certain that EditorA is stopped before EditorB is allowed to start it's changing. Then stop EditorB before restarting EditorA. If you set this up as I described, then EditorB can be run continually in the background, but put in it an automatic notifier (maybe a message box with options, or a little button that is set formost on the screen when activated) that allows you to turn off (on continue with) EditorA before using EditorB. Or, as I would do it, put in a detector to keep EditorB from executing its own edits as long as EditorA is running.
B Lean

Use dictionary as lookup table

I have a jinja template that I want to pass a value into (an identifier for a country). The format of the data is a two letter country code (for example "PL" for Poland).
Through the template, I need to pass the corresponding flag as an output, but the flag is saved in the app folder structure, so i need to get the path to the image.
My problem: I could not figure out a way to use os.path in jinja, so now im trying to solve it by creating a dictionary that matches country string and relative path like this:
countries = {"PL" : "countries/flags/poland.png"}
where the system path to the app folder gets added afterwards in Python through os.path.
My question: How can i use the country string I am getting to automate transformation into the path format of the country? Something like:
for data in countries:
if data in countries.keys:
return countries.value
Thanks in advance!

Assuming data is a country code (like "PL" for example):
def get_path(data):
return countries.get(data)
The get() method checks if the dictionary has the key, and if so returns the corresponding value, otherwise it returns None.
If you want a default value other than None when the key is not present you can specify it as second argument, like this:
def get_path(data):
return countries.get(data, "default/path/x/y/z")

Using A Python List or String in Dictionary Lookup?

Use Case
I am making a factory type script in Python that consumes XML and based on that XML, returns information from a specific factory. I have created a file that I call FactoryMap.json that stores the mapping between the location an item can be found in XML and the appropriate factory.
Issue
The JSON in my mapping file looks like:
{
"path": "['project']['builders']['hudson.tasks.Shell']",
"class": "bin.classes.factories.step.ShellStep"
}
path is where the element can be found in the xml once its converted to a dict.
class is the corresponding path to the factory that can consume that elements information.
In order to do anything with this, I need to descend into the dictionaries structure, which would look like this if I didn't have to draw this information from a file(note the key reference = 'path' from my json'):
configDict={my xml config dict}
for k,v in configDict['project']['builders']['hudson.tasks.Shell'].iteritems():
#call the appropriate factory
The issue is that if I look up the path value as a string or a list, I can not use it in 'iteritems'():
path="['project']['builders']['hudson.tasks.Shell']" #this is taken from the JSON
for k,v in configDict[path].iteritems():
#call the appropriate factory
This returns a key error stating that I can't use a string as the key value. How can I used a variable as the key for that python dictionary?

You could use eval:
eval( "configDict"+path )

You can use the eval() function to evaluate your path into an actual dict object vs a string. Something like this is what I'm referring to:
path="['project']['builders']['hudson.tasks.Shell']" #this is taken from the JSON
d = eval("configDict%s" % path)
for k,v in d.iteritems():
#call the appropriate factory

Python: Transform a string into a variable and its value

I am relatively new on Python.
The program I am writing reads line by line a XML file using a while loop. The data read is split so the information that I get is something like:
datas = ['Name="Date"', 'Tag="0x03442333"', 'Level="Acquisition"', 'Type="String"']
-Inside my program, I want to assign to some variables called exactly as the word before the = sign, the information after the = sign in the previous strings. And then I will introduce them as attributes for a class (this already works)
- What I have done until the moment is:
Name = ''
Tag = ''
Level = ''
Type = ''
for i in datas:
exec(i)
-It works fine that way. However, I do not want to use the exec function. Is there any other way of doing that?
Thank you

exec is generally the way to go about this. You could also add it to the globals() dictionary directly, but this would be slightly dangerous sometimes.
for pair in datas:
name, value = pair.split("=")
globals()[name] = eval(value)

You are right that you should avoid exec for security reasons, and you should probably keep the field values in a dict or similar structure. It's better to let a Python library do the whole parsing. For example, using ElementTree:
import xml.etree.ElementTree as ET
tree = ET.parse('myfile.xml')
root = tree.getroot()
and then iterating over root and its children, depending on how exactly your XML data looks like.

At most what you expect to do is discussed here. To convert string to variable name.
But what you should ideally do is to create a dictionary. Like this.
for i in datas:
(key,value)=i.split("=")
d[key] = eval(value)
NOTE: Still avoid using eval.

As Pelle Nilsson says, you should use a proper XML parser for this. However, if the data is simple and the format of your XML file is stable, you can do it by hand.
Do you have a particular reason to put this data into a class? A dictionary may be all you need:
datas = ['Name="Date"', 'Tag="0x03442333"', 'Level="Acquisition"', 'Type="String"']
datadict = {}
for s in datas:
key, val = s.split('=')
datadict[key] = val.strip('"')
print(datadict)
output
{'Tag': '0x03442333', 'Type': 'String', 'Name': 'Date', 'Level': 'Acquisition'}
You can pass such a dictionary to a class, if you want:
class MyClass(object):
def __init__(self, data):
self.__dict__ = data
def __repr__(self):
s = ', '.join('{0}={1!r}'.format(k,v) for k, v in self.__dict__.items())
return 'Myclass({0})'.format(s)
a = MyClass(datadict)
print(a)
print(a.Name, a.Tag)
output
Myclass(Tag='0x03442333', Type='String', Name='Date', Level='Acquisition')
Date 0x03442333
All of the code in this answer should work correctly on any recent version of Python 2 as well as on Python 3, although if you run it on Python 2 you should put
from __future__ import print_function
at the top of the script.

Working with Parameters containing Escaped Characters in Python Config file

I have a config file that I'm reading using the following code:
import configparser as cp
config = cp.ConfigParser()
config.read('MTXXX.ini')
MT=identify_MT(msgtext)
schema_file = config.get(MT,'kbfile')
fold_text = config.get(MT,'fold')
The relevant section of the config file looks like this:
[536]
kbfile=MT536.kb
fold=:16S:TRANSDET\n
Later I try to find text contained in a dictionary that matches the 'fold' parameter, I've found that if I find that text using the following function:
def test (find_text)
return {k for k, v in dictionary.items() if find_text in v}
I get different results if I call that function in one of two ways:
test(fold_text)
Fails to find the data I want, but:
test(':16S:TRANSDET\n')
returns the results I know are there.
And, if I print the content of the dictionary, I can see that it is, as expected, shown as
:16S:TRANSDET\n
So, it matches when I enter the search text directly, but doesn't find a match when I load the same text in from a config file.
I'm guessing that there's some magic being applied here when reading/handling the \n character pattern in from the config file, but don't know how to get it to work the way I want it to.
I want to be able to parameterise using escape characters but it seems I'm blocked from doing this due to some internal mechanism.
Is there some switch I can apply to the config reader, or some extra parsing I can do to get the behavior I want? Or perhaps there's an alternate solution. I do find the configparser module convenient to use, but perhaps this is a limitation that requires an alternative, or even self-built module to lift data out of a parameter file.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.