Develop Reference

Merge two dictionaries in Jinja2

I have some code which basically searches some YAML files for Jinja2 variables and replaces them where appropriate. I want to be able to combine to YAML dicts together and was looking for a way to do this using the default Jinja2 library.
Here is some example YAML of what I am trying to a achieve:
vars.yml
vars:
default:
name: test1
location: UK
new:
name: test2
country: UK
template.yml
test: {{ vars.default.update(vars.new) }}
This would ideally output something like:
output.yml
test:
name: test2
location : UK
country: UK
I am looking for a way to merge two dictionaries together using this method. The Jinja2 documentation don't appear to have an inbuilt function like the Ansible combine filter.
Is it possible that I just import this feature into my code or create a similar feature? I noticed that it is not included in the standard library for Ansible Jinja2 filters
When trying to use the update filter (which doesn't appear to exist in the Jinja2 documentation) I get a None result.

The update method of a dictionary is actually updating the source dictionary, not returning the merged dictionaries.
update([other])
Update the dictionary with the key/value pairs from other, overwriting existing keys. Return None.
Source: https://docs.python.org/3/library/stdtypes.html#dict.update
So, you have two ways of achieving it:
either you have the expression-statement extension loaded and you can use the {% do ... %} construct:
{% do vars.default.update(vars.new) -%}
test: {{ vars.default }}
or it is not activated and you can use a dummy assignation:
{% set noop = vars.default.update(vars.new) -%}
test: {{ vars.default }}

As others have indicated, update operates in place and doesn't return the updated dictionary.
A possible solution is based on the fact the YAML format has its own built in way to base one dictionary on another:
import sys
from pathlib import Path
import ruamel.yaml
path = Path('vars.yaml')
path.write_text("""\
vars:
default: &base
name: test1
location: UK
new:
<<: *base
name: test2
country: UK
""")
yaml = ruamel.yaml.YAML(typ='safe')
data = yaml.load(path)
print(data['vars']['new'])
which gives:
{'name': 'test2', 'location': 'UK', 'country': 'UK'}
So if Ansible uses a standard YAML library that implements the merge key, you can simply make your Jinja2 template:
test: {{ vars.new }}
, but this requires you to "specify" the merge in you YAML input and not in your Jinja2 file,
and that might not be an option.
Please note that the recommended file extension for files containing YAML
files has been .yaml since at least September 2006.
There is also a YML format, which is XML based and at least as old as YAML,
so you should exclusively use .yml for that format.

I managed to do it by copying the combine function (or something like it) from the ansible libary. Here is what I made.
my_filters.py
def combine(*args):
ret = {}
for arg in args:
if arg is None:
continue
if isinstance(arg, dict):
ret.update(arg)
else:
raise ValueError("combine filter arguments must be dictionaries")
return ret
When inside my code (I can't post all of it), I updated my library dependencies
from jinja2 import Environment, FileSystemLoader
from my_filters import combine
"""
omitted stuff
"""
env = Environment(loader=FileSystemLoader('.'))
env.filters['combine'] = combine
yaml = env.from_string(yaml_file)

How to grab dict value from string of square bracket notation

Let's say I have the dictionary my_dict:
my_dict = {"vms": {"vm": {"name": "test-vm", "desc": "test"}}}
Obviously, in code I can grab the VM data with:
my_dict['vms']['vm']
More to the point, I can create an object of vms with that value and then reference the data:
vms = my_dict['vms']
vm = vms['vm']
My question is, if I were to save the value vms['vm'] as a string in a configuration file, how can I then use said string to grab the value from the dictionary? Something like:
my_dict.grab("vms['vm']") # To grab my_dict['vms']['vm']
I'm hoping a solution that I can further use with, say, vms['vm']['name'] regardless of how nested it gets, akin to Ansible Facts in a YAML playbook:
my_dict.grab("vms['vm']['name']")

So it turns out what I was looking for was Jinja2
from jinja2 import Template
my_dict = {"vms": {"vm": {"name": "test-vm", "desc": "test"}}}
string_location = "{{ vms['vm'] }}"
print(Template(string_location).render(my_dict))

Dictionary lookup of loaded yaml via a string representation in Python

So I am using ruamel to read a yaml file located in github and all goes well. I can see it loaded correctly. Now my scenario is that this load is in a function in a class that I am accessing. Now this function has a string variable "entry" which is what I want to search for. The goal is to search at varying depths and I know the locations so I am not hunting for it.
Sample Yaml file:
description: temp_file
image: beta1
group: dev
depends:
- image: base
basetag: 1.0.0
Entry string variable passing in
I want to keep this a string for the variable "entry" as if looking for a top level lookup using the .get will return my value just fine. Its just if I want to gather something like like the value of ["depends"][0]["image"], I cannot figure out how construct this so I can do the proper get.
entry = "image" # works fine
entry = '["depends"][0]["image"]' #never works
gho.get_entry_from_ivpbuild_yml(repo, commit, entry)
Code in Class
# imports
from ruamel.yaml import YAML
yaml = YAML()
yaml.preserve_quotes = True
def get_entry_from_loaded_yml(self, repo, commit, entry, failflag=True):
"""
:param repo: str (github repo)
:param commit: str (sha of commit to view)
:param entry: str (entry within yml you want to search for)
:param failflag: bool (Determines if script fails or not if entry is not found within yaml)
:return: str: (value of entry in yml you want to search for)
"""
yml_file = "sample.yaml"
try:
logger.debug("opening yaml for commit:{}".format(commit))
yml = self.gho.get_repo(repo).get_file_contents(yml_file, commit)
except Exception as e:
logger.error("Could not open yaml file:{} for repo:{} commit:{}:{}".format(yml_file, repo, commit, e))
sys.exit(1)
loaded = yaml.load(yml.decoded_content)
if not loaded.get(entry, default=None):
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None

As you already know, you can't pass something like the string '["depends"][0]["image"]' to dict.get and expect that to work. But there are a couple of options if you really need to specify a "path" to the object within a nested data structure like this.
The first is to do it explicitly, just passing a sequence of keys instead of a single key:
def get_entry_from_loaded_yml(self, repo, commit, entry_keys, failflag=True):
# ...
try:
entry = loaded
for key in entry_keys:
entry = entry[key]
except LookupError:
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None
else:
return entry
And now, you can do this:
gho.get_entry_from_ivpbuild_yml(repo, commit, ('depends', 0, 'image'))
Alternatively, you can use a library that handles "key paths", in a format like dpath (which is essentially a simplified version of XPath) or ObjectiveC's KVC. There are multiple libraries on PyPI for doing this (although some work on undecoded JSON strings rather than decoded nested objects, to allow searching huge JSON texts efficiently; those obviously won't work for you… and I don't know of any that work on YAML instead of JSON, but they might exist). Then your code would look something like this:
def get_entry_from_loaded_yml(self, repo, commit, entry, failflag=True):
# ...
result = dpath_search(loaded, entry, default=None):
if result is None:
logger.error("Could not find value for {} in {}".format(entry, yml_file))
if failflag:
sys.exit(1)
return None
else:
return result
# ...
gho.get_entry_from_ivpbuild_yml(repo, commit, 'depends/0/image')
This has the advantage that if you ever need to look up a (possibly nested) sequence of multiple values, it can be as simple as this:
for result in dpath_search(loaded, 'depends/*/image'):

Python insert YAML in MongoDB

Folks,
Having a problem with inserting the following yaml document into MongoDB:
works:
---
URLs:
- "http://www.yahoo.com":
intensity: 5
port: 80
does not:
---
URLs:
- "foo":
intensity: 5
port: 80
The only difference is the url. Here is the python code:
stream = open(options.filename, 'r')
yamlData = yaml.load(stream)
jsonData = json.dumps(yamlData)
io = StringIO(jsonData)
me = json.load(io)
... calling classes, etc, then
self.appCollection.insert(me)
err:
bson.errors.InvalidDocument: key 'http://yahoo.com' must not contain '.'
So, what is the correct way to transform this YML file? :)
Thanks!

You cannot use "." in field names (i.e. keys). If you must, then replace occurences of "." with the unicode representation "\uff0E".
Hope this helps.

As the errors says, you have errors in your key. MongoDB uses dot for nested document keys, you cannot have a key that contains dot as part of the key.

Building a set of records incrementally as app progresses

I have an sysadmin type CLI app that reads in info from a config file. I cannot change the format of the config file below.
TYPE_A = "value1,value2,value2"
TYPE_A = "value3,value4,value5"
TYPE_B = "valuex,valuey,valuez"
Based on the TYPE, I'll need to do some initial processing with each one. After I'm done with that step for all, I need to do some additional processing and depending on the options chosen either print the state and intended action(s) or execute those action(s).
I'd like to do the initial parsing of the config into a dict of lists of dicts and update every instance of TYPE_A, TYPE_B, TYPE_C, etc with all the pertinent info about it. Then either print the full state or execute the actions (or fail if the state of something was incorrect)
My thought is it would look something like:
dict
TYPE_A_list
dict_A[0] key:value,key:value,key:value
dict_A[1] key:value,key:value,key:value
TYPE_B_list
dict_A[0] key:value,key:value,key:value
dict_A[1] key:value,key:value,key:value
I think I'd want to read the config into that and then add keys and values or update values as the app progresses and reprocesses each TYPE.
Finally my questions.
I'm not sure how iterate over each list of dicts or to add list elements and add or update key:value pairs.
Is what I describe above the best way to go about this?
I'm fairly new to Python, so I'm open to any advice. FWIW, this will be python 2.6.
A little clarification on the config file lines
CAR_TYPE = "Ford,Mustang,Blue,2005"
CAR_TYPE = "Honda,Accord,Green,2009"
BIKE_TYPE = "Honda,VTX,Black,2006"
BIKE_TYPE = "Harley,Sportster,Red,2010"
TIRE_TYPE = "170R15,whitewall"
Each type will have the same order and number of values.

No need to "remember" there are two different TYPE_A assignments - you can combine them.
TYPE_A = "value1,value2,value2"
TYPE_A = "value3,value4,value5"
would be parsed as only one of them, or both, depends on the implementation of your sysadmin CLI app.
Then the data model should be:
dict
TYPE_A: list(value1, value2, value3)
TYPE_B: list(valuex, valuey, valuez)
That way, you can iterate through dict.items() pretty easily:
for _type, values in dict.items():
for value in values:
print "%s: %s" % (_type, value)
# or whatever you wish to do