Merge two dictionaries in Jinja2 - python

I have some code which basically searches some YAML files for Jinja2 variables and replaces them where appropriate. I want to be able to combine to YAML dicts together and was looking for a way to do this using the default Jinja2 library.
Here is some example YAML of what I am trying to a achieve:
vars.yml
vars:
default:
name: test1
location: UK
new:
name: test2
country: UK
template.yml
test: {{ vars.default.update(vars.new) }}
This would ideally output something like:
output.yml
test:
name: test2
location : UK
country: UK
I am looking for a way to merge two dictionaries together using this method. The Jinja2 documentation don't appear to have an inbuilt function like the Ansible combine filter.
Is it possible that I just import this feature into my code or create a similar feature? I noticed that it is not included in the standard library for Ansible Jinja2 filters
When trying to use the update filter (which doesn't appear to exist in the Jinja2 documentation) I get a None result.


The update method of a dictionary is actually updating the source dictionary, not returning the merged dictionaries.
update([other])
Update the dictionary with the key/value pairs from other, overwriting existing keys. Return None.
Source: https://docs.python.org/3/library/stdtypes.html#dict.update
So, you have two ways of achieving it:
either you have the expression-statement extension loaded and you can use the {% do ... %} construct:
{% do vars.default.update(vars.new) -%}
test: {{ vars.default }}
or it is not activated and you can use a dummy assignation:
{% set noop = vars.default.update(vars.new) -%}
test: {{ vars.default }}


As others have indicated, update operates in place and doesn't return the updated dictionary.
A possible solution is based on the fact the YAML format has its own built in way to base one dictionary on another:
import sys
from pathlib import Path
import ruamel.yaml
path = Path('vars.yaml')
path.write_text("""\
vars:
default: &base
name: test1
location: UK
new:
<<: *base
name: test2
country: UK
""")
yaml = ruamel.yaml.YAML(typ='safe')
data = yaml.load(path)
print(data['vars']['new'])
which gives:
{'name': 'test2', 'location': 'UK', 'country': 'UK'}
So if Ansible uses a standard YAML library that implements the merge key, you can simply make your Jinja2 template:
test: {{ vars.new }}
, but this requires you to "specify" the merge in you YAML input and not in your Jinja2 file,
and that might not be an option.
Please note that the recommended file extension for files containing YAML
files has been .yaml since at least September 2006.
There is also a YML format, which is XML based and at least as old as YAML,
so you should exclusively use .yml for that format.


I managed to do it by copying the combine function (or something like it) from the ansible libary. Here is what I made.
my_filters.py
def combine(*args):
ret = {}
for arg in args:
if arg is None:
continue
if isinstance(arg, dict):
ret.update(arg)
else:
raise ValueError("combine filter arguments must be dictionaries")
return ret
When inside my code (I can't post all of it), I updated my library dependencies
from jinja2 import Environment, FileSystemLoader
from my_filters import combine
"""
omitted stuff
"""
env = Environment(loader=FileSystemLoader('.'))
env.filters['combine'] = combine
yaml = env.from_string(yaml_file)

Related

Conditional access to items in Python

I want to write some tests for Kubernetes with python. This is a sample of my deployment file in Kubernetes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-service
namespace: test
creationTimestamp: '2022-12-08T18:18:49Z'
labels:
app: test-service
team: teamA
.
.
what I want to do is get access to the items inside the deployment file:
some codes here....
result = set()
some codes here.....
with open(deoloyment_file, "r") as stream:
for data in yaml.safe_load_all(stream.read().replace('\t', ' ')):
if data and data['kind'] == 'Deployment':
result.add(f"{data['metadata']['namespace']}:{data['metadata']['name']}:{data['metadata']['labels']['team']}")
This throws an exception because in some deployments files there are no labels or team. I'm wondering how can I conditionally access items with Python.

You can specify a default value for the dict.get method to make it fall back to the value when a given key does not exist.
Assuming you want an empty string when either labels or team doesn't exist, you can change:
data['metadata']['labels']['team']
to:
data['metadata'].get('labels', {}).get('team', '')

How to pass Airflow 'ds' default variable?

I'm new to Airflow and a bit confused with Default Variables. I have custom operator and I want to use ds in file name.
file_name = str(f) + '{{ ds }}' + str(ext)
The actual result I get is file{{ds}}.docx but I expect file2020-01-01.docx

The strings containing curly braces (e.g.: {{ ds }}) are templated values where Airflow renders certain operator attributes through Jinja2 templating, providing details regarding the current run.
This documentation outlines how to use templating in a custom operator: https://airflow.apache.org/docs/stable/howto/custom-operator.html#templating

How do I populate a list field variable in Ansible (jinja2, yaml) without inserting the unicode 'u' prefix?

For my python application, I am trying to create and populate a list field in a config file created and deployed by Ansible. Right now it just has one list element (phone_number_), in the future it could have more, hence being a list.
In my jinja2 yaml template I have:
FIELD_NAMES: {{field_names}}
And in my variable var yaml file I have:
field_names: ['phone_number_']
The resulting config file deployed to be server has this line produced:
FIELD_NAMES: [u'phone_number_']
I don't want that unicode "u" in there. How can I get rid of it?

There are to_json and to_yaml filters, that may be helpful.
FIELD_NAMES: {{ field_names | to_json }}
Will give you clean JSON list without unicode markers:
FIELD_NAMES: ["phone_number_"]

You have defined a list with one element:
field_names: ['phone_number_']
I see two options:
1) Reference the variable in your jinja2 template as:
FIELD_NAMES: {{field_names[0]}}
2) Define the variable as a string:
field_names: 'phone_number_'

PyYAML : Control ordering of items called by yaml.load()

I have a yaml setting file which creates some records in db:
setting1:
name: [item,item]
name1: text
anothersetting2:
name: [item,item]
sub_setting:
name :[item,item]
when i update this file with setting3 and regenerate records in db by:
import yaml
fh = open('setting.txt', 'r')
setting_list = yaml.load(fh)
for i in setting_list:
add_to_db[i]
it's vital that the order of them settings (id numbers in db) stay the same each time as im addig them to the db... and setting3 just gets appended to the yaml.load()'s end so that its id doesn't confuse any records which are already in the db ...
At the moment each time i add another setting and call yaml.load() records get loaded in different order which results in different ids. I would welcome any ideas ;)
EDIT:
I've followed abarnert tips and took this gist https://gist.github.com/844388
Works as expected thanks !

My project oyaml is a drop-in replacement for PyYAML, which will load maps into collections.OrderedDict instead of regular dicts. Just pip install it and use as normal - works on both Python 3 and Python 2.
Demo with your example:
>>> import oyaml as yaml # pip install oyaml
>>> yaml.load('''setting1:
... name: [item,item]
... name1: text
... anothersetting2:
... name: [item,item]
... sub_setting:
... name :[item,item]''')
OrderedDict([('setting1',
OrderedDict([('name', ['item', 'item']), ('name1', 'text')])),
('anothersetting2',
OrderedDict([('name', ['item', 'item']),
('sub_setting', 'name :[item,item]')]))])
Note that if the stdlib dict is order preserving (Python >= 3.7, CPython >= 3.6) then oyaml will use an ordinary dict.

You can now use ruaml.yaml for this.
From https://pypi.python.org/pypi/ruamel.yaml:
ruamel.yaml is a YAML parser/emitter that supports roundtrip
preservation of comments, seq/map flow style, and map key order

The YAML spec clearly says that the key order within a mapping is a "representation detail" that cannot be relied on. So your settings file is already invalid if it's relying on the mapping, and you'd be much better off using valid YAML, if at all possible.
Of course YAML is extensible, and there's nothing stopping you from adding an "ordered mapping" type to your settings files. For example:
!omap setting1:
name: [item,item]
name1: text
!omap anothersetting2:
name: [item,item]
!omap sub_setting:
name :[item,item]
You didn't mention which yaml module you're using. There is no such module in the standard library, and there are at least two packages just on PyPI that provide modules with that name. However, I'm going to guess it's PyYAML, because as far as I know that's the most popular.
The extension described above is easy to parse with PyYAML. See http://pyyaml.org/ticket/29:
def omap_constructor(loader, node):
return loader.construct_pairs(node)
yaml.add_constructor(u'!omap', omap_constructor)
Now, instead of:
{'anothersetting2': {'name': ['item', 'item'],
'sub_setting': 'name :[item,item]'},
'setting1': {'name': ['item', 'item'], 'name1': 'text'}}
You'll get this:
(('anothersetting2', (('name', ['item', 'item']),
('sub_setting', ('name, [item,item]'),))),
('setting1', (('name', ['item', 'item']), ('name1', 'text'))))
Of course this gives you a tuple of key-value tuples, but you can easily write a construct_ordereddict and get an OrderedDict instead. You can also write a representer that stores OrdereredDict objects as !omaps, if you need to output as well as input.
If you really want to hook PyYAML to make it use an OrderedDict instead of a dict for default mappings, it's pretty easy to do if you're already working directly on parser objects, but more difficult if you want to stick with the high-level convenience methods. Fortunately, the above-linked ticket has an implementation you can use. Just remember that you're not using real YAML anymore, but a variant, so any other software that deals with your files can, and likely will, break.

For a given single item that is known to be an ordered dictionary just make the items of a list and used collections.OrderedDict:
setting1:
- name: [item,item]
- name1: text
anothersetting2:
- name: [item,item]
- sub_setting:
name :[item,item]
import collections
import yaml
fh = open('setting.txt', 'r')
setting_list = yaml.load(fh)
setting1 = collections.OrderedDict(list(x.items())[0] for x in setting_list['setting1'])

Last I heard, PyYAML did not support this, though it would probably be easy to modify it to accept a dictionary or dictionary-like object as a starting point.

Get translatable text from an external source into catalog

Possibly i am overlooking an obvious solution or thinking the wrong way...
I have a limited amount of text, words in a database, that I want to display translated to users in a flask/jinja/babel webapp. eg. "running" is a possible value of an "activity" column and that should be "laufen" for my german users.
Words in templates and code are extracted and put into the catalog, but how do i get additional words into the catalog? Is there a simple text file extractor?
The only thing i could think of is, just create a .py file and put lots of _('...') lines in them, but that feels just wrong... is it?

I created a messages.txt with my "words" like gettext function calls:
_('cycling')
_('running')
and added it to my babel.cfg as python source:
[python: messages.txt]
plain, simple, stupid, but works.

First, start with http://flask.pocoo.org/snippets/4/.
Secondly, you need to store these 'limited' values as integers or enums in database and then create the lookup table for all these enums in code (so Babel knows about them):
i18n_val = {0: _('running'), ...}
# Or multi-level dict with different categories:
i18n_all = {
'activity': {
0: _('running'), ...
'foo': {
0: _('bar..'), ...
}
}
And accessing the translated string from template is now as simple as:
{{ i18n_val[obj.activity] }}
{{ i18n_all['activity'][obj.activity] }}
In order to make the i18n_val and i18n_all variables available for all the templates, just register them with context processors.

Categories

Resources