I am new in saltstack and i have some troubles creating a python function to make some regex checks.
i have this function
from re import sub, match, search
def app_instance_match(app):
instance_no = 0
m = search('^(.*)(-)(\d)$', app)
if m is not None:
app = m.group(1)
instance_no = int(m.group(3))
return app, instance_no
when i call it from console with
salt-ssh -i 'genesis-app-1' emod.app_instance_match test-14
i get
$ salt-ssh -i 'genesis-app-1' emod.app_instance_match test-14
genesis-app-1:
- test-14
- 0
When i try to use it inside a sls file like
{% set app = salt['emod.app_instance_match'](app) %}
i cannot use the app anymore. i tried
{% for x,y in app %}
test:
cmd.run:
- names:
- echo {{x} {{y}}
or like
cmd.run:
- names:
- echo {{app}}
I know that it return to me a dictionary but i am unable to access the values of it. The only thing that i need is the 2 returns from the python function: test-14 and 0.
when i echo for testing the X from the loop fox x,y in app i saw values like retcode, stdout, stderror.
Is there any other way to syntax the
{% set app = salt['emod.app_instance_match'](app) %}
something like that so will have 2 set variables in sls
{% set app,no = salt['emod.app_instance_match'](app) %}
i also tried like
{% set app = salt['emod.app_instance_match'](app).items() %}
I am missing something in the syntax but i cannot find anything in the internet to help me continue. I have the values that i want inside app, but i am not able to access them to take the part that i want.
First, You are not getting a dict back, you are getting a tuple back. there is a big difference. second {% set app,no = salt['emod.app_instance_match'](app) %} is exactly what you should be using. that will split the variables into two parts app and no. I should note sometimes using salt-ssh actually makes debugging things in salt harder. I would suggest installing a local minion to at least test these basic things.
Here is an example using your own code. I named it epp instead of emod.
[root#salt00 tests]# cat tests.sls
{% set x,y = salt['epp.app_instance_match']('test-14') %}
x: {{x}}
y: {{y}}
[root#salt00 tests]# salt-call slsutil.renderer salt://tests/tests.sls default_render=jinja
local:
----------
x:
test-14
y:
0
[root#salt00 tests]# cat ../_modules/epp.py
from re import sub, match, search
def app_instance_match(app):
instance_no = 0
m = search('^(.*)(-)(\d)$', app)
if m is not None:
app = m.group(1)
instance_no = int(m.group(3))
return app, instance_no
The second thing is you might want to look at https://docs.saltproject.io/en/latest/topics/jinja/index.html#regex-search which is already a regex search.
And third. Your regex looks off. ^ and $ don't really work well with single strings. which would explain why test-14 didn't come back as ('test',1) but instead came back as ('test-14',0)
I'm thinking you want '(.*)-(\d*)' as your real regex. which will return ('test',14) for test-14
Related
I am new in the Airflow world and trying to understand one thing. For example I have a DAG that contains 2 tasks. The first task is submitting spark job, and the second one is Sensor that waits for a file in s3.
RUN_DATE_ARG = datetime.utcnow().strftime(DATE_FORMAT_PY)
DATE = datetime.strptime(RUN_DATE_ARG, DATE_FORMAT_PY) - timedelta(hours=1)
with DAG() as dag:
submit_spark_job = EmrContainerOperator(
task_id="start_job",
virtual_cluster_id=VIRTUAL_CLUSTER_ID,
execution_role_arn=JOB_ROLE_ARN,
release_label="emr-6.3.0-latest",
job_driver=JOB_DRIVER_ARG,
configuration_overrides=CONFIGURATION_OVERRIDES_ARG,
name=f"spark-{RUN_DATE_ARG}",
retries=3
)
validate_s3_success_file = S3KeySensor(
task_id='check_for_success_file',
bucket_name="bucket-name",
bucket_key=f"blabla/date={DATE.strftime('%Y-%m-%d')}/hour={DATE.strftime('%H')}/_SUCCESS",
poke_interval=10,
timeout=60,
verify=False,
)
I have a RUN_DATE_ARG that by default should be taken from datetime.utcnow() and this is one of sparks java arguments that I should provide to my job.
I want to add an ability to submit job with custom date argument (via airflow UI).
When I am trying to retrieve it as '{{ dag_run.conf["date"] | None}}' it replaces with value inside task configuration (bucket_key=f"blabla/date={DATE.strftime('%Y-%m-%d')}/hour={DATE.strftime('%H')}/_SUCCESS",), but not for DAG's python code if I do following:
date='{{ dag_run.conf["date"] | None}}'
if date is None:
RUN_DATE_ARG = datetime.utcnow().strftime(DATE_FORMAT_PY)
else:
RUN_DATE_ARG = date
Do I have any way to use this value as a code variable?
You can not use templating outside of operators scope.
You should use Jinja if statements in the operator templated parameter. The following is just a general idea:
submit_spark_job = EmrContainerOperator(
task_id="start_job",
...
name="spark-{{ dag_run.conf["date"] if dag_run.conf["date"] is not None else jinja_utc_now }}",
)
You will need to replace jinja_utc_now with code that retrieve the timestamp probably something like what is shown in this answer.
You can also use:
{% if something %}
code
{% else %}
another code
{% endif %}
From Airflow point of view it takes the parameter and pass it though Jinja engine for templating so the key issue here is just to use the proper Jinja syntax.
I'm trying to refacto some pretty heavy template with jinja2 and I'm stucked on an include.
This is the behaviour i'm expecting :
<h1>{{ key }} </h1>
{% set file = key | include_text %}
{% include file %}
The custom filter returns a string like this one ::
texts/my_include.html
But instead I got this error:
jinja2.exceptions.TemplatesNotFound: Tried to select from an empty list of templates
Some hack I've already tried :
Place the templates in the same folder and remove the 'texts/' from the returned string
Add the path in the Env loader
But it keeps sending this error
I'm now wondering if jinja2 allows this implementation or if I'll have to keep this template the way it was (even if it takes a very long time to be generated).
Does someone know about some trick here ?
Well, for those who eventually met this problem in the futur, I've solved it by removing the unecessary single quotes and by sending some empty file from my custom filter when the condition is not verified... (my mistake)
Here is my custom filter :
#environmentfilter
def include_text(ctx, key):
res_dict = {
'key_value_1' : 'file_name_1',
'key_value_2' : 'file_name_2'
}
try:
return "texts/" + res_dict[key] + ".html"
except KeyError:
return "texts/empty.html"
Now, the first solution I was trying works fine.
I am trying to use jinja2 templates. I have custom filter called highlight, that takes string and language name and passes them to pyhments for code highlightning. I am trying to use it like this:
{% filter highlight("python") %}
import sys
def main():
pass
{% endfilter %}
But I get this error:
AttributeError: 'str' object has no attribute 'get_tokens'
Then I tried this:
{% filter highlight "python" %}
It does not work either.
There might be a trick via set block filtering and then pasting it back via {{ ... }}, but this technique is not merged in master source code yet, and seems too hacky for me.
So, is that even possible currently, or I am just doing it wrong?
EDIT: Here is filter:
#jinja2.contextfilter
def highlight(context, code, lang):
print("HIGHLIGHT")
print(code)
return jinja2.Markup(pygments.highlight(code, lexer=lang, formatter='html'))
I am an idiot, that was pygments error. By some mistake, I didn't see that last entry in stacktrace was from there.
You should use:
pygments.highlight(
code,
lexer=pygments.lexers.get_lexer_by_name(lang),
formatter=pygments.formatters.get_formatter_by_name('html')
)
instead of:
pygments.highlight(code, lexer=lang, formatter='html')
I have two Python files and one HTML file. One of the Python files is using Flask to connect with the HTML file.
In file1.py(the non Flask one) I set a for loop to print the variable volume
for volume in current_volumes:
print volume
which prints out two strings in Terminal
Volume:vol-XXXXXXX
Volume:vol-YYYYYYY
Now I put from file1 import * on the top of file2.py.
Additionally, file2.py contains
def template(name=volume):
return render_template('index.html', name=name)
Index.html contains
<p>{{ name }}</p>
but only reads Volume:vol-YYYYYYY when launched.
How do I get it to print out both values of volume?
I think you want to use a for loop to create a new string:
volume_string = ""
for volume in current_volumes:
volume_string += volume
def template(name=volume_string):
...
You can insert a "\n" (newline) at the end of every volume appended to get it to 2 print lines.
I haven't played around with Flask, but you may also want to just try
def template(name=current_volumes):
Perhaps it's smart enough to make that work.
You are using an escaping for variable volume rather than the list of volumes (current_volumes). (Should you switch to Python 3 this will raise a ReferenceError instead of working). Change:
def template(name=volume):
return render_template('index.html', name=name)
to:
def template(name=current_volumes):
return render_template('index.html', name=name)
You will also want to change your {{ name }} to a loop - let's go ahead and change the name:
def template(volumes=current_volumes):
return render_template('index.html', volumes=volumes)
and then add a loop in our Jinja template:
{% for volume in volumes %}
<p>Volume Data: {{ volume }}</p>
{% endfor %}
I'm using Jinja on my site and I like it.
I've come across a simple need. How to display today's date? Is there a way to inline some Python code in a Jinja template?
import datetime
now = datetime.datetime.utcnow()
print now.strftime("%Y-%m-%d %H:%M")
This article says no, but suggests using a macro or a filter?
Really? Must we resort to all that? OK, what would that look like in this case?
No, there is no way to inline Python into Jinja. However, you can add to the constructs that Jinja knows by extending the Environment of the template engine or the global namespace available to all templates. Alternately, you can add a filter that let's you format datetime objects.
Flask stores the Jinja2 Environment on app.jinja_env. You can inject new context into the environment by either adding to this dictionary directly, or by using the #app.context_processor decorator.
Whatever path you choose, this should be done while you are setting up the application, before you have served any requests. (See the snippets section of the website for some good examples of how to set up filters - the docs contain a good example of adding to the global variables).
The current answers are correct for pretty much every situation. However there are some very rare cases where you would want to have python code inside the template. In my case I want to use it to preprocess some latex files and I would prefer to keep the python code generating table values, plots, etc, inside the latex file it self.
So I made a Jinja2 extension that adds a new "py" block allowing python code to be written inside the template. Please keep in mind that I had to do some questionable work-arounds to get this to work, so I'm not 100% sure in which situations it fails or behaves unexpectedly.
This is an example template.
Foo was given to the template
foo: {{ foo }}
Bar was not, so it is missing
bar is missing: {{ bar == missing }}
{% py %}
# Normal python code in here
# Excess indentation will be removed.
# All template variables are accessible and can be modified.
import numpy as np
a = np.array([1, 2])
m = np.array([[3, 4], [5, 6]])
bar = m # a * foo
# It's also possible to template the python code.
{% if change_foo %}
foo = 'new foo value'
{% endif %}
print("Stdio is redirected to the output.")
{% endpy %}
Foo will have the new value if you set change_foo to True
foo: {{ foo }}
Bar will now have a value.
bar: {{ bar }}
{% py %}
# The locals from previous blocks are accessible.
m = m**2
{% endpy %}
m:
{{ m }}
The output if we set the template parameters to foo=10, change_foo=True is:
Foo was given to the template
foo: 10
Bar was not, so it is missing
bar is missing: True
Stdio is redirected to the output.
Foo will have the new value if you set change_foo to True
foo: new foo value
Bar will now have a value.
bar: [110 170]
m:
[[ 9 16]
[25 36]]
The extension with a main function to run the example.
from jinja2 import Environment, PackageLoader, nodes
from jinja2.ext import Extension
from textwrap import dedent
from io import StringIO
import sys
import re
import ctypes
def main():
env = Environment(
loader=PackageLoader('python_spike', 'templates'),
extensions=[PythonExtension]
)
template = env.get_template('emb_py2.txt')
print(template.render(foo=10, change_foo=True))
var_name_regex = re.compile(r"l_(\d+)_(.+)")
class PythonExtension(Extension):
# a set of names that trigger the extension.
tags = {'py'}
def __init__(self, environment: Environment):
super().__init__(environment)
def parse(self, parser):
lineno = next(parser.stream).lineno
body = parser.parse_statements(['name:endpy'], drop_needle=True)
return nodes.CallBlock(self.call_method('_exec_python',
[nodes.ContextReference(), nodes.Const(lineno), nodes.Const(parser.filename)]),
[], [], body).set_lineno(lineno)
def _exec_python(self, ctx, lineno, filename, caller):
# Remove access indentation
code = dedent(caller())
# Compile the code.
compiled_code = compile("\n"*(lineno-1) + code, filename, "exec")
# Create string io to capture stdio and replace it.
sout = StringIO()
stdout = sys.stdout
sys.stdout = sout
try:
# Execute the code with the context parents as global and context vars and locals.
exec(compiled_code, ctx.parent, ctx.vars)
except Exception:
raise
finally:
# Restore stdout whether the code crashed or not.
sys.stdout = stdout
# Get a set of all names in the code.
code_names = set(compiled_code.co_names)
# The the frame in the jinja generated python code.
caller_frame = sys._getframe(2)
# Loop through all the locals.
for local_var_name in caller_frame.f_locals:
# Look for variables matching the template variable regex.
match = re.match(var_name_regex, local_var_name)
if match:
# Get the variable name.
var_name = match.group(2)
# If the variable's name appears in the code and is in the locals.
if (var_name in code_names) and (var_name in ctx.vars):
# Copy the value to the frame's locals.
caller_frame.f_locals[local_var_name] = ctx.vars[var_name]
# Do some ctypes vodo to make sure the frame locals are actually updated.
ctx.exported_vars.add(var_name)
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(caller_frame),
ctypes.c_int(1))
# Return the captured text.
return sout.getvalue()
if __name__ == "__main__":
main()
You can add to global variables which can be accessed from Jinja templates. You can put your own function definitions in there, which do whatever you need.