Develop Reference

How I can use Airflow template reference in the DAG python code

I am new in the Airflow world and trying to understand one thing. For example I have a DAG that contains 2 tasks. The first task is submitting spark job, and the second one is Sensor that waits for a file in s3.
RUN_DATE_ARG = datetime.utcnow().strftime(DATE_FORMAT_PY)
DATE = datetime.strptime(RUN_DATE_ARG, DATE_FORMAT_PY) - timedelta(hours=1)
with DAG() as dag:
submit_spark_job = EmrContainerOperator(
task_id="start_job",
virtual_cluster_id=VIRTUAL_CLUSTER_ID,
execution_role_arn=JOB_ROLE_ARN,
release_label="emr-6.3.0-latest",
job_driver=JOB_DRIVER_ARG,
configuration_overrides=CONFIGURATION_OVERRIDES_ARG,
name=f"spark-{RUN_DATE_ARG}",
retries=3
)
validate_s3_success_file = S3KeySensor(
task_id='check_for_success_file',
bucket_name="bucket-name",
bucket_key=f"blabla/date={DATE.strftime('%Y-%m-%d')}/hour={DATE.strftime('%H')}/_SUCCESS",
poke_interval=10,
timeout=60,
verify=False,
)
I have a RUN_DATE_ARG that by default should be taken from datetime.utcnow() and this is one of sparks java arguments that I should provide to my job.
I want to add an ability to submit job with custom date argument (via airflow UI).
When I am trying to retrieve it as '{{ dag_run.conf["date"] | None}}' it replaces with value inside task configuration (bucket_key=f"blabla/date={DATE.strftime('%Y-%m-%d')}/hour={DATE.strftime('%H')}/_SUCCESS",), but not for DAG's python code if I do following:
date='{{ dag_run.conf["date"] | None}}'
if date is None:
RUN_DATE_ARG = datetime.utcnow().strftime(DATE_FORMAT_PY)
else:
RUN_DATE_ARG = date
Do I have any way to use this value as a code variable?

You can not use templating outside of operators scope.
You should use Jinja if statements in the operator templated parameter. The following is just a general idea:
submit_spark_job = EmrContainerOperator(
task_id="start_job",
...
name="spark-{{ dag_run.conf["date"] if dag_run.conf["date"] is not None else jinja_utc_now }}",
)
You will need to replace jinja_utc_now with code that retrieve the timestamp probably something like what is shown in this answer.
You can also use:
{% if something %}
code
{% else %}
another code
{% endif %}
From Airflow point of view it takes the parameter and pass it though Jinja engine for templating so the key issue here is just to use the proper Jinja syntax.

Airflow - How to incerase a value stored in XCom

I have fetched a value from a database and stored it in XCom which I would like to increase with 1. I have tried to increment it with following approaches without any luck. Is it possible to increase a value stored in XCom?
'{{ ti.xcom_pull("task_id") + 1}}'
'{{ int(ti.xcom_pull("task_id")) + 1}}'
EDIT
Here is part of my airflow DAG. I have one task that extract data from Hbase:
pull_data_hbase = BashOperator(
task_id='pull_data_hbase',
dag=dag,
bash_command=<My_command_for_exract_data_from_hbase>,
xcom_push=True)
Another task for update the table with increment 1:
data_to_hbase = BashOperator(
task_id='data_to_hbase',
dag=dag,
bash_command=<Command_for_update_table_with_XCom_value>
% ('{{ ti.xcom_pull("pull_data_hbase") +1 }}')
)
when I am using '{{ int(ti.xcom_pull("task_id")) + 1}}' I get the following message:
[2022-01-13 20:39:47,104] {base_task_runner.py:101} INFO - Job
3868282: Subtask print_prev_task ('type:', "{{
ti.xcom_pull('pull_data_hbase') }}") [2022-01-13 20:39:47,105]
{base_task_runner.py:101} INFO - Job 3868282: Subtask print_prev_task
[2022-01-13 20:39:47,103] {cli.py:520} INFO - Running <TaskInstance:
tv_paramount_monthly_report2.0.7-SNAPSHOT.print_prev_task
2021-11-15T00:00:00+00:00 [running]> on host
dl100ven01.ddc.teliasonera.net
[2022-01-13 20:39:47,159]
{models.py:1788} ERROR - 'int' is undefined

You don't have access to Python libraries/functions inside Jinja templates. The TLDR answer is:
"{{ ti.xcom_pull('pull_data_hbase') | int + 1 }}"
You can use certain functions in Jinja templates, these are called "macros" in Jinja. Airflow provides several macros out of the box: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#macros. You can also supply your own macros as shown by #Hitobat.
The other thing you can use in Jinja templates are "filters" (see built-in filters). These can be applied with a pipe (|), as shown above using the int filter.

You can write an actual Python function, and pass this in your DAG as a macro.
Then the function can be callable from airflow templated value.
The name of the key in user macro dict is the name used from template.
eg.
def increment(task_instance, task_id):
return int(task_instance.xcom_pull(task_id)) + 1
with DAG(
dag_id='dag_id',
user_defined_macros={'increment': increment},
) as dag:
pull_data_hbase = BashOperator(
task_id='pull_data_hbase',
dag=dag,
bash_command='echo x+1={{ increment(ti, "task_id") }}',
xcom_push=True,
)

Duplicate entry for 'name of dag' for key in 'dag_id'

I'm running a dag that triggers another dag two times with different payloads using TriggerDagRunOperator.
The first one starts to run, but the second always fails saying:
sqlalchemy.exc.IntegrityError: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'external_dag-2021-11-01 00:00:00.000000' for key 'dag_id'"
Both TriggerDagRunOperator have different execution_dates and "reset_dag_run" set to true:
x = TriggerDagRunOperator(
task_id="x_external_dag",
trigger_dag_id="external_dag",
python_callable= pass_args_for_x,
execution_date="{{ execution_date }}",
reset_dag_run = True
)
y = TriggerDagRunOperator(
task_id="y_external_dag",
trigger_dag_id="external_dag",
python_callable=pass_args_for_y,
execution_date="{{ ds }}",
reset_dag_run = True
)
I have run out of ideas on how to solve this. Any help would be much appreciated.
Regards!

I think the error is related to the fact that boths DagRuns are being triggered with the same execution_date. "{{ ds }}" and "{{ execution_date }}" are different representations of the same value, different types, str and DateTime. Also note that the {{ execution_date }} variable will soon be deprecated.
To solve the error, try not defining execution_date during Tasks definition. Docstrings don't specify it but, the default value is actually None. Taking a look at the execute() method of the TriggerDagRunOperator, you will find that when is not defined, the execution_date of the DAG that is being triggered is set to timezone.utcnow().
Following this path, I think you won't need to set reset_dag_run = True either. I haven't tested it myself, but should be:
x = TriggerDagRunOperator(
task_id="x_external_dag",
trigger_dag_id="external_dag",
python_callable= pass_args_for_x
)
y = TriggerDagRunOperator(
task_id="y_external_dag",
trigger_dag_id="external_dag",
python_callable=pass_args_for_y
)
Let me know if it worked for you.

Use XCOM Value In Operators

I want to use XCOM values as a parameter of my Operator.
Firstly, was executed OracleReadOperator, which read table from db, and return values.
This is value in XCOM:
[{'SOURCE_HOST': 'TEST_HOST'}]
Using this function I want to get value from xcom
def print_xcom(**kwargs):
ti = kwargs['ti']
ti.xcom_pull(task_ids='task1')
Then use values as as parameter:
with DAG(
schedule_interval='#daily',
dagrun_timeout=timedelta(minutes=120),
default_args=args,
template_searchpath=tmpl_search_path,
catchup=False,
dag_id='test'
) as dag:
test_l = OracleLoadOperator(
task_id = "task1",
oracle_conn_id="orcl_conn_id",
object_name='table'
)
test_l
def print_xcom(**kwargs):
ti = kwargs['ti']
ti.xcom_pull(task_ids='task1', value='TARGET_TABLE')
load_from_db = MsSqlToOracleTransfer(
task_id= 'task2',
mssql_conn_id = "{task_instance.xcom_pull(task_ids='task1') }",
oracle_conn_id = 'conn_def_orc',
sql= 'test.sql',
oracle_table = "oracle_table"
tasks.append(load_from_db)
I don't know do I need print_xcom function.
Or I can get value without it, if yes how?
I got this error:
airflow.exceptions.AirflowNotFoundException: The conn_id `{ task_instance.xcom_pull(task_ids='task1') }` isn't defined

To resolve the immediate NameError exception, Jinja expressions are strings so the arg for oracle_table needs to be updated to:
oracle_table = "{{ task_instance.xcom_pull(task_ids='print_xcom', key='task1') }}"
EDIT
(Since the question and problem changed.)
Only template_fields declared for an operator can use Jinja expressions. It looks like MsSqlToOracleTransfer is a custom operator and if you want to use a Jinja template for the mssql_conn_id arg, it needs to be declared as part of template_fields otherwise the literal string is used as the arg value (which is what you're seeing). Also you need the expression in the "{{ ... }}" format as well.
Here is some guidance on Jinja templating with custom operators if you find it helpful.
However, it seems like there is more to this picture than what we have context for. What is task1? Are you simply trying to retrieve a connection ID? What is it exactly you are trying to accomplish accessing XComs in the DAG?

The Airflow tasks has implemented the output attribute that returns an intance of XComArs. For example:
def push_xcom(ti):
return {"key": "value"}
def pull_xcom(input):
print(f'XCom: {input}')
with DAG(...) as dag:
start = PythonOperator(task_id='dp_start', python_callable=push_xcom)
end = PythonOperator(task_id='dp_start', python_callable=pull_xcom,
op_kwargs={'input': start.output})
start >> end
Maybe you could use test_l.output in load_from_db.mssql_conn_id, But I think in the case of whatever_conn_id parameters, the value should be the ID of an Airflow connection.

Airflow: pass {{ ds }} as param to PostgresOperator

i would like to use execution date as parameter to my sql file:
i tried
dt = '{{ ds }}'
s3_to_redshift = PostgresOperator(
task_id='s3_to_redshift',
postgres_conn_id='redshift',
sql='s3_to_redshift.sql',
params={'file': dt},
dag=dag
)
but it doesn't work.

dt = '{{ ds }}'
Doesn't work because Jinja (the templating engine used within airflow) does not process the entire Dag definition file.
For each Operator there are fields which Jinja will process, which are part of the definition of the operator itself.
In this case, you can make the params field (which is actually called parameters, make sure to change this) templated if you extend the PostgresOperator like this:
class MyPostgresOperator(PostgresOperator):
template_fields = ('sql','parameters')
Now you should be able to do:
s3_to_redshift = MyPostgresOperator(
task_id='s3_to_redshift',
postgres_conn_id='redshift',
sql='s3_to_redshift.sql',
parameters={'file': '{{ ds }}'},
dag=dag
)

PostgresOperator / JDBCOperator inherit from BaseOperator.
One of the input parameters of BaseOperator is params:
self.params = params or {} # Available in templates!
So, you should be able to use it without creating a new class:
(even though params is not included into template_fields)
t1 = JdbcOperator(
task_id='copy',
sql='copy.sql',
jdbc_conn_id='connection_name',
params={'schema_name':'public'},
dag=dag
)
SQL statement (copy.sql) might look like:
copy {{ params.schema_name }}.table_name
from 's3://.../table_name.csv'
iam_role 'arn:aws:iam::<acc_num>:role/<role_name>'
csv
IGNOREHEADER 1
Note:
copy.sql resides at the same location where the DAG is located.
OR
you can define "template_searchpath" variable in "default_args" and specify absolute path to the folder where template file resides.
For example: 'template_searchpath': '/home/user/airflow/templates/'