How to get python dictionary or list from Jinja templated string in Airflow? - jinja2

Question

Let's assume I've an operator which needs a python list (or dict) as an argument for it's property
doExampleTask = ExampleOperator(
task_id = "doExampleTask",
property_needs_list = [
("a", "x"),
("b", "y")
],
property_needs_dict = {
"dynamic_field_1": "dynamic_value",
# ...
"dynamic_field_N": "dynamic_value",
},
)
The problem is that I can't define the python data structure of the list (how many list elements is needed) or the dict (which fields were generated) in the creation time of the DAG.
I could only get this structure dynamically by executing a previous task or macro.
the task could write the data structure with dynamic fields into the XCOM
the macro could return a data structure
But in both of the above cases there is no way to convert the dynamic data structure (which is returned by XCOM or custom macro) to python data structure and use it as a property of the operator.
This will not return list or dict:
doExampleTask = ExampleOperator(
task_id = "doExampleTask",
property_needs_list = '{{ generate_list() }}',
property_needs_dict = '{{ generate_dict() }}',
)
This will also not return dict or list:
doExampleTask = ExampleOperator(
task_id = "doExampleTask",
property_needs_list = '{{ ti.xcom_pull(task_ids="PreviousTask", key="list_structure") }}',
property_needs_dict = '{{ ti.xcom_pull(task_ids="PreviousTask", key="dict_structure") }}',
)
If I use something like eval() function, it will not be able to evaluate the string argument in execution time of the Task. It will try to evaluate it in creation time of the DAG, but the values will obviously not be there.
doExampleTask = ExampleOperator(
task_id = "doExampleTask",
property_needs_list = eval('{{ ti.xcom_pull(task_ids="PreviousTask", key="list_structure") }}'),
property_needs_dict = eval('{{ ti.xcom_pull(task_ids="PreviousTask", key="dict_structure") }}'),
)
or
doExampleTask = ExampleOperator(
task_id = "doExampleTask",
property_needs_list = eval('{{ generate_list() }}'),
property_needs_dict = eval('{{ generate_dict() }}'),
)
How can I workaround this problem?
I'm mostly interested in Airflow 1.x, but I'm open to Aitflow 2.x solution.
Thank you!

Answer 1

In Airflow 1, Jinja expressions are always evaluated as strings. You'll have to either subclass the operator or build in logic to your custom operator to translate the stringified list/dict arg as necessary.
However, in Airflow 2.1, there was an option added to render templates as native Python types. You can set render_templates_as_native_obj=True at the DAG level and lists will render as a true list, dict as a true dict, etc. Check out the docs here.