I’m running my DAG with configuration JSON, which includes parameters which includes -, e.g. market-name
When I’m trying to read it using the following Jinja template:
path_prefix = f"market={{{{ params.market-name }}}}/configuration"
I’m getting the following error:
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'market'
It seems to me that Jinja doesn’t identify the full param name market-name but get the first part before the - which is market .
My questions:
Does Jinja supports having - in the middle of a param (e.g. market-name) or should I avoid it from the beginning and use market_name instead?
If Jinja doesn’t support having a - in the middle of a param
Should I escape the market-name ?
How should I escape it?
Should I use {{ params.market-name }} instead of {{{{ params.market-name }}}}?
This is not Airflow bug. I reported it in this issue and this is a problem coming from Jinja/Python.
I could have not find any official documentation in Jinja that explains it but several reports mention that Jinja uses python interpreter's identifiers (see this answer and this answer). It might be best to report this issue to Jinja add documentation about it.
You can just avoid using -.
Related
I'm writing a dag model and my goal is to make it as generic as possible. For this, I use the jinja models coupled with the airflow connection to store the customer credentials. it works fine.
In order to go further, during my tasks, I want to be able to retrieve data in an airflow connection with the jinja models which I will then use to format another jinja model that I then need to retrieve.
For instance:
--context_param MYSQL_MONITORING_FLOW_HOST={{ conn.{{ conn.NEWCLIENTTEST_DATASOURCE.extra_dejson.MYSQL_BDD_SERVER_ID }}.host }}
{{ conn.NEWCLIENTTEST_DATASOURCE.extra_dejson.MYSQL_BDD_SERVER_ID }} tested alone returns the correct value (MYSQL_GENERIC_ID) and then {{ conn.MYSQL_GENERIC_ID.host }} also returns the correct value.
I would like it to do a double interpretation of the jinja template, is that possible?
Thank you very much for your help
cdt
Matthew
While upgrading Django I noticed that the html escape function used to translate an apostrophe into ' whereas it now translates it into '. I understand the old value is the HTML code and the new value is the hex code. The change seems to have come about because Django used to do the translation itself but now hands it off to python's html module.
My question is, are these functionally equivalent? Are there any case where I would need to worry about this change?
Also, in case I am not asking the right questions, can anyone direct me to what I should read to understand why these 2 types of codes exist and how they are used?
I am currently working with the DatabricksSubmitRunOperator in Airflow. When I tried to integrate the jinja template into the operator, by using it in the "json"-parameter of the operator, I was facing an error. The problem is, that jinja returns a string but Databricks operator needs a dict type.
I already looked up the source code of the operator. "json" seems to be a template_field, which should be fine.
Is there a way to make jinja return a dict type instead of a string in this case? Or maybe another workaround?
Thanks in advance!
There is the new feature in Airflow 2.1/0 that can help with that https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects - it uses JINJA capability of "safe evaluation" of the string and returning and object that the string represents directly in your code.
Question:
Can I use jinja2 string syntax into task_id string or we are limited to f-string?
Example:
Let's say I use a BashOperator and inside I have assigned params. I can use successfully these params into my bash_command, however once I use them into the task_id string, I get the following error:
airflow.exceptions.AirflowException: The key (my_task_id_string{{ params.paramname1}}{{ params.paramname2}}) has to be made of alphanumeric characters, dashes, dots and underscores
Instead, in order to mitigate this, I use an f-string instead like so: f"my_task_id_string{paramname1}{paramname2}"
Not all the parameters in the BashOperator is templated, the same applies for all the operators available in Airflow.
For the BashOperator the fields that Jinja will template on are bash_command and env.
For further reference, you can always refer to any operators source doc-strings, which will indicate if the field is templatable.
For BashOperator, you can refer here https://airflow.apache.org/docs/stable/_modules/airflow/operators/bash_operator.html
I have a $scope variable scopeVar which contains a JSON Object. The JSON object has some complicated key names such as "onStatus[1]".
Is it possible to resolve such key names inside the view template so that I can use them like {{scopeVar.onStatus[1]}} or ng-bind="scopeVar.onStatus[1]" ?.
PS- I assume that using JSON keys in such a fashion is possible after reading this answer.However I am still skeptical regarding using symbols like '[' etc in key names as they may also be used to represent array elements.
If onStatus[1] is actually a property name and not the second element of the onStatus array you should use bracket notation to access the property:
{{ scopeVar['onStatus[1]'] }}
or as the expression in ngBind:
ng-bind="scopeVar['onStatus[1]']"
Use it like this in the view it works for me.
{{ scopeVar['onStatus[1]'] }}
Basically in the interpolation anything you put is treated as plain JS code so anything that works in the console of your browser will also work in the between the curly braces.