I am currently working with the DatabricksSubmitRunOperator in Airflow. When I tried to integrate the jinja template into the operator, by using it in the "json"-parameter of the operator, I was facing an error. The problem is, that jinja returns a string but Databricks operator needs a dict type.
I already looked up the source code of the operator. "json" seems to be a template_field, which should be fine.
Is there a way to make jinja return a dict type instead of a string in this case? Or maybe another workaround?
Thanks in advance!
There is the new feature in Airflow 2.1/0 that can help with that https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects - it uses JINJA capability of "safe evaluation" of the string and returning and object that the string represents directly in your code.
Related
I’m running my DAG with configuration JSON, which includes parameters which includes -, e.g. market-name
When I’m trying to read it using the following Jinja template:
path_prefix = f"market={{{{ params.market-name }}}}/configuration"
I’m getting the following error:
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'market'
It seems to me that Jinja doesn’t identify the full param name market-name but get the first part before the - which is market .
My questions:
Does Jinja supports having - in the middle of a param (e.g. market-name) or should I avoid it from the beginning and use market_name instead?
If Jinja doesn’t support having a - in the middle of a param
Should I escape the market-name ?
How should I escape it?
Should I use {{ params.market-name }} instead of {{{{ params.market-name }}}}?
This is not Airflow bug. I reported it in this issue and this is a problem coming from Jinja/Python.
I could have not find any official documentation in Jinja that explains it but several reports mention that Jinja uses python interpreter's identifiers (see this answer and this answer). It might be best to report this issue to Jinja add documentation about it.
You can just avoid using -.
I found a very strange behavior in spring rest.
Having an endpoint like the following
#GetMapping("/foo")
public String foo() {
return "bar";
}
returns the value bar. Sounds correct, but this is not a valid json, the effective result should be "bar" (note the ""). One might argue that spring expects that if a method returns a string, you have already manually serialized the object, but if all other objects are serialized by spring, then i would expect to have a special way to tell that its already serialized but the default way should be to serialize the value.
Maybe i'm missing something here, that's the reason why i didn't created a ticket in the spring issue tracker jet.
Unregistering the StringHttpMessageConverter as described in this answer should do the trick: https://stackoverflow.com/a/37906098/505621
While not really an answer, there are 2 possible workarounds:
Encode it yourself
Just add the required quotation mark explicitly to the response string \".
This could for example be done by using JSONObject.quote(yourString).
Wrapping
Wrap your string into a simple object or custom class. Then Jackson knows what to do with it.
Collections.singletonMap("value", yourString);
In my dbt macro, I am passing a dictionary argument which looks like
random_mapping={"<string1>":"<string2>","<string3>":"<string4>"}
Now, when I check if a given string is a key in my random_mapping, it returns False, I used log and checked my dbt.log file and it looks right.
Below is how I am checking container-ship
dbt_utils.string_literal(relation) in random_mapping
I resolved it on my own. But, I hope this helps someone else.
The environment I was working on was jinja in dbt
dbt_utils.string_literal(relation) returns a string with quotes like '<some_string>', whereas we wanted <some_string>.
This can be resolved simply by using strip function on a string, so it would look like dbt_utils.string_literal(relation).strip("\'") for my case.
Note: \' is used because ' escapes when alone.
I trying to format my output from a lambda function into JSON. The lambda function queries my Amazon Aurora RDS instance and returns an array of rows in the following format:
[[name,age,town,postcode]]
which gives the an example output:
[["James", 23, "Maidenhead","sl72qw"]]
I understand that mapping templates are designed to translate one format to another but I don't understand how I can take the output above and map in to a JSON format using these mapping templates.
I have checked the documentation and it only covers converting one JSON to another.
Without seeing the code you're specifically using, it's difficult to give you a definitely correct answer, but I suspect what you're after is returning the data from python as a dictionary then converting that to JSON.
It looks like this thread contains the relevant details on how to do that.
More specifically, using the DictCursor
cursor = connection.cursor(pymysql.cursors.DictCursor)
I'm trying to pass a value via JSON that I am having trouble accessing. We have a data structure (that was obviously not built by me otherwise I would likely understand it) that looks something like this when sent to the browser:
{Foo(Bar(List(Baz(List(G3),w))),G3,None)}
This is sent via a JSON write method, but the originating Scala line looks like:
val hint = Some(s"{$question}") where $question is of type Foo.
I've tried using dot notation to access the list items in ways that I thought would work:
val hint = Some(s"{$question.Bar.Baz})"
val hint = Some(s"{$question.Bar(0).Baz(0)"})
It's the deepest G3 I wanted to strip out and send, but instead the JSON object comes through looking like:
{Foo(Bar(List(Baz(List(G3),w))),G3,None)}.Bar.Baz or
{Foo(Bar(List(Baz(List(G3),w))),G3,None)}.Bar(0).Baz(0)
I must be fundamentally missing something about the data structures involved here.
I think you're just using the wrong syntax. The $ needs to come before the {} and the {} is necessary for any expression more complicated than just a variable name:
s"${question.bar(0).baz(0)}"