I got a pillar item which looks something like below:
"fileabspath": [
"Z:\\customer\region\chicago"
]
And on my sls file, I am trying to access the "filepath" pillar value with the below:
{% set fileabspath = salt['pillar.get']('fileabspath', None) %}
customer:
region.filepath:
- name: '{{fileabspath}}'
As soon as job hits above last line, I get below error.
failed: while parsing a block mapping\n in \"<unicode string>\", line 10, column 1\ndid not find expected key\n in \"<unicode string>\"
Any advice how to fix the issue ? Thank you.
You need to escape the backslashes if you're using that data format
"fileabspath": [
"Z:\\\\customer\\region\\chicago"
]
Related
When I try to use "Get Value From Json" on a Json with specific JsonPath , I always have KeyError
even web simple example doesn't work for me...
When I try that code :
Library JSONLibrary
*** Test Cases ***
Example
${tmp} = Convert String To JSON {"foo": {"bar": [1,2,3]}}
Log to console tmp: ${tmp}
${values_test}= Get Value From Json ${tmp} $.foo.bar
Log to console values_test: ${values_test}
I always have this kind of Errors and log :
tmp: {'foo': {'bar': [1, 2, 3]}}
...
Resolving variable '${tmp['$.foo.bar']}' failed: KeyError: '$.foo.bar'
Can somebody Help me please ?
it is really basic example by the way and community always says that it works like that in comments..
I'm using Airflow 2.2.2 with the latest providers installed as appropriate.
I'm trying to use the Azure and MySQL hooks and have created custom operators with templates defined for what variables can be templated.
When I do so, I get an error saying that conn or var cannot be found
e.g. my passed parameter is
{{ conn.<variable_name> }}
or
{{ var.json.value.<variable_name> }}
I believe this should be possible in > v2.0 but not working for me, any ideas why?
EDIT: Below are snippets of code with some sensitive information removed, let me know if anything else is needed?
DAG error -
Broken DAG: [/home/dags/dag.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/dags/dag.py", line 52, in <module>
wasb_conn_id = {{ conn.wasb }},
NameError: name 'conn' is not defined
task in dag.py
t1 = WasbLogBlobsToCSVOperator(
task_id='task_xyz',
wasb_conn_id = {{ conn.wasb }},
Custom Operator using an extended version of the Microsoft Azure wasb hook , used by dag.py -
class WasbLogBlobsToCSVOperator(BaseOperator):
template_fields = (
'wasb_conn_id',
)
def __init__(
self,
*,
wasb_conn_id: str = 'wasb',
**kwargs,
) -> None:
super().__init__(**kwargs)
self.wasb_conn_id = wasb_conn_id
self.hook = ExtendedWasbHook(wasb_conn_id=self.wasb_conn_id)
There looks to be a few things going on here that should help.
Jinja templates are string expressions. Try wrapping your wasb_conn_id arg in quotes.
wasb_conn_id = "{{ conn.wasb }}",
Templated fields are not rendered until the task runs meaning the Jinja expression won't be evaluated until an operator's execute() method is called. This is why you are seeing an exception from your comment below. The literal string "{{ conn.wasb }}" is being evaluated as the conn_id. If you want to use a template field in the custom operator, you need to move that logic to be in the scope of the execute() method.
Why do you need to use a Jinja expression here? Since the format of accessing the Connection object via Jinja is {{ conn.<my_conn_id> }}, you could just use the value "wasb" directly.
I have a json file filled with a list of data where each element has one field called url.
[
{ ...,
...,
"url": "us.test.com"
},
...
]
In a different file I have a list of mappings that I need to replace the affected url fields with, formatted like this:
us.test.com test.com
hello.com/se hello.com
...
So the end result should be:
[
{ ...,
...,
"url": "test.com"
},
...
]
Is there a way to do this in Vim or do I need to do it programmatically?
Well, I'd do this programmatically in Vim ;-) As you'll see it's quite similar to Python and many other scripting languages.
Let's suppose we have json file open. Then
:let foo = json_decode(join(getline(1, '$')))
will load json into VimScript variable. So :echo foo will show [{'url': 'us.test.com'}, {'url': 'hello.com/se'}].
Now let's switch to a "mapping" file. We're going to split all lines and make a Dictionary like that:
:let bar = {}
:for line in getline(1, '$') | let field = split(line) | let bar[field[0]] = field[1] | endfor
Now :echo bar shows {'hello.com/se': 'hello.com', 'us.test.com': 'test.com'} as expected.
To perform a substitution we do simply:
:for field in foo | let field.url = bar->get(field.url, field.url) | endfor
And now foo contains [{'url': 'test.com'}, {'url': 'hello.com'}] which is what we want. The remaining step is to write the new value into a buffer with
:put =json_encode(foo)
You could…
turn those lines in your mappings file (/tmp/mappings for illustration purpose):
us.test.com test.com
hello.com/se hello.com
...
into:
g/"url"/s#us.test.com#test.com#g
g/"url"/s#hello.com/se#hello.com#g
...
with:
:%normal Ig/"url"/s#
:%s/ /#
The idea is to turn the file into a script that will perform all those substitutions on all lines matching "url".
If you are confident that those strings are only in "url" lines, you can just do:
:%normal I%s#
:%s/ /#
to obtain:
%s#us.test.com#test.com#g
%s#hello.com/se#hello.com#g
...
write the file:
:w
and source it from your JSON file:
:source /tmp/mappings
See :help :g, :help :s, :help :normal, :help :range, :help :source, and :help pattern-delimiter.
I have different results in the bash command line and in a SLS file for pillar.get().
In the command line I receive the expected results.
salt-master:/srv/pillar # salt salt-minion2 pillar.get mqobjdetails
salt-minion2:
----------
QM200#QLOCAL#LQ2:
----------
DEFPSIST:
NO
GET:
ENABLED
MAXDEPTH:
5000
MAXMSGL:
4194304
MONQ:
QMGR
QM200#QMGR#QM200:
----------
DEADQ:
MAXMSGL:
4194304
MONCHL:
OFF
MONQ:
OFF
and with filter
salt-master:/srv/pillar # salt salt-minion2 pillar.get mqobjdetails:QM200#QLOCAL#LQ2
salt-minion2:
----------
DEFPSIST:
NO
GET:
ENABLED
MAXDEPTH:
5000
MAXMSGL:
4194304
MONQ:
QMGR
When I try the same thing in a state file with the following jinja loop statement
{% for objkey, parameters in pillar.get('mqobjdetails', { } ).items() %}
the loop works as expected.
When I try it with the additional filter
{% for name, value in pillar.get("mqobjdetails:QM200#QMGR#LQ2", { } ).items() %}
the loop is never run through. So it seems the I get back an empty set in this case.
What do I wrong? Can somebody help me, please?
I just found the answer in the documentation.
The pillar.get Function
There is a note, which compares pillar.get() vs salt'pillar.get'.
Well, when I use something like
{% for name, value in salt['pillar.get']('mqobjdetails:' ~ foo, { } ).items() %}
I get the expected results.
Still I do not like the idea, that the same function has different behaviours.
Is there a method for jinja2 to raise an exception when we pass a variable that is not present in the template?
PS: This is different(or opposite) from raising an exception when a variable is present in the template but it is not passed. For this I use "undefined=StrictUndefined"
When you load your jinja2.Environment, set the 'undefined' parameter to 'jinja2.StrictUndefined', e.g.:
env = jinja2.Environment(loader=<someloader>, undefined=jinja2.StrictUndefined)
You can catch and examine the render exception to see what was missing
EDIT It would help if I read your full question. :)
Maybe this could help you
https://jinja.palletsprojects.com/en/2.11.x/api/#the-meta-api
>>> from jinja2 import Environment, meta
>>> env = Environment()
>>> ast = env.parse('{% set foo = 42 %}{{ bar + foo }}')
>>> meta.find_undeclared_variables(ast)
set(['bar'])
You can also do that:
from jinja2 import Template, StrictUndefined
Template('name: {{ name }} , city: {{ city }}',undefined=StrictUndefined).render(**{"name":"foo","city":"bar"})