Insert Environment Variable using Jinja in SaltStack - jinja2

I am trying to read a JSON file inside a folder. using import_json.
Here is my code
{% set instance_id = grains['INSTANCE_ID'] %}
INSTANCE_ID Env Var:
environ.setenv:
- name: INSTANCE_ID
- value: {{ grains['INSTANCE_ID'] }}
- update_minion: True
{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
But I am getting this error
Data failed to compile:
Rendering SLS 'base:cloud.steps.conf' failed: Jinja error: /tmp/$INSTANCE_ID/conf.json.
Although when I insert the INSTANCE_ID manually it works as expected.
What I want is to be able to insert either $INSTANCE_ID or directly the grain value {{ grains['INSTANCE_ID'] }}
Can someone please help me with this?
Thanks.

{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
I imagine you are trying to evaluate the variable $INSTANCE_ID in the above statement. Jinja template evaluates the variables in expression statements.
In this case, the variable is set in the first line, using set
{% set instance_id = grains['INSTANCE_ID'] %}
So, you can use it in expression along with string appends, like
{% import_json "/tmp/" ~ instance_id ~ "/conf.json" as config_properties %}
The above statement should resolve your error.
Also, I would suggest using a variable to evaluate the value of the string expression above, like
{% set conf_json_path = "/tmp/" ~ instance_id ~ "/conf.json" %}
and use it like this
{% import_json conf_json_path as config_properties %}
Hope this help!
In case, you wish to use grains dictionary directly, you can use the value like so
{% set conf_json_path = "/tmp/" ~ grains['INSTANCE_ID'] ~ "/conf.json" %}

Related

In Jinja how to access Macro parameters in a set block that runs SQL?

Hi all I'm currently writing Jinja macro that runs a sql query based on some parameters passed to it.
I'm having some issue with the macro definition itself. I cannot find a way to use the macros paramters within the set block. It interprets the value as literally 'col1' not the value stored in it.
How can I run a SQL query within the macro that uses my parameters stored value?
How I run the macro:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ select_test('col1') }} as output
FROM
DRIVER
The macro definition is shown below:
{% macro select_test(val1) %}
{% set query_to_run%}
select
concat({{val1}},'banana')
{% endset %}
{% set results = run_query(query_to_run) %}
{% if execute %}
{# Return the first column #}
{% set results_list = results.columns[0].values() %}
{% else %}
{% set results_list = [] %}
{% endif %}
{{ return(results_list) }}
{% endmacro %}
I currently get the following output:
col1banana
My expected output:
applebanana
note: edited for clarity
You're very close -- I think you're confused by the nested nature of this query.
I suspect when you created this example, you didn't include a set of quotes that is in your original query. This query will return col1banana if col1 is double-quoted (either in the macro or the model), and will return a Database Error otherwise:
select
concat('{{ val1 }}','banana')
-- {{ select_test('col1') }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test("'col1'") }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test('col1') }} raises a Database Error: column "col1" does not exist
The last example is always going to be a database error, since it compiles to this:
select
concat(col1, 'banana')
and there is no from clause in your query_to_run, so col1 can't exist.
Most macros don't use run_query and are just shortcuts for snippets of sql. Those then get templated back into the model query and executed against your database when you build your model.
A macro to concatenate a string onto a column name is as simple as:
{% macro cat_banana(column_name) %}
concat({{ column_name }}, 'banana')
{% endmacro %}
And then you call it from your model:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ cat_banana('col1') }} as output
FROM
DRIVER
Then after dbt run -s my_model, if you select * from my_model you'll get applebanana.

Passing column divided by value as parameter in macro dbt jinja

I'd like to pass a column divided by a value as a parameter in a jinja macro.
I'm using the macro in a dbt model like this {{ pmt('rate'/1200, 'nper', 'pv', 'fv') }}
However, this gives the error message
"Encountered an error: unsupported operand type(s) for /: 'str' and 'int'"
Most likely you have to treat the whole argument as a string literal (quote the whole thing):
{{ pmt('rate/1200', 'nper', 'pv', 'fv') }}
The reason this works is because it is likely that the macro templates this string into SQL code, e.g.,
{% macro pmt(arg1, arg2, arg3, arg4) %}
...
select {{ arg1 }}
...
{% endmacro %}
In this toy example, {{ arg1 }} will take on the value {{ 'rate/1200' }}, which enters the template (unquoted) as
...
select rate/1200
...
which is valid sql (if you have a field called rate).
It's possible this won't work with all macros, though! In dbt, since the macros are typically templating SQL code, you usually want to pass in arguments that contain field or table references as string literals. However, the argument to the macro could stay inside the jinja context, in which case, you'll need to keep the argument unquoted, or modify a variable before it's passed into the jinja macro. As another toy example:
{% macro print_value(val) %}
{{ log(val, info=True) }}
{% endmacro %}
-- if val is a string literal:
{{ print_value('rate/1200') }}
-- $ rate/1200
-- if val is unquoted:
{% set rate = 2400 %}
{{ print_value(rate/1200) }}
-- $ 2

Unable to use Jinja variable inside adapter.get_relation in DBT

I am trying to use a Jinja variable inside my adapter.get_relation but I am unable to do so :confused: The code
{% set the_var = 'amazon_full_orders_denormalized_' ~ company_uuid %}
{{ log(the_var, info=True) }}
{%- set source_relation = adapter.get_relation(
database='282615',
schema='airbyte_mumbai',
identifier= the_var ) -%}
The command I am using
dbt compile --vars '{"company_uuid":"0703afd3_496b_4ed5_8e0c_594b71f4718b","dataset":"airbyte_mumbai"}' --models tag:copy_reports
The variable the_var is getting set to amazon_full_orders_denormalized_
Found 6 models, 0 tests, 0 snapshots, 0 analyses, 165 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
17:11:08 | Concurrency: 3 threads (target='dev')
17:11:08 |
amazon_full_orders_denormalized_
Table does not exist
17:11:08 | Done.
Whereas I am expecting it to be set to amazon_full_orders_denormalized_0703afd3_496b_4ed5_8e0c_594b71f4718b
Using variables in dbt can be hard sometimes! I think the commenter #Kay is on the right track here in that you have three variables happening here: the_var, company_uuid, and dataset. It looks as if you'd like the a table name that is the concatenation of the_var and company_uuid, which you can do using jinja's concat operators, ~, like this:
-- just for testing
{{ log(the_var, info=True) }}
{{ log(var('company_uuid'), info=True) }}
{{ log(the_var ~ var('company_uuid'), info=True) }}
-- see change to `identifier`
{%- set source_relation = adapter.get_relation(
database='282615',
schema='airbyte_mumbai',
identifier= the_var ~ var('company_uuid') ) -%}

saltstack state file: how to access a list element in a salt grain via Jinja

I try to evaluate the value of a grains list in jinja but do not know how. The list entry I am looking for is the minor osversion in
grain:osrelease_info
salt-call -g |grep -C2 osrelease_info
osrelease:
15.99
osrelease_info:
- 15
- 99
In a state-file I would like to evaluate the minior osrelease value in a jinja expression like this one:
{% if grains['osmajorrelease'] == '15' and grains['osrelease_info'][1] >= 99 %}
...
{% endif %}
However the syntax I tried above to access index:1 of the osrelease_info list does not report an error but doesn't work either.
So how can I access the list entry containing value "99" in Jinja?
I think I found the answer myself alltough it is a little strange. The problem was not the expression
{% ... and grains['osrelease_info'][1] >= 99 %}
but instead the string comparison before:
{% if grains['osmajorrelease'] == '15' ... %}
All I changed was to convert the string '15' to an int. So the following expression does what I originally wanted:
{% if grains['osmajorrelease'] == 15 and grains['osrelease_info'][1] >= 99 %}
...
{% endif %}
This is weird since I use exactly the same comparision (grains['osmajorrelease'] == '15') in other state files with success.

Get value using mine.get from a ordered dictionary in state.sls

I have a mine defined in pillar to get the ip of a salt minion. When I use the mine in a salt state I get a dictionary. Is there a way to filter out and get the first item from it.
Here is my mine which I have defined in a pillar:
ip_add:
- mine_function: grains.get
- ipv4
Here is my state file:
{% set id = 'aws-vm1' %}
{% set addrs = salt['mine.get'](id, 'ip_add') %}
{% for key in salt['mine.get'](id, 'ip_add') %}
Test:
file.append:
- name: C:\test
- text: {{addrs}}
The output i get is :
OrderedDict([('aws-vm1', ['10.93.143.235', '127.0.0.1'])])
I want to get the first ip so that I can share it between minions
I was able to find a work around. I get the list of IPs from the mine
aws-vm1:
----------
aws-vm1:
- 10.93.143.235
- 127.0.0.1
and grab the first value by {{ip[0]}} which is the private IP I want.
{% set id = 'aws-vm1' %}
# Returns a dict OrderedDict
{% set ip_list = salt['mine.get'](id, 'ip_add') %}
{% set ip= ip_list.values() | first %}
Test:
file.append:
- name: C:\test
- text: {{ip[0]}}