We define external redshift spectrum table using an yml file. Like the one below.
- name: visitor
description: 'visitor data'
external:
location: "s3://{{ env_var('ENV') }}-temp-raw-s3/{{ s3_prefix({{ env_var('ENV') }}) }}/Visitor"
file_format: parquet
partitions:
- name: year
data_type: VARCHAR(10)
vals: # macro w/ keyword args to generate list of values
macro: dbt.dates_in_range
args:
start_date_str: '{{ var("START_DT") }}'
end_date_str: '{{ var("START_DT") }}'
in_fmt: '%Y-%m-%d'
out_fmt: '%Y'
I would like to change the prefix of the s3 bucket based on the environment variable.
Here is the macro I have tried to change the prefix. I am calling that macro in in the yml file above in line 4 (location).
{% macro s3_prefix(env_var) %}
{% if env_var == "prod" %}
prefix_string = "prefix_prod"
{% else %}
prefix_string = "prefix_non_prod"
{% endif %}
{{ return(prefix_string) }}
{% endmacro %}
I am getting some compilation errors. Is there an easier way to achieve this? Any help would be really appreciated.
Related
Thanks in advance to anyone looking at this. I'm new to jinja and am trying to test something with the following 3 simple files:
pipe.j2:
{% include 'resources.yml' %}
resources.yml:
resources:
- name: resource1
type: aThing
- name: resource2
type: anotherThing
test.sh
j2 pipe.j2 > pipeline.yml
The test.sh script runs fine, but i'm trying to exclude either of the resources in the resources.yml file. I've tried the following in the j2 file getting different errors:
example1:
{% include 'resources.yml' | reject('equalto','resource1') %}
TypeError: generator is not subscriptable
example2:
{% include 'resources.yml' %}
{% set r_list = 'resources' %}
{% for rsc in r_list | rejectattr('name', 'equalto', 'resource1') %}
{% endfor %}
UndefinedError: 'str.object' has no attribute 'name'
I checked online and haven't found any examples of what I'm trying to accomplish and am not sure if it's possible or if I'm doing it completely wrong. I'm trying to get a file that only contains non-rejected items as a final result.
Hoping to get in pipeline.yml file:
resources:
- name: resource2
type: anotherThing
I have a template folder containing a file.j2 , e.g
{% set summary %}
{% if summary %}
{{ summary }}
% endif %}
{% endset %}
I would like to use this summary variable in my yaml file but I can't seem to be able to find the information on how to reference the j2 file in the yaml file
Can I do this?
template: file.j2
summary: '{{ summary }}'
Any suggestions are greatly appreciated!
I have a state using file.managed, which generates a config file via a jinja for loop from a key in pillar.
My pillar looks like this:
configuration:
server01:
key1: value1
key2: value2
server02:
key03: value03
key04: value04
and the managed file:
{% set kv = pillar['configuration']['server01'] %}
{% for key, value in kv.iteritems() %}
{{ key }}:{ value }};
{% endfor %}
The way I differentiate between different servers right now in my state file is
config:
file.managed:
- name: /etc/config.conf
- source: salt://files/{{ grains['id'] }}.conf.jinja
- template: jinja
but this is less than ideal, since I have to create an almost identical file for every server.
Is there a way to dynamically replace server01 with the ID of the actual server, something like
{% set kv = pillar['configuration']['{{ grains[id''] }}'] %}
The goal is to generally limit the necessary changes only to the corresponding pillar file, when adding a new server, so other suggestions are also welcome too.
i think you should use pillar info in your state file.
your state file like bellow :
{% if grains['id'] in pillar['configuration'] %}
{% set nodeinfo = pillar['configuration'][grains['id']] %}
config:
file.managed:
- name: /etc/config.conf
- source: salt://conf.jinja
- template: jinja
- defaults :
nodeinfo: {{nodeinfo}}
{% endif %}
then, conf.jinja:
{% for key, value in nodeinfo.iteritems() -%}
{{ key }}:{{ value }};
{% endfor -%}
i hope that will solve your problem, thanks.
Trying to use Google Deployment Manager with YAML and Jinja with a multi-line variables, such as:
startup_script_passed_as_variable: |
line 1
line 2
line 3
And later:
{% if 'startup_script_passed_as_variable' in properties %}
- key: startup-script
value: {{properties['startup_script_passed_as_variable'] }}
{% endif %}
Gives MANIFEST_EXPANSION_USER_ERROR:
ERROR: (gcloud.deployment-manager.deployments.create) Error in
Operation operation-1432566282260-52e8eed22aa20-e6892512-baf7134:
MANIFEST_EXPANSION_USER_ERROR
Manifest expansion encountered the following errors: while scanning a simple key in "" could not found expected ':' in ""
Tried (and failed):
{% if 'startup_script' in properties %}
- key: startup-script
value: {{ startup_script_passed_as_variable }}
{% endif %}
also
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{{ startup_script_passed_as_variable }}
{% endif %}
and
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{{ startup_script_passed_as_variable|indent(12) }}
{% endif %}
The problem is the combination of YAML and Jinja. Jinja escapes the variable but fails to indent it as YAML would require when being passed as a variable.
Related: https://github.com/saltstack/salt/issues/5480
Solution: Pass the multi-line variable as an array
startup_script_passed_as_variable:
- "line 1"
- "line 2"
- "line 3"
The quoting is important if your value starts with # (which startup script on GCE does, ie #!/bin/bash) since it will be treated as a comment otherwise.
{% if 'startup_script' in properties %}
- key: startup-script
value:
{% for line in properties['startup_script'] %}
{{line}}
{% endfor %}
{% endif %}
Putting it here since there aren't much Q&A material for Google Deployment manager.
There is no clean way to do this in Jinja. As you yourself have pointed out, because YAML is a whitespace-sensitive language, it is difficult to effectively template around.
One possible hack is to split the string property into a list and then iterate over the list.
For example, providing the property:
startup-script: |
#!/bin/bash
python -m SimpleHTTPServer 8080
you can use it in your Jinja template:
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{% for line in properties['startup-script'].split('\n') %}
{{ line }}
{% endfor %}
Here is also a full working example of this.
This method will work, but generally cases like this are when people start considering using a python template. Because you are working with an object model in python, you do not have to deal with indentation problems. For example:
'metadata': {
'items': [{
'key': 'startup-script',
'value': context.properties['startup_script']
}]
}
An example of the python template can be found in the Metadata From File example.
Starting off with Ansible and I am trying to use ReST API to interact with an external application.Maybe I am missing something simple here.
I am trying to compare every host in my inventory file with the POD name specified in the variable file used by the role that invokes the jinja2 template.
My inventory file looks like this:
[all]
'POD-9'
'POD-10'
Variable file :
pods:
params:
- name: POD-9
- name: POD-10
{% for pod in pods.params %}
{% if '{{ inventory_hostname }}' == '{{ pod.name }}' %}
<generate JSON template here>
{% endif %}
{% endfor %}
The if statement however does not take effect. I want the template to be generated only in the inventory_hostname is equal to the pod name in the variable file
The current JSON file includes both :
{
"pod": {
"name": "POD-9"
}
"pod": {
"name": "POD-10"
}
}
In Jinja2 the double curly braces are used as a print statement. If you access variables inside tags don’t put the braces around them
{% for pod in pods.params %}
{% if inventory_hostname == pod.name %}
<generate JSON template here>
{% endif %}
{% endfor %}
Found the problem :
{% if pod.name == inventory_hostname %}