Thanks in advance to anyone looking at this. I'm new to jinja and am trying to test something with the following 3 simple files:
pipe.j2:
{% include 'resources.yml' %}
resources.yml:
resources:
- name: resource1
type: aThing
- name: resource2
type: anotherThing
test.sh
j2 pipe.j2 > pipeline.yml
The test.sh script runs fine, but i'm trying to exclude either of the resources in the resources.yml file. I've tried the following in the j2 file getting different errors:
example1:
{% include 'resources.yml' | reject('equalto','resource1') %}
TypeError: generator is not subscriptable
example2:
{% include 'resources.yml' %}
{% set r_list = 'resources' %}
{% for rsc in r_list | rejectattr('name', 'equalto', 'resource1') %}
{% endfor %}
UndefinedError: 'str.object' has no attribute 'name'
I checked online and haven't found any examples of what I'm trying to accomplish and am not sure if it's possible or if I'm doing it completely wrong. I'm trying to get a file that only contains non-rejected items as a final result.
Hoping to get in pipeline.yml file:
resources:
- name: resource2
type: anotherThing
Related
We define external redshift spectrum table using an yml file. Like the one below.
- name: visitor
description: 'visitor data'
external:
location: "s3://{{ env_var('ENV') }}-temp-raw-s3/{{ s3_prefix({{ env_var('ENV') }}) }}/Visitor"
file_format: parquet
partitions:
- name: year
data_type: VARCHAR(10)
vals: # macro w/ keyword args to generate list of values
macro: dbt.dates_in_range
args:
start_date_str: '{{ var("START_DT") }}'
end_date_str: '{{ var("START_DT") }}'
in_fmt: '%Y-%m-%d'
out_fmt: '%Y'
I would like to change the prefix of the s3 bucket based on the environment variable.
Here is the macro I have tried to change the prefix. I am calling that macro in in the yml file above in line 4 (location).
{% macro s3_prefix(env_var) %}
{% if env_var == "prod" %}
prefix_string = "prefix_prod"
{% else %}
prefix_string = "prefix_non_prod"
{% endif %}
{{ return(prefix_string) }}
{% endmacro %}
I am getting some compilation errors. Is there an easier way to achieve this? Any help would be really appreciated.
I have a state using file.managed, which generates a config file via a jinja for loop from a key in pillar.
My pillar looks like this:
configuration:
server01:
key1: value1
key2: value2
server02:
key03: value03
key04: value04
and the managed file:
{% set kv = pillar['configuration']['server01'] %}
{% for key, value in kv.iteritems() %}
{{ key }}:{ value }};
{% endfor %}
The way I differentiate between different servers right now in my state file is
config:
file.managed:
- name: /etc/config.conf
- source: salt://files/{{ grains['id'] }}.conf.jinja
- template: jinja
but this is less than ideal, since I have to create an almost identical file for every server.
Is there a way to dynamically replace server01 with the ID of the actual server, something like
{% set kv = pillar['configuration']['{{ grains[id''] }}'] %}
The goal is to generally limit the necessary changes only to the corresponding pillar file, when adding a new server, so other suggestions are also welcome too.
i think you should use pillar info in your state file.
your state file like bellow :
{% if grains['id'] in pillar['configuration'] %}
{% set nodeinfo = pillar['configuration'][grains['id']] %}
config:
file.managed:
- name: /etc/config.conf
- source: salt://conf.jinja
- template: jinja
- defaults :
nodeinfo: {{nodeinfo}}
{% endif %}
then, conf.jinja:
{% for key, value in nodeinfo.iteritems() -%}
{{ key }}:{{ value }};
{% endfor -%}
i hope that will solve your problem, thanks.
Trying to use Google Deployment Manager with YAML and Jinja with a multi-line variables, such as:
startup_script_passed_as_variable: |
line 1
line 2
line 3
And later:
{% if 'startup_script_passed_as_variable' in properties %}
- key: startup-script
value: {{properties['startup_script_passed_as_variable'] }}
{% endif %}
Gives MANIFEST_EXPANSION_USER_ERROR:
ERROR: (gcloud.deployment-manager.deployments.create) Error in
Operation operation-1432566282260-52e8eed22aa20-e6892512-baf7134:
MANIFEST_EXPANSION_USER_ERROR
Manifest expansion encountered the following errors: while scanning a simple key in "" could not found expected ':' in ""
Tried (and failed):
{% if 'startup_script' in properties %}
- key: startup-script
value: {{ startup_script_passed_as_variable }}
{% endif %}
also
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{{ startup_script_passed_as_variable }}
{% endif %}
and
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{{ startup_script_passed_as_variable|indent(12) }}
{% endif %}
The problem is the combination of YAML and Jinja. Jinja escapes the variable but fails to indent it as YAML would require when being passed as a variable.
Related: https://github.com/saltstack/salt/issues/5480
Solution: Pass the multi-line variable as an array
startup_script_passed_as_variable:
- "line 1"
- "line 2"
- "line 3"
The quoting is important if your value starts with # (which startup script on GCE does, ie #!/bin/bash) since it will be treated as a comment otherwise.
{% if 'startup_script' in properties %}
- key: startup-script
value:
{% for line in properties['startup_script'] %}
{{line}}
{% endfor %}
{% endif %}
Putting it here since there aren't much Q&A material for Google Deployment manager.
There is no clean way to do this in Jinja. As you yourself have pointed out, because YAML is a whitespace-sensitive language, it is difficult to effectively template around.
One possible hack is to split the string property into a list and then iterate over the list.
For example, providing the property:
startup-script: |
#!/bin/bash
python -m SimpleHTTPServer 8080
you can use it in your Jinja template:
{% if 'startup_script' in properties %}
- key: startup-script
value: |
{% for line in properties['startup-script'].split('\n') %}
{{ line }}
{% endfor %}
Here is also a full working example of this.
This method will work, but generally cases like this are when people start considering using a python template. Because you are working with an object model in python, you do not have to deal with indentation problems. For example:
'metadata': {
'items': [{
'key': 'startup-script',
'value': context.properties['startup_script']
}]
}
An example of the python template can be found in the Metadata From File example.
I have the following structure in my Jekyll app:
_data/
test.json
items/
test/
index.html
I am using the following to grab just the name of the ending folder name of my item:
{% assign listing = {{ page.url | remove: 'items/' | replace:'/',' ' | truncatewords: 1 | remove:'...' | escape }} %}
What I'm then trying to do is access the data file which the matching folder name from the _data directory.
I've gotten it to:
{{ site.data.{{ listing }} }}
which allows me to see the data, but I can't actually go inside the JSON object to grab a specific item, like {{ site.data.{{ listing }}.test }} does not work. Any help would be greatly appreciated. Thanks!
Use some brackets like this :
{% assign datas = site.data[{{listing}}] %}
You can now access datas.test.
I am attempting to template yum .repo files. We have multiple internal and external yum repos that the various hosts we manage may or may not use.
I want to be able to specify any number of repos and what .repo file they will be templated in. It makes sense to group these repos in the same .repo file where they have a common purpose (e.g. all centos repos)
I am unable to determine how to combine ansible, yaml and j2 to achieve this. I have tried using the ansible 'with_items', 'with_subelements' and 'with_dict' unsuccessfully.
YAML data
yum_repo_files:
- centos:
- name: base
baseurl: http://mirror/base
- name: updates
baseurl: http://mirror/updates
- epel:
- name: epel
baseurl: http://mirror/epel
Ansible task
- name: create .repo files
template: src=yumrepos.j2 dest="/etc/yum.repos.d/{{ item }}.repo"
with_items: yum_repo_files
j2 template
{% for repofile in yum_repo_files.X %} {# X being the relative index for the current repofile, e.g. centos = 0 and epel = 1 #}
{% for repo in repofile %}
name={{ repo.name }}
baseurl={{ repo.baseurl }}
{% endfor %}
{% endfor %}
When you use with_items with the template module the special variable item will be passed into your jinja template.
Try this:
{% for repofile in item %}
{% for repo in repofile %}
name={{ repo.name }}
baseurl={{ repo.baseurl }}
{% endfor %}
{% endfor %}
user24364's answer helped solve half the issue, I then used some python methods to get the correct data out of the lists and dicts.
Giving the full filename 'centos.repo' rather than 'centos' simplified the logic (and aligned better with the logic for other tasks):
yum_repo_files:
- centos.repo:
- name: base
baseurl: http://mirror/base
- name: updates
baseurl: http://mirror/updates
- epel.repo:
- name: epel
baseurl: http://mirror/epel
The .iterkeys() and .next() methods are used on items to get the repo filenames out of the list of dicts:
- name: create .repo files
template: src=yumrepos.j2 dest="/etc/yum.repos.d/{{item.iterkeys().next()}}"
with_items: yum_repo_files
The .itervalues() method is used to get the list of dicts containing all the keys/values for each given repo:
{% for repofile in item.itervalues() %}
{% for repo in repofile %}
[{{repo.repo}}]
name={{ repo.name }}
baseurl={{ repo.baseurl }}
{% endfor %}
{% endfor %}
I also added some other tasks to clean up unmanaged files, etc. Once I've sanitised the code, I'll post it to the ansible galaxy as nobody else seems to have shared such a role.
Your files would be named as: *.repo.j2; then, you can use fileglob:
- name: create x template
- template: src={{ item }} dest=/tmp/{{ item | basename | regex_replace('.j2','') }}
- with_fileglob:
- files/*.j2
Reference:
https://serverfault.com/questions/578544/deploying-a-folder-of-template-files-using-ansible