Insert multiline json string into helm template for base64 encoding - json

I am trying to insert multiline json string into helm template for base64 encoding required for Kubernetes secret.
Goals:
helm value is injected into json string
multi-line json string must be base64 encoded using b64enc
myfile1.json does not work but myfile2.json works.
I prefer not to put entire json file in values.yaml.
apiVersion: v1
kind: Secret
metadata:
name: {{ template "mychart.fullname" . }}
labels:
app: {{ template "mychart.name" . }}
chart: {{ template "mychart.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
type: Opaque
data:
myfile.json: {{ |-
{
"item1": {
"name": "{{ .Values.item1.name }}"
},
"item2": {
}
} | b64enc }}
myfile2.json: {{ .Values.myfile2 | b64enc }}

You actually don't need to base64-encode the secret in the helm chart. If you use the stringData field instead of data field, Kubernetes knows that it needs to base64 encode the data upon the secret's deployment.
From the docs (Source):
The Secret contains two maps: data and stringData. The data field is used to store arbitrary data, encoded using base64. The stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.
So we can rewrite your secret using stringData instead of data and keep multiline json strings in templates like so:
apiVersion: "v1"
kind: "Secret"
metadata:
name: {{ template "mychart.fullname" . }}
labels:
app: {{ template "mychart.name" . }}
chart: {{ template "mychart.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
type: "Opaque"
stringData:
myfile.json: |-
{
"item1": {
"name": "{{ .Values.item1.name }}"
},
"item2": {
}
}
myfile2.json: {{ .Values.myfile2 }}
Note that this does not mean you suddenly need to worry about having unencoded secrets. stringData will ultimately be base64-encoded and converted to data when it is installed, so it will behave exactly the same once it's loaded into Kubernetes.
Again, from the docs (emphasis mine) (Source):
stringData allows specifying non-binary secret data in string form. It is provided as a write-only convenience method. All keys and values are merged into the data field on write, overwriting any existing values. It is never output when reading from the API.

My impression (and others seem to have hit it too) is you have to compromise either on it being multi-line or on not putting it in a file. I think the problem is that you have to use a yaml instruction (the |-) to get multiple lines and that is part of the template itself so you can't get an 'output' from it in a way that you can then feed into b64enc.
If this were a ConfigMap you wouldn't need to feed into b64enc so it would be as simple as:
myfile.json: |
{
"item1": {
"name": "{{ .Values.item1.name }}"
},
"item2": {
}
}
Or if you were to compromise on single-line approach then that could be:
myfile.json: {{ tpl ("{ 'item1': { 'name': '{{ .Values.item1.name }}' }, 'item2': { } }") . | toJson | b64enc }}
If it were coming from a file then you could use {{ tpl (.Files.Get "files/myfile.json") . | b64enc | quote }}
Another option would be to put the whole json in the values file
Or you could have a myfile entry in your values file like:
myfile:
item1:
name: "bob"
item2:
name: "fred"
And then use it with myfile.json: {{ .Values.myfile | toJson | b64enc }}

I found a solution. You can use tpl function on json file to render template.
apiVersion: v1
kind: Secret
metadata:
name: {{ template "mychart.fullname" . }}
labels:
app: {{ template "mychart.name" . }}
chart: {{ template "mychart.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
type: Opaque
data:
myfile.json: {{ tpl(.Files.Get "myfile.json") . | b64enc }}
myfile.json
{
"item1": {
"name": "{{ .Values.item1.name }}"
},
"item2": {
}
}

Related

Is there any way to prevent ansible changing my JSON stored in a variable?

Edit: As suggested by Vladimir Botka here is a minimal example.
I have this template file json.j2
{"foo":"{{ var }}"}
which I would like to render into a variable and insert that variable into another template
- name render it
ansible.builtin.template:
src: another.j2
[...]
vars:
json_from_template: "{{ lookup('template', 'json.j2') }}"
As it seems ansible interprets the json from json.j2 and changes it to
{ 'foo': 'var_value' }
But "because of reasons" I need the json unchanged, no spaces, no single quotes, as specified in the json.j2.
Can I tell ansible to keep the result of json.j2 as a raw string?
I already tried !unsafe, but this
vars:
json_from_template: !unsafe "{{ lookup('template', 'json.j2') }}"
only renders the raw lookup command without any templating at all, and this
vars:
json_from_template: "{{ !unsafe lookup('template', 'json.j2') }}"
does not work at all.
Note: In the snippets below, the callback yaml is used to display the output to the screen.
shell> ansible-config dump | grep DEFAULT_STDOUT_CALLBACK
DEFAULT_STDOUT_CALLBACK(/home/admin/.ansible.cfg) = yaml
Ansible, when templating Jinja2, tries and converts strings that look like python data structures to python data structures. For example
shell> cat json.j2
{"foo":"{{ var }}"}
- set_fact:
json_from_template: "{{ lookup('template', 'json.j2') }}"
vars:
var: var_value
- debug:
var: json_from_template
- debug:
var: json_from_template|type_debug
gives
json_from_template:
foo: var_value
json_from_template|type_debug: dict
You can see the type of the variable json_from_template is the dictionary. What you need is a string. You have to explicitly declare it
- set_fact:
json_from_template: "{{ lookup('template', 'json.j2')|string }}"
vars:
var: var_value
- debug:
var: json_from_template|string
- debug:
var: json_from_template|type_debug
gives
json_from_template|string: |-
{"foo":"var_value"}
json_from_template|type_debug: str
It seems that this "magic" conversion works for valid JSON only. See the variable test_2 below. Other variables remain strings despite the fact that they are valid YAML dictionaries
vars:
val1: foo
test1: |-
{"key1": "foo"}
test2: |-
{"key1": "{{ val1 }}"}
test3: |-
{"key1": {{ val1 }}}
test4: |-
{key1: {{ val1 }}}
test5: |-
key1: {{ val1 }}
tasks:
- debug:
msg: |
test1: {{ test1|type_debug }}
test2: {{ test2|type_debug }}
test3: {{ test3|type_debug }}
test4: {{ test4|type_debug }}
test5: {{ test5|type_debug }}
- debug:
msg: |
test1|from_yaml: {{ test1|from_yaml|type_debug }}
test3|from_yaml: {{ test3|from_yaml|type_debug }}
test4|from_yaml: {{ test4|from_yaml|type_debug }}
test5|from_yaml: {{ test5|from_yaml|type_debug }}
gives
msg: |-
test1: AnsibleUnicode
test2: dict
test3: str
test4: str
test5: str
msg: |-
test1|from_yaml: dict
test3|from_yaml: dict
test4|from_yaml: dict
test5|from_yaml: dict
The "magic" conversion applies to Jinja2 templates in Ansible expressions only. For example
- debug:
msg: "{{ json_from_template|string }}"
- debug:
msg: "{{ json_from_template }}"
gives
msg: |-
{"foo":"var_value"}
msg:
foo: var_value
Jinja2 templates inside the template module evaluate always to strings. For example
shell> cat another.j2
{{ json_from_template|string }}
{{ json_from_template }}
- template:
src: another.j2
dest: output.txt
gives
shell> cat output.txt
{"foo":"var_value"}
{"foo":"var_value"}

Extracting values from YAML into Jinja template for Ansible playbook

I have a YAML file with content as like below:
cat ../../ansible/playbooks/vars/patching-config.yml
---
patching_tag_name: "Patching"
my_windows_patching:
- {
OS: "WINDOWS",
tag_value: "myProdA",
frequency: "Month", #patching frequency. OneTime|Day|Hour|Week|Month|Minute
interval: 1, #interval of the schedule.
rebootSetting: "never", #ifRequired|never|always
PatchGroup: testA,
startDate: "2020-01-16T23:59:59Z",
expiryDate: "2020-02-16T23:59:59Z",
duration: "PT2H0M",
timeZone: "Australia/Sydney",
updateClassifications: "Critical,Important,Moderate"
}
I want to extract the values of updateClassifications from above YML file in Jinja Template file MaintenanceWindow.yml.j2
Resources:
WindowsNonProdBaseline:
Type: AWS::SSM::PatchBaseline
Properties:
Name: Windows-Non-Prod-Baseline
Description: Baseline containing all updates approved for Windows instances
OperatingSystem: {{ item.OS }}
PatchGroups:
- {{ item.PatchGroup }}
ApprovalRules:
PatchRules:
- PatchFilterGroup:
PatchFilters:
- Values:
# - Critical
# - Important
# - Moderate
{% for item in item.updateClassifications %}
- {{ item }}
{% endfor %}
I'm trying with the code described above, below one more time snippet:
{% for item in item.updateClassifications %}
- {{ item }}
{% endfor %}
I'm calling patching-config.yml in my tasks/main.yml as below
- include_vars: "{{playbook_dir}}/vars/patching-config.yml"
ignore_errors: yes
- name: create a cloudformation stack
cloudformation:
stack_name: "New-Ansible-cloudformation"
state: "present"
disable_rollback: true
template_body: "{{ lookup('template', '../../cloudformation/patching/MaintenanceWindow.yml.j2') }}"
with_items: "{{ telstra_windows_patching }}"
Finally, invoking role as below
cat ansible/playbooks/patching.yml
---
- hosts: localhost
roles:
- patching-cf-ssm
Unfortunately, it is not working.
Any lead shall be greatly appreciated.
Couple of things:
Your task is using telstra_windows_patching in with_items where as your variable file has variable name as my_windows_patching.
Assuming you are using the same name say my_windows_patching in your task and var file, if you are trying to save json object in yaml variable my_windows_patching you don't need - before curly braces. You can define something like this
my_windows_patching:
{
OS: "WINDOWS",
tag_value: "myProdA",
frequency: "Month", #patching frequency. OneTime|Day|Hour|Week|Month|Minute
interval: 1, #interval of the schedule.
rebootSetting: "never", #ifRequired|never|always
PatchGroup: testA,
startDate: "2020-01-16T23:59:59Z",
expiryDate: "2020-02-16T23:59:59Z",
duration: "PT2H0M",
timeZone: "Australia/Sydney",
updateClassifications: "Critical,Important,Moderate"
}
If you want to use elements inside my_windows_patching object with dot notation directly you could change the variable from object to a list something like,
my_windows_patching:
- OS: "WINDOWS"
tag_value: "myProdA"

How do I use json variables in a yaml file (Helm)

I have a HELM values file which looks like so:
service:
environment: dev
spring_application_json: >-
{
"spring" : {
"boot" : {
"admin" : {
"client" : {
"enabled" : "false",
"url" : "http://website1",
"instance" : {
"service-base-url" : "http://website2",
"management-base-url" : "http://website3"
}
}
}
}
}
}
And a corresponding template file which grabs this value and inserts it as an environment variable to a container.
spec:
replicas: {{ .Values.replicaCount }}
template:
spec:
imagePullSecrets:
- name: {{ .Values.image.pullSecret }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: ENVIRONMENT
value: "{{ .Values.service.environment }}"
- name: SPRING_APPLICATION_JSON
value: "{{ .Values.service.spring_application_json }}"
However when I run the helm install I get the following error:
Error: YAML parse error on deployment.yaml: error converting YAML to JSON: yaml: line 40: did not find expected key
Which points to the line:
value: "{{ .Values.service.spring_application_json }}"
I believe its a problem with the way I'm trying to parse in a json string as a multiline environment variable? The ENVIRONMENT 'dev' variable works perfectly and this same YAML also works perfectly with docker-compose.
There's an example a bit like this in the docs for spring cloud dataflow but the format in their documentation has the quotes escaped.
I was able to recreate the error and get past it by changing the values file entry to:
service:
spring_application_json:
{
"spring" : {
"boot" : {
"admin" : {
"client" : {
"enabled" : "false",
"url" : "http://website1",
"instance" : {
"service-base-url" : "http://website2",
"management-base-url" : "http://website3"
}
}
}
}
}
}
And the deployment entry to:
- name: SPRING_APPLICATION_JSON
value: {{ .Values.service.spring_application_json | toJson | quote }}
Notice no quotes around this part as that is handled anyway.

how it I parse each row of data from the google sheets api with ansible's with_items?

I am using the Google Sheets V4 Values collection and I am having trouble figuring out how to get each row to parse in to an {{ item }}
My Ansible ymal looks like.
tasks:
- name: debug url
debug:
msg: "{{ g_url }}"
- name: GET data from google spead sheet api
uri:
url: "{{ g_url }}"
return_content: yes
dest: /tmp/o_gd_form.json
register: google_data
- name: whats the dump?
debug:
var: "{{ item.0 |to_json }}"
with_items: "{{ google_data.json.values() }}" # this is the line that needs to be fixed
And the the responding json looks like:
{
"range": "Sheet1!A1:D5",
"majorDimension": "ROWS",
"values": [
["Item", "Cost", "Stocked", "Ship Date"],
["Wheel", "$20.50", "4", "3/1/2016"],
["Door", "$15", "2", "3/15/2016"],
["Engine", "$100", "1", "30/20/2016"],
["Totals", "$135.5", "7", "3/20/2016"]
],
}
I can't seem figure out how to write the the with_items to return an {{ item }} of json like ["Engine", "$100", "1", "30/20/2016"].
any help or do I need to split this task out to some middleware parser?
The Google sheets api documentation is at:
https://developers.google.com/sheets/samples/reading
To get what you want, use:
- debug: msg="Name={{ item.0 }} Cost={{ item.1 }}"
with_list: "{{ google_data.json['values'] }}"
There are two problems in your code:
values is a special keyword, so you can't access json.values, but should use json['values'] instead.
with_items flattens the list of lists, so you end up with long list of strings. You should use with_list to iterate over outer list (rows) and get inner lists (column values) as items.

Is it possible to map multiple attributes using Jinja/Ansible?

I would like to build an output that shows the key and value of a variable.
The following works perfectly ...
# Format in Ansible
msg="{{ php_command_result.results | map(attribute='item') | join(', ') }}"
# Output
{'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}, {'value': {'svn_tag': '20150702r1_6.36_homeland'}, 'key': 'api'}
What I would like is to show the key and svn_tag together as so:
I'm able to display either the key or svn_tag but getting them to go together doesn't work.
msg="{{ php_command_result.results | map(attribute='item.key') | join(', ') }}"
# Output
ui, api
However, this is what I want.
# Desired Output
api - 20150702r1_6.36_homeland
ui - 20150703r1_6.36_homeland
Using Jinja statements:
- set_fact:
php_command_result:
results: [{"value":{"svn_tag":"20150703r1_6.36_homeland"},"key":"ui"},{"value":{"svn_tag":"20150702r1_6.36_homeland"},"key":"api"}]
- debug:
msg: "{% for result in php_command_result.results %}\
{{ result.key }} - {{ result.value.svn_tag }} |
{% endfor %}"
Outputs:
ok: [localhost] => {
"msg": "ui - 20150703r1_6.36_homeland | api - 20150702r1_6.36_homeland | "
}
If you want the results on separate lines:
- debug:
msg: "{% set output = [] %}\
{% for result in php_command_result.results %}\
{{ output.append( result.key ~ ' - ' ~ result.value.svn_tag) }}\
{% endfor %}\
{{ output }}"
Outputs:
ok: [localhost] => {
"msg": [
"ui - 20150703r1_6.36_homeland",
"api - 20150702r1_6.36_homeland"
]
}
Either of these can be put on one line if desired:
- debug:
msg: "{% for result in php_command_result.results %}{{ result.key }} - {{ result.value.svn_tag }} | {% endfor %}"
- debug:
msg: "{% set output = [] %}{% for result in php_command_result.results %}{{ output.append( result.key ~ ' - ' ~ result.value.svn_tag) }}{% endfor %}{{ output }}"
Here is solution without custom filter_plugin or running shell command. However, it requires additional fact to be set in a with_items loop(php_fmt).
- hosts: localhost
connection: local
gather_facts: false
tasks:
- set_fact:
php_command_result:
results: '[{"value":{"svn_tag":"20150703r1_6.36_homeland"},"key":"ui"},{"value":{"svn_tag":"20150702r1_6.36_homeland"},"key":"api"}]'
- set_fact:
php_fmt: "{{ php_fmt|default([])|union([item.key+' -- '+item.value.svn_tag ]) }}"
with_items: "{{ php_command_result.results }}"
- debug:
msg: "{{php_fmt|join(',')}}"
I found this question when I searched for a way to handle a similar task, but I wanted the output to be another JSON list with the individual elements. It took me many diversions over map, filter and even Jinja2 for loops, so I will post the answer here too. Just create the following file and run via ansible-playbook:
- hosts: localhost
connection: local
vars:
to_test: [ {'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}, {'value': {'svn_tag': '20150702r1_6.36_homeland'}, 'key': 'api'} ]
tasks:
- debug:
msg: "to_test: {{ to_test }}"
- debug:
msg: "to_test reduced: {{ to_test | json_query('[].{key: key, svn_tag: value.svn_tag}') | list }}"
- debug:
msg: "to_test reduced: {{ to_test | json_query(query1) | list }}"
vars:
query1: "[].{key: key, svn_tag: value.svn_tag}"
The end result is a nice JSON array:
[{'key': 'ui', 'svn_tag': '20150703r1_6.36_homeland'}, {'key': 'api', 'svn_tag': '20150702r1_6.36_homeland'}]"
Here is another answer using filter_plugins which I found very easy to use.
If anyone still needs this you can use the following code (put in playbooks/filter_plugins/mapattributes.py):
#!/usr/bin/env python
class FilterModule(object):
def filters(self):
return { 'mapattributes': self.mapattributes }
def mapattributes(self, list_of_dicts, list_of_keys):
l = []
for di in list_of_dicts:
newdi = { }
for key in list_of_keys:
# newdi[key] = di[key]
if di.get(key, None) != None:
newdi[key] = di[key]
l.append(newdi)
return l
And let say you have this list which you only need to keys from:
INTERFACES:
- { name: GigabitEthernet1/0/24 , enabled: yes, state: up , description: FRONT }
- { name: GigabitEthernet1/0/9 , enabled: yes, state: up , description: BACK }
Let's create another variable filtering only the needed key:values
test: "{{ INTERFACES | mapattributes(['name', 'description']) }}"
Test the output
- debug:
var: test
ok: [R1] => {
"test": [
{
"description": "FRONT",
"name": "GigabitEthernet1/0/24"
},
{
"description": "BACK",
"name": "GigabitEthernet1/0/9"
}
]
}
This allow me to have a big dictionary and only slice the keys i need to pass to ios_interface/aggregate
Thx to Nee6ione, i found it on pallets/jinja github issue
You can do it by using the following techniques:
Create filter_plugin. Add filter_plugins = <path to the folder> in ansible.cfg. Then create a file say my_plugin.py:
class FilterModule(object):
''' Custom filter '''
def filters(self, my_arg):
return <parse it here......>
Example:
playbook.yml
---
- hosts: localhost
gather_facts: no
connection: local
tasks:
- set_fact:
php_command_result:
results: {'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}
- debug: msg="Hey look what I got '{{ php_command_result.results | a }}'"
my_plugin.py
import json
class FilterModule(object):
def filters(self):
return {'a': a}
def a(a):
r = '%s - %s' % (a['key'], a['value']['svn_tag'])
return r
Fast and easy approach: just use python/php/shell or what ever you prefer with shell module. Something like this:
- name: Pars output
shell: python -c "import json; json.loads('{{ php_command_result.results }}') ....