Is it possible to map multiple attributes using Jinja/Ansible? - jinja2

I would like to build an output that shows the key and value of a variable.
The following works perfectly ...
# Format in Ansible
msg="{{ php_command_result.results | map(attribute='item') | join(', ') }}"
# Output
{'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}, {'value': {'svn_tag': '20150702r1_6.36_homeland'}, 'key': 'api'}
What I would like is to show the key and svn_tag together as so:
I'm able to display either the key or svn_tag but getting them to go together doesn't work.
msg="{{ php_command_result.results | map(attribute='item.key') | join(', ') }}"
# Output
ui, api
However, this is what I want.
# Desired Output
api - 20150702r1_6.36_homeland
ui - 20150703r1_6.36_homeland

Using Jinja statements:
- set_fact:
php_command_result:
results: [{"value":{"svn_tag":"20150703r1_6.36_homeland"},"key":"ui"},{"value":{"svn_tag":"20150702r1_6.36_homeland"},"key":"api"}]
- debug:
msg: "{% for result in php_command_result.results %}\
{{ result.key }} - {{ result.value.svn_tag }} |
{% endfor %}"
Outputs:
ok: [localhost] => {
"msg": "ui - 20150703r1_6.36_homeland | api - 20150702r1_6.36_homeland | "
}
If you want the results on separate lines:
- debug:
msg: "{% set output = [] %}\
{% for result in php_command_result.results %}\
{{ output.append( result.key ~ ' - ' ~ result.value.svn_tag) }}\
{% endfor %}\
{{ output }}"
Outputs:
ok: [localhost] => {
"msg": [
"ui - 20150703r1_6.36_homeland",
"api - 20150702r1_6.36_homeland"
]
}
Either of these can be put on one line if desired:
- debug:
msg: "{% for result in php_command_result.results %}{{ result.key }} - {{ result.value.svn_tag }} | {% endfor %}"
- debug:
msg: "{% set output = [] %}{% for result in php_command_result.results %}{{ output.append( result.key ~ ' - ' ~ result.value.svn_tag) }}{% endfor %}{{ output }}"

Here is solution without custom filter_plugin or running shell command. However, it requires additional fact to be set in a with_items loop(php_fmt).
- hosts: localhost
connection: local
gather_facts: false
tasks:
- set_fact:
php_command_result:
results: '[{"value":{"svn_tag":"20150703r1_6.36_homeland"},"key":"ui"},{"value":{"svn_tag":"20150702r1_6.36_homeland"},"key":"api"}]'
- set_fact:
php_fmt: "{{ php_fmt|default([])|union([item.key+' -- '+item.value.svn_tag ]) }}"
with_items: "{{ php_command_result.results }}"
- debug:
msg: "{{php_fmt|join(',')}}"

I found this question when I searched for a way to handle a similar task, but I wanted the output to be another JSON list with the individual elements. It took me many diversions over map, filter and even Jinja2 for loops, so I will post the answer here too. Just create the following file and run via ansible-playbook:
- hosts: localhost
connection: local
vars:
to_test: [ {'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}, {'value': {'svn_tag': '20150702r1_6.36_homeland'}, 'key': 'api'} ]
tasks:
- debug:
msg: "to_test: {{ to_test }}"
- debug:
msg: "to_test reduced: {{ to_test | json_query('[].{key: key, svn_tag: value.svn_tag}') | list }}"
- debug:
msg: "to_test reduced: {{ to_test | json_query(query1) | list }}"
vars:
query1: "[].{key: key, svn_tag: value.svn_tag}"
The end result is a nice JSON array:
[{'key': 'ui', 'svn_tag': '20150703r1_6.36_homeland'}, {'key': 'api', 'svn_tag': '20150702r1_6.36_homeland'}]"

Here is another answer using filter_plugins which I found very easy to use.
If anyone still needs this you can use the following code (put in playbooks/filter_plugins/mapattributes.py):
#!/usr/bin/env python
class FilterModule(object):
def filters(self):
return { 'mapattributes': self.mapattributes }
def mapattributes(self, list_of_dicts, list_of_keys):
l = []
for di in list_of_dicts:
newdi = { }
for key in list_of_keys:
# newdi[key] = di[key]
if di.get(key, None) != None:
newdi[key] = di[key]
l.append(newdi)
return l
And let say you have this list which you only need to keys from:
INTERFACES:
- { name: GigabitEthernet1/0/24 , enabled: yes, state: up , description: FRONT }
- { name: GigabitEthernet1/0/9 , enabled: yes, state: up , description: BACK }
Let's create another variable filtering only the needed key:values
test: "{{ INTERFACES | mapattributes(['name', 'description']) }}"
Test the output
- debug:
var: test
ok: [R1] => {
"test": [
{
"description": "FRONT",
"name": "GigabitEthernet1/0/24"
},
{
"description": "BACK",
"name": "GigabitEthernet1/0/9"
}
]
}
This allow me to have a big dictionary and only slice the keys i need to pass to ios_interface/aggregate
Thx to Nee6ione, i found it on pallets/jinja github issue

You can do it by using the following techniques:
Create filter_plugin. Add filter_plugins = <path to the folder> in ansible.cfg. Then create a file say my_plugin.py:
class FilterModule(object):
''' Custom filter '''
def filters(self, my_arg):
return <parse it here......>
Example:
playbook.yml
---
- hosts: localhost
gather_facts: no
connection: local
tasks:
- set_fact:
php_command_result:
results: {'value': {'svn_tag': '20150703r1_6.36_homeland'}, 'key': 'ui'}
- debug: msg="Hey look what I got '{{ php_command_result.results | a }}'"
my_plugin.py
import json
class FilterModule(object):
def filters(self):
return {'a': a}
def a(a):
r = '%s - %s' % (a['key'], a['value']['svn_tag'])
return r
Fast and easy approach: just use python/php/shell or what ever you prefer with shell module. Something like this:
- name: Pars output
shell: python -c "import json; json.loads('{{ php_command_result.results }}') ....

Related

JMESPath Query in Ansible

--- EDIT ---
Partial-Solution: Messed around with the JMESPath syntax and was able to successfully get a match for the first test case (without the optional variable) using:
jmesquery: "{{ datacenter }}{{ subcategory }}.{{ refine_hosts }}.[*][].[*][][]"
I am writing an Ansible Playbook that takes a list of hosts from a network server, parses the JSON list, and finds hostnames that matches the user's input when they deploy the playbook as a Jenkin's Job through it's API.
The issue I am encountering is that I am unable to successfully query the JSON host list. Currently, I am only trying to run the following test case:
datacenter: a
subcategory: bc
refine_hosts: QA
However, the final version of this playbook should be able to take in values for datacenter, subcategory, and refine_hosts with an optional input value of host_type. An example test case including the optional input value would be the following:
datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
In my playbook, I am using JMESPath within the following task:
- name: Build HOSTS list
set_fact:
hosts_list: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: '%%datacenter%%-%%subcategory%%.%%refine_hosts%%.[*][*][][]'
The JSON host list is structured in the following manner (I am unable to edit the structure of the host list, but it will always follow the following structure nonetheless):
{
"a-bc":{
"all":{
"webServer":[
],
"archive":[
"someHostAlias-123.privateDomain.com"
],
"central":[
"someHostAlias-456.privateDomain.com"
]
},
"QA":{
"xyz":{
"INBOUND_HTTP":[
"someHostAlias-789.privateDomain.com"
],
"WEBSITE":[
"someHostAlias-1011.privateDomain.com"
]
}
}
}
}
I have been using the following websites for this issue:
JMESPath Tutorial
Ansible JMESPath Documentation
JSONPath Expression Tester
StackOverflow: How to Use Variable in JMESPath Expression
Gitter: JMESPath/chat
I apologize if the query seems obvious, this is my first attempt at an Ansible Playbook. All help/feedback is greatly appreciated.
One of the issue of your query is that you are confusing [*] — a list projection — that selects all the elements of a list with .* — an object projection — that selects all the properties of a dictionary.
So, one solution in JMESPath, would be to do:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
Given the playbook:
- hosts: localhost
gather_facts: no
tasks:
- debug:
msg: "{{ jsondata | json_query(jmesquery) }}"
loop: "{{ fake_user_input }}"
loop_control:
label: "{{ jmesquery }}"
vars:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
datacenter: "{{ item.datacenter }}"
subcategory: "{{ item.subcategory }}"
refine_hosts: "{{ item.refine_hosts }}"
host_type: "{{ item.host_type | default('') }}"
fake_user_input:
- datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
- datacenter: a
subcategory: bc
refine_hosts: QA
jsondata:
a-bc:
all:
webServer: []
archive:
- someHostAlias-123.privateDomain.com
central:
- someHostAlias-456.privateDomain.com
QA:
xyz:
INBOUND_HTTP:
- someHostAlias-789.privateDomain.com
WEBSITE:
- someHostAlias-1011.privateDomain.com
This yields:
ok: [localhost] => (item="a-bc".QA.*. WEBSITE[] | []) =>
msg:
- someHostAlias-1011.privateDomain.com
ok: [localhost] => (item="a-bc".QA.*. *[] | []) =>
msg:
- someHostAlias-789.privateDomain.com
- someHostAlias-1011.privateDomain.com

Ansible loop JSON array contains objects and replace each found element in JSON in file

i have this JSON :
[{
"${foo1}": "somevalue1",
"${foo2}": "somevalue2"
}, {
"${zoo1}": "somevalue111",
"${zoo2}": "somevalue222"
}]
In which i need to loop over in ansible and each object in the JSON array each key / pair value
I like to check if the key exists in the file and replace it with the value .
For example:
if in the file i have
key1=${foo1}
it will be replaced with
key1=somevalue1
i use to have hard coded ansible task :
replace:
path: "/myfile.txt"
regexp: "{{ item.regexp }}"
replace: "{{ item.replace }}"
with_items:
- { regexp: '${foo1}', replace: "new_value" }
i like to convert it to dynamically search and replace according to the given JSON
i capture the JSON like this : ( this is trim down YAML )
---
-
gather_facts: false
hosts: localhost
name: test
tasks:
- name: debug
debug:
msg: "{{ AAA| to_json }}"
- name: set fact
set_fact:
multi_parms: "{{ AAA| to_json }}"
- name: debug multi_parms
debug:
var: multi_parms
Read the JSON file into a single dictionary, e.g.
- set_fact:
my_data: "{{ my_data|d({})|combine(item) }}"
loop: "{{ lookup('file', 'test.json') }}"
gives
my_data:
${foo1}: somevalue1
${foo2}: somevalue2
${zoo1}: somevalue111
${zoo2}: somevalue222
Given an INI file, e.g.
shell> cat conf.ini
key1=${foo1}
key2=${foo2}
key3=xyz
parse the configuration into a dictionary, e.g.
- set_fact:
my_ini: "{{ dict(_keys|zip(_vals)) }}"
vars:
_lines: "{{ lookup('file', 'conf.ini').splitlines() }}"
_regex: '^(.*)=(.*)$'
_keys: "{{ _lines|map('regex_replace', _regex, '\\1') }}"
_vals: "{{ _lines|map('regex_replace', _regex, '\\2') }}"
gives
my_ini:
key1: ${foo1}
key2: ${foo2}
key3: xyz
Then use the module ini_file to substitute the data, e.g.
- ini_file:
path: conf.ini
section:
option: "{{ item }}"
value: "{{ my_data[my_ini[item]] }}"
no_extra_spaces: true
loop: "{{ my_ini|list }}"
when: my_ini[item] in my_data
gives
shell> cat conf.ini
key1=somevalue1
key2=somevalue2
key3=xyz
Q: "How can I do the search and replace in the destination file?"
A: See below the "--diff" output of each iteration.
TASK [ini_file] ***************************************************
--- before: conf.ini (content)
+++ after: conf.ini (content)
## -1,3 +1,3 ##
-key1=${foo1}
+key1=somevalue1
key2=${foo2}
key3=xyz
changed: [localhost] => (item=key1)
--- before: conf.ini (content)
+++ after: conf.ini (content)
## -1,3 +1,3 ##
key1=${foo1}
-key2=${foo2}
+key2=somevalue2
key3=xyz
changed: [localhost] => (item=key2)
skipping: [localhost] => (item=key3)
this code in j2 template
{% for item in json_data %}
{% for k, v in item.items() %}
key{{loop.index}} = {{ v }}
{% endfor %}
{% endfor %}
gets you this result
key1 = somevalue1
key2 = somevalue2
key1 = somevalue111
key2 = somevalue222
is this what you were looking for?

Why is Ansible unable to read unicode string as JSON?

Summary
When retrieving data using the uri module in Ansible, I am unable to parse a section of it as JSON to retrieve a nested value.
The desired value is the ci field inside the content.data or json.data field (see output below).
Steps to Reproduce
site.yml
---
- hosts: localhost
gather_facts: false
tasks:
- name: Get String
uri:
url: "http://localhost/get-data"
method: POST
body_format: json
body: "{ \"kong-jid\": \"run-sn-discovery\" }"
return_content: yes
register: output
- set_fact:
ci: "{{ output.json.data.ci }}"
- debug:
msg: "{{ ci }}"
The {{ output }} variable
{
u'status': 200,
u'cookies': {},
u'url': u'http://kong-demo:8000/get-data',
u'transfer_encoding': u'chunked',
u'changed': False,
u'connection': u'close',
u'server': u'kong/0.34-1-enterprise-edition',
u'content':
u'{"data":"\\"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}\\""}',
'failed': False,
u'json': {u'data': u'"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}"'},
u'content_type': u'application/json',
u'date': u'Thu, 18 Apr 2019 15:50:25 GMT',
u'redirected': False,
u'msg': u'OK (unknown bytes)'
}
Result
[user#localhost]$ ansible-playbook site.yml
[WARNING]: Could not match supplied host pattern, ignoring: all
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [localhost] ***************************************************************************************************************
TASK [Pass Redis data to next task as output] **********************************************************************************
ok: [localhost]
TASK [set_fact] ****************************************************************************************************************
fatal: [localhost]: FAILED! => {}
MSG:
The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
The error appears to have been in 'site.yml': line 19, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- set_fact:
^ here
exception type: <class 'ansible.errors.AnsibleUndefinedVariable'>
exception: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
Important Troubleshooting Information
It appears the root issue is related to which Ansible type being interpreted. I desire to parse ci from the output in one task.
The two-task solution shown below works, but this leads me to believe this should be possible in one line...
Two-Task Solution
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci['ci'] }}"
But the ci fact set from {{ output.json.data | from_json }} reports a different TYPE than the inline type...
Unicode or Dict?
- debug:
msg: "{{ output.json.data | from_json | type_debug }}" # returns unicode
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci | type_debug }}" # returns dict
Why isn't {{ output.json.data | from_json | type_debug }}
the same as {{ ci | type_debug }}?
Although json and data are keys in their resp objects, ci is just part of a larger string (which happens to look like a JSON object
If the relevant line in your datastructure would be:
u'json': {u'data': {'ci': u'3bb8d625dbac3700e4f07b6e0f96195b'}},
then you could expect to use "{{ output.json.data.ci }}" but not when the .ci part is just a normal part of a string.

Ansible Extract JSON Tag

I'm trying to work with Infoblox API, and it's responses. I would need to extract values of tags from the response, that seems to be in JSON format, but I cannot find the way to do it.
Here is my playbook:
- name: "Checking _node_exporter Service Record for {{ inventory_hostname }}"
local_action:
module: uri
url: "{{ infobloxapiurl }}record:srv?name=_node_exporter.domain.com&target={{ inventory_hostname }}"
force_basic_auth: yes
user: "{{ infobloxuser }}"
password: "{{ infobloxpassword }}"
validate_certs: no
return_content: yes
register: _infoblox_results
- debug:
var: _infoblox_results.json
The _infoblox_results.json variable looks like this:
TASK [prometheus : debug] *******************************************************************************************************************************************************************************************
task path: /ansible/roles/tasks/task.yml:38
ok: [server.domain.com] => {
"_infoblox_results.json": [
{
"_ref": "record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default",
"name": "_node_exporter.domain.com",
"port": 9100,
"priority": 0,
"target": "server.domain.com",
"view": "default",
"weight": 0
}
]
}
I want to use the data of _ref from _infoblox_results.json, but I wasn't able to extract it with regex_replace (it drops back the full _infoblox_results.json):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | regex_replace('record:srv.*\\/default,', '\\1') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Neither with json_query (it drops back nothing):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | json_query('_ref') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Can someone please point me into the right direction?
You have already an object in the memory, so simply refer to its value: _infoblox_results.json[0]._ref contains the string record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default.
With that you can split the string and select the second element:
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json[0]._ref.split('/')[1] }}"

Ansible parse json and read result into different variables

I've set up a task which queries the github api meta endpoint and returns the following
{
"verifiable_password_authentication": true,
"github_services_sha": "f9e3a6b98d76d9964a6613d581164039b8d54d89",
"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
],
"git": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20",
"13.229.188.59/32",
"13.250.177.223/32",
"18.194.104.89/32",
"18.195.85.27/32",
"35.159.8.160/32",
"52.74.223.119/32"
],
"pages": [
"192.30.252.153/32",
"192.30.252.154/32",
"185.199.108.153/32",
"185.199.109.153/32",
"185.199.110.153/32",
"185.199.111.153/32"
],
"importer": [
"54.87.5.173",
"54.166.52.62",
"23.20.92.3"
]
}
What I need to do is get the 3 hook IPs and read them each into their own variable.
I've tried a couple of solutions i've found around but nothing is seeming to work for me.
I've got as far as drilling down into the json so i'm being returned only the 3 IPs, but how do I get them out and into variables individually?
i gave it a shot using j2 syntax in the variable name part, and - TIL - looks like the jinja2 syntax is allowed in that part as well!
please see playbook to process the hooks list variable and assign to variables variable_1, variable_2, variable_3 and so on:
- hosts: localhost
gather_facts: false
vars:
counter: 1
hooks:
- 192.30.252.0/22
- 185.199.108.0/22
- 140.82.112.0/20
tasks:
- name: populate vars
set_fact:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ hooks }}"
- name: print vars
debug:
msg: "variable_1: {{variable_1}}, variable_2: {{variable_2}}, variable_3: {{variable_3}}"
and the output:
[root#optima-ansible ILIAS]# ansible-playbook 50257063.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [populate vars] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.30.252.0/22)
ok: [localhost] => (item=185.199.108.0/22)
ok: [localhost] => (item=140.82.112.0/20)
TASK [print vars] **********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "variable_1: 192.30.252.0/22, variable_2: 185.199.108.0/22, variable_3: 140.82.112.0/20"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[root#optima-ansible ILIAS]#
hope it helps
UPDATE:
something weird i noticed - also TIL - is that if you reverse the lines:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
to:
counter: "{{ counter | int + 1 }}"
variable_{{counter}}: "{{ item }}"
you still end up with the same variable names, _1 to _3, while i would expect to get _2 to _4.
I guess ansible loops behave differently than expected from other programming languages.
---
- name: Query Github Meta API and get Hook Ips
hosts: local
connection: local
vars:
counter: 1
tasks:
- name: Query API
uri:
url: https://api.github.com/meta
return_content: yes
register: response
- name: Populate Hook Variables
set_fact:
webhook_ip_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ response['json']['hooks'] }}"
- name: print vars
debug:
msg: "Variable_1: {{ webhook_ip_1 }}, Variable_2: {{ webhook_ip_2 }}, Variable_3: {{ webhook_ip_3 }}"
Works with GitHub Webhook IPs in a loop
- name: get request to github
uri:
url: "https://api.github.com/meta"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
#X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: GitHub webhook IPs
debug:
msg: "{{ item }}"
with_items: "{{ (json_response.content | from_json).hooks }}"