Ansible: json element strip key with colon - json

How do you remove a json Key that contains a colon (:) with jinja2 rejectattr.
Environment:
ansible 2.9.1
config file = None
configured module search path = [u'/home/<user>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.15+ (default, Oct 7 2019, 17:39:04) [GCC 7.4.0]
json data:
{
"tag:environment": "qa",
"tag:instance_id": "i-123456789"
}
Ansible playbook:
- name: Remove InstanceID
debug:
msg: "{{ instance_filter | rejectattr('['tag:environment'], 'defined' ') | list }}
Actual Results:
fatal: [localhost]: FAILED! => {
"msg": "template error while templating string: expected token ',', got 'tag'. String: {{ instance_filter | rejectattr('['tag:environment'], 'defined' ') | list }}"
}
Expected results:
{
"tag:environment": "qa"
}

The rejectattr is indeed one of the key filters to use to achieve your goal, but a few more things are needed. Here is the correct sequence of filters to remove that particular key from the dictionary variable you have:
Playbook:
---
- hosts: localhost
gather_facts: false
vars:
instance_filter:
tag:environment: qa
tag:instance_id: i-123456789
tasks:
- name: print var
debug:
var: instance_filter
- name: manipulate the var
debug:
msg: "{{ instance_filter | dict2items | rejectattr('key', 'equalto', 'tag:instance_id') | list | items2dict }}"
Output:
PLAY [localhost] *******************************************************************************************************************************************************************************************************
TASK [print var] *******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"instance_filter": {
"tag:environment": "qa",
"tag:instance_id": "i-123456789"
}
}
TASK [manipulate the var] **********************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": {
"tag:environment": "qa"
}
}
PLAY RECAP *************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
hope it helps.

Q: "How do you remove a JSON key?"
A: It's possible to create custom filter plugins. For example
$ cat filter_plugins/dict_utils.py
def dict_del_key(d, key):
del d[key]
return d
class FilterModule(object):
''' Ansible filters. Interface to Python dictionary methods.'''
def filters(self):
return {
'dict_del_key' : dict_del_key
}
The play below
- hosts: localhost
vars:
dict:
'tag:environment': 'qa'
'tag:instance_id': 'i-123456789'
tasks:
- debug:
msg: "{{ dict|dict_del_key('tag:instance_id') }}"
gives
msg:
tag:environment: qa
Notes:
See If you quote those config keys, they will become strings.
See the difference between 7.3.1. Double-Quoted Style and 7.3.2. Single-Quoted Style.
FWIW. See other filters available at GitHub.

Related

Using Key(#) function to extract keys from an object in Ansible

I have a file file.sub which contains this JSON object {"kas_sub.test1": "true", "kas_sub.test2": "true"}. I would extract the keys and to get this: kas_sub.test1 kas_sub.test1.
When i try
- shell: 'cat path/to/file.sub'
register: file1
- debug:
var: file1.stdout_lines
I got:
TASK [shell] *****************************************************************************************************************
changed: [ansible4]
changed: [control]
TASK [debug] *****************************************************************************************************************
ok: [control] => {
"file1.stdout_lines": [
"{\"kas_sub.tes1\": \"true\", \"kas_sub.test2\": \"true\"}"
]
}
So it's not conserving the same JSON format because i would use the json_query filter.
- debug:
msg: "{{ file1.stdout_lines| json_query(value1)}}"
vars:
value1: "#[?keys(#)]"
keys(#)function doesn't return anything
ok: [control] => {
"msg": ""
}
note: taking for granted you want to read a file on the target machine
In a nutshell:
- hosts: your_group
gather_facts: false
vars:
file_to_read: /path/to/file.sub
tasks:
- name: slurp file content from target
slurp:
src: "{{ file_to_read }}"
register: slurped_file
- name: display keys from json inside file
debug:
msg: "{{ (slurped_file.content | b64decode | from_json).keys() }}"
Given the file
shell> cat /tmp/file.sub
{"kas_sub.test1": "true", "kas_sub.test2": "true"}
Use jq (if you can). For example, get the keys
- command: jq 'keys' /tmp/file.sub
register: result
and convert them to a list
keys: "{{ result.stdout|from_yaml }}"
gives
keys:
- kas_sub.test1
- kas_sub.test2
Example of a complete playbook
- hosts: localhost
vars:
keys: "{{ result.stdout|from_yaml }}"
tasks:
- command: jq 'keys' /tmp/file.sub
register: result
- debug:
var: keys

Ansible reading nested json values and matching variable

I am using this in an Ansible playbook:
- name: Gather info from Vcenter
vmware_vm_info:
hostname: "{{ result_item.vcenter }}"
username: "{{ ansible_username }}"
password: "{{ ansible_password }}"
validate_certs: no
register: vminfo
loop: "{{ result.list }}"
loop_control:
loop_var: result_item
I loop through a csv which has a list of VMs and their Vcenters. The json output from the Ansible task is this:
{
"results": [
{
"changed": false,
"virtual_machines": [
{
"guest_name": "Server1",
"guest_fullname": "SUSE Linux Enterprise 11 (64-bit)",
"power_state": "poweredOn",
},
{
"guest_name": "Server2",
"guest_fullname": "FreeBSD Pre-11 versions (64-bit)",
"power_state": "poweredOn",
},
Now I need to query this output for the VMs in my csv (guest_name matches vmname) and use set_fact to indicate whether the VMs in the csv are poweredOff or poweredOn. Next I can use it as a conditional on whether to power off the VM or not based on its current status.
I can't seem to get the json_query to work when matching to the VM name in the csv to the json output and then getting the corresponding power status. Any ideas?
CSV file:
vmname vcenter
Server1 Vcenter1
Server2 Vcenter1
Q: "set_fact to indicate whether the VMs in the CSV are powered off or powered on."
A: For example
- read_csv:
path: servers.csv
dialect: excel-tab
register: result
- set_fact:
servers: "{{ result.list|map(attribute='vmname')|list }}"
- set_fact:
virtual_machines: "{{ virtual_machines|default([]) +
[dict(_servers|zip(_values))] }}"
loop: "{{ vminfo.results }}"
vars:
_servers: "{{ servers|intersect(_dict.keys()|list) }}"
_values: "{{ _servers|map('extract',_dict)|list }}"
_dict: "{{ item.virtual_machines|
items2dict(key_name='guest_name', value_name='power_state') }}"
- debug:
var: virtual_machines
gives
virtual_machines:
- Server1: poweredOn
Server2: poweredOn
Servers missing in the vminfo.results will be silently ignored.
Q: "Use it as a conditional on whether to power off the VM or not."
A: For example Server1 in the first host
- debug:
msg: "Host={{ _host }} VM={{ _vm }} is poweredOn"
when: virtual_machines[_host][_vm] == 'poweredOn'
vars:
_host: 0
_vm: Server1
gives
msg: Host=0 VM=Server1 is poweredOn
I suppose, from your your example that you do have a TSV, so a tab separated values and not a CSV, which stands for comma separated values.
Based on this, the read_csv module, along with the dialect: excel-tab will help you read your TSV.
Then, you will need to use a filter projection to query the JSON based on the data in your TSV file.
You could also need to flatten the projection to get rid of the doubles list created by both the list in results and in virtual_machines.
An example of the resulting JMESPath query, for the Server1 ends up being:
results[].virtual_machines[?
guest_name == `Server1`
]|[]|[0].power_state
Then with all this in a playbook we do end up with:
- hosts: localhost
gather_facts: no
tasks:
- read_csv:
path: servers.csv
dialect: excel-tab
register: servers
- debug:
msg: >-
For {{ item.vmname }}, the state is {{
vminfo |
json_query(
'results[].virtual_machines[?
guest_name == `' ~ item.vmname ~ '`
]|[]|[0].power_state'
)
}}
loop: "{{ servers.list }}"
loop_control:
label: "{{ item.vmname }}"
vars:
vminfo:
results:
- changed: false
virtual_machines:
- guest_name: Server1
guest_fullname: SUSE Linux Enterprise 11 (64-bit)
power_state: poweredOn
- guest_name: Server2
guest_fullname: FreeBSD Pre-11 versions (64-bit)
power_state: poweredOn
Which yields the recap:
PLAY [localhost] **************************************************************************************************
TASK [read_csv] ***************************************************************************************************
ok: [localhost]
TASK [debug] ******************************************************************************************************
ok: [localhost] => (item=Server1) =>
msg: For Server1, the state is poweredOn
ok: [localhost] => (item=Server2) =>
msg: For Server2, the state is poweredOn
PLAY RECAP ********************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Why is Ansible unable to read unicode string as JSON?

Summary
When retrieving data using the uri module in Ansible, I am unable to parse a section of it as JSON to retrieve a nested value.
The desired value is the ci field inside the content.data or json.data field (see output below).
Steps to Reproduce
site.yml
---
- hosts: localhost
gather_facts: false
tasks:
- name: Get String
uri:
url: "http://localhost/get-data"
method: POST
body_format: json
body: "{ \"kong-jid\": \"run-sn-discovery\" }"
return_content: yes
register: output
- set_fact:
ci: "{{ output.json.data.ci }}"
- debug:
msg: "{{ ci }}"
The {{ output }} variable
{
u'status': 200,
u'cookies': {},
u'url': u'http://kong-demo:8000/get-data',
u'transfer_encoding': u'chunked',
u'changed': False,
u'connection': u'close',
u'server': u'kong/0.34-1-enterprise-edition',
u'content':
u'{"data":"\\"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}\\""}',
'failed': False,
u'json': {u'data': u'"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}"'},
u'content_type': u'application/json',
u'date': u'Thu, 18 Apr 2019 15:50:25 GMT',
u'redirected': False,
u'msg': u'OK (unknown bytes)'
}
Result
[user#localhost]$ ansible-playbook site.yml
[WARNING]: Could not match supplied host pattern, ignoring: all
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [localhost] ***************************************************************************************************************
TASK [Pass Redis data to next task as output] **********************************************************************************
ok: [localhost]
TASK [set_fact] ****************************************************************************************************************
fatal: [localhost]: FAILED! => {}
MSG:
The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
The error appears to have been in 'site.yml': line 19, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- set_fact:
^ here
exception type: <class 'ansible.errors.AnsibleUndefinedVariable'>
exception: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
Important Troubleshooting Information
It appears the root issue is related to which Ansible type being interpreted. I desire to parse ci from the output in one task.
The two-task solution shown below works, but this leads me to believe this should be possible in one line...
Two-Task Solution
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci['ci'] }}"
But the ci fact set from {{ output.json.data | from_json }} reports a different TYPE than the inline type...
Unicode or Dict?
- debug:
msg: "{{ output.json.data | from_json | type_debug }}" # returns unicode
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci | type_debug }}" # returns dict
Why isn't {{ output.json.data | from_json | type_debug }}
the same as {{ ci | type_debug }}?
Although json and data are keys in their resp objects, ci is just part of a larger string (which happens to look like a JSON object
If the relevant line in your datastructure would be:
u'json': {u'data': {'ci': u'3bb8d625dbac3700e4f07b6e0f96195b'}},
then you could expect to use "{{ output.json.data.ci }}" but not when the .ci part is just a normal part of a string.

Ansible "set_fact" repository url from json file using filters like "from_json"

Using Ansible "set_fact" module, I need to get repository url from json file using filters like "from_json". I tried in couple ways, and still doesn't get it how is should work.
- name: initial validation
tags: bundle
hosts: localhost
connection: local
tasks:
- name: register bundle version_file
include_vars:
file: '/ansible/playbook/workbench-bundle/bundle.json'
register: bundle
- name: debug registered bundle file
debug:
msg: '{{ bundle }}'
I get json that I wanted:
TASK [debug registered bundle file] ************************************************
ok: [127.0.0.1] => {
"msg": {
"ansible_facts": {
"engine-config": "git#bitbucket.org/engine-config.git",
"engine-monitor": "git#bitbucket.org/engine-monitor.git",
"engine-server": "git#bitbucket.org/engine-server.git",
"engine-worker": "git#bitbucket.org/engine-worker.git"
},
"changed": false
}
}
And then I'm trying to select each value by key name to use this value as URL to "npm install" each package in separate instances.
- name: set_fact some paramater
set_fact:
engine_url: "{{ bundle.('engine-server') | from_json }}"
And then I get error:
fatal: [127.0.0.1]: FAILED! => {"failed": true, "msg": "template error
while templating string: expected name or number. String: {{
bundle.('engine-server') }}"}
I many others ways like this loopkup, and it still fails with others errors. Can someone help to understand, how I can find each parameter and store him as "set_fact"? Thanks
Here is a sample working code to set a variable like in the question (although I don't see much sense in it):
- name: initial validation
tags: bundle
hosts: localhost
connection: local
tasks:
- name: register bundle version_file
include_vars:
file: '/ansible/playbook/workbench-bundle/bundle.json'
name: bundle
- debug:
var: bundle
- debug:
var: bundle['engine-server']
- name: set_fact some paramater
set_fact:
engine_url: "{{ bundle['engine-server'] }}"
The above assumes your input data (which you did not include) is:
{
"engine-config": "git#bitbucket.org/engine-config.git",
"engine-monitor": "git#bitbucket.org/engine-monitor.git",
"engine-server": "git#bitbucket.org/engine-server.git",
"engine-worker": "git#bitbucket.org/engine-worker.git"
}

Ansible: Remove whitespaces in json file

I have a json file content loaded in my ansible variables.
The json content (and the resulting file I write from it) has unnecessary file spaces and blank lines. I want to minify the json file by removing all that unnecessary stuff.
Is it possible to do something like {{ myjson_content| to_json_minify }} ?
Maybe this can be done through a regex ?
You can read it from json and convert back into json with separators option.
{{ my_json_content | from_json | to_json(separators=(',',':')) }}
Note: this is not documented, but if you look at the source code you will see that the filter accepts arbitrary keywords args which are later passed to the python json.dumps function. So you can basically pass to to_json any parameter accepted by json.dumps.
playbook.yml
---
- hosts: localhost
vars:
my_json_content:
'
{ "a" : 0,
"b": 1,
"c": 2}
'
tasks:
- debug:
msg: "json = {{ my_json_content }}"
- debug:
msg: "minified_json = {{ my_json_content | from_json | to_json(separators=(',',':')) }}"
$ ansible-playbook playbook.yml
TASK [debug] *******************************************************************
ok: [localhost] => {
"msg": "json = { \"a\" : 0,\n\"b\": 1,\n\"c\": 2}\n"
}
TASK [debug] *******************************************************************
ok: [localhost] => {
"msg": "minified_json = {\"a\":0,\"c\":2,\"b\":1}"
}