Ansible | Process data which can be either JSON or YAML - json

I'm using Ansible to read a config, which can be either JSON or YAML and extract values from some of the nodes in the file.
I know I can use from_json or from_yaml to process it in Ansible, but since I don't know which format the config will be in, I'm having difficulty making it work.
The file is Kubernetes' Kubeconfig. Examples below:
in YAML
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://my-k8s-cluster.com
name: k8s-clstr-master
contexts:
- context:
cluster: k8s-clstr-master
namespace: kube-system
user: k8s-clstr-master-admin
name: k8s-clstr-master
current-context: k8s-clstr-master
kind: Config
preferences: {}
users:
- name: k8s-clstr-master-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
in JSON
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "k8s-clstr-master",
"cluster": {
"server": "https://my-k8s-cluster.com",
"certificate-authority-data": "REDACTED"
}
}
],
"users": [
{
"name": "k8s-clstr-master-admin",
"user": {
"client-certificate-data": "REDACTED",
"client-key-data": "REDACTED"
}
}
],
"contexts": [
{
"name": "k8s-clstr-master",
"context": {
"cluster": "k8s-clstr-master",
"user": "k8s-clstr-master-admin",
"namespace": "kube-system"
}
}
],
"current-context": "k8s-clstr-master"
}
Ansible I'm using:
vars:
kubeconfig: "{{ lookup('hashivault', '/kubeconfig/admin', 'config') }}"
tasks:
- name: Find cluster server name
shell: "echo {{ kubeconfig.clusters[0].cluster.server }}"
Above Ansible block will work okay if kubeconfig is retrieved in JSON format, but it will fail if it's retrieved as in YAML format.
I might be able to make a task with |from yaml and then add ignore_errors: true, but that just doesn't feel like right way of doing it.
Anyone has any tips for me on how I can approach this problem?

There are some built-in tests in Jinja2.
The way Ansible templator works if you have JSON string inside {{...}} expression, it is automatically converted to object. So if you fetch JSON from your vault, kubeconfig becomes object, otherwise it is a string.
Here's a recipe for you:
vars:
kubeconfig_raw: "{{ lookup('hashivault', '/kubeconfig/admin', 'config') }}"
kubeconfig: "{{ kubeconfig_raw if kubeconfig_raw is mapping else kubeconfig_raw | from_yaml }}"
tasks:
- name: Find cluster server name
shell: "echo {{ kubeconfig.clusters[0].cluster.server }}"

If you use the include_vars task, it does not matter which format you provide. The task accepts both.
---
- hosts: localhost
connection: local
tasks:
- include_vars:
file: config
name: kubeconfig
- debug: var=kubeconfig

Related

ansible obtain values from changing key

I would like to get value from json, but one of the key can be differet.
here is example json
{
"json": {
"id": "9758b1e5-442e-4545-9364-45f28477edfb",
"results": [{
"code": 200,
"host": "localhost",
"message": "no change",
"runTime": 1233,
"tenant": "http-validate-2.usa-dc.com"
}],
"traces": {
"http-validate-2.usa-dc.comCurrent": {
"/Common/10.10.100.10": {
"command": "ltm node"
},
"http-validate-2.usa-dc.comDiff": [{
"command": "ltm virtual",
"kind": "D",
"lhs": {
"default": "yes"
},
"lhsCommand": "ltm virtual",
"path": [
"/http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80",
"properties",
"persist",
"/Common/cookie"
],
"rhsCommand": "ltm virtual",
"tags": [
"tmsh"
]
}]
}
}
}
}
my ansible playbook
tasks:
- name : deploy json file AS3 to F5
debug:
msg: "{{ lookup('file', 'parse.json') }}"
register: atc_AS3_status
- name: debug
debug:
msg: "{{ atc_AS3_status.msg.json['traces']['.*Diff']}}"
I would like to reach key "path" but the key above "http-validate-2.usa-dc.comDiff" can be different like "http-validate-3.can-dc.comDiff" but always finish with Diff
Use json_query, e.g.
- debug:
msg: "{{ json.traces|json_query('*.*[][].path') }}"
should give the list of the paths (there might be more of them)
msg:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie
Q: "I need path only in the key which ends Diff."
A: JMESPath is not able to search key wildcards, AFAIK. Instead, use select and create the list of the nested keys that match the regex, e.g.
- debug:
msg: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
gives
msg:
- http-validate-2.usa-dc.comDiff
Then iterate this list, select path and concatenate the list paths, e.g.
- set_fact:
paths: "{{ paths|d([]) + json.traces|json_query(query) }}"
loop: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
vars:
query: '*."{{ item }}"[].path'
gives the list of paths for the keys that match the regex
paths:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie

Parsing json output retrieved from an API using Ansbile

I am new to Ansible. I am trying to fetch some API info from a centralized server in the json format, parse the json and set "certificate" to "yes" for each and every server and make a PUT operation back. GET and PUT operations in my code are working fine.
I am not able to find correct syntax to parse listofservers.json.
What is the syntax to display a specific json element??
Can you help me on how to parse the json to change "Certificate" value to Yes for all the servers?
My script
- hosts: serverlist
vars_prompt:
- name: "USER"
prompt: "Enter Admin Username"
private: no
- name: "PASS"
prompt: "Enter Admin Password"
private: yes
tasks:
- name: "Get the current values of all Servers"
delegate_to: localhost
register: listofservers
check_mode: no
uri:
validate_certs: "no"
user: "{{USER}}"
password: "{{PASS}}"
url: "https://console.exmaple.com/adminapi/serverlist"
method: "GET"
body_format: "json"
return_content: "no"
status_code: "200"
- name: Print json dictionary
debug:
var: listofservers.json
Output
TASK [Print json dictionary] *****************************************************************************************
ok: [centalServer.example.com] =>
{
"listofservers.json": {
"serverlist": [
{
"id": 1,
"servername": "redhat.example.com",
"apachestatus": "running",
"certificate": "no"
},
{
"id": 2,
"servername": "solaris.example.com",
"apachestatus": "down",
"certificate": "yes"
} ] } }
Q: "Change "certificate" value to "yes" for all servers."
A: The task below does the job
- set_fact:
listofservers: "{{ {'json':
{'serverlist': serverlist}} }}"
vars:
serverlist: "{{ listofservers.json.serverlist|
map('combine', {'certificate': 'yes'})|
list }}"
- debug:
var: listofservers.json
give
{
"listofservers.json": {
"serverlist": [
{
"apachestatus": "running",
"certificate": "yes",
"id": 1,
"servername": "redhat.example.com"
},
{
"apachestatus": "down",
"certificate": "yes",
"id": 2,
"servername": "solaris.example.com"
}
]
}
}
Q: "What is the syntax to display a specific JSON element?"
A: There are two main options 1) Ansible filters and 2) json_query, and many variations which depend on the structure of the data and requested query
For example, display first servername where id is equal to 1
- debug:
msg: "{{ listofservers.json.serverlist|
selectattr('id', 'eq', 1)|
map(attribute='servername')|
first }}"
gives
"msg": "redhat.example.com"
The same result will give json_query below
- debug:
msg: "{{ listofservers.json.serverlist|
json_query('[?id == `1`].servername')|
first }}"

Ansible pretty print json

when I'm running my playbook I get the debug output in the correct json format I would want it
"ansible_facts": {
"routes": [
{
"subnet": "10.0.0.0/24"
},
{
"subnet": "10.0.1.0/24"
}
]
},
but when I export it to a file using the local_action directive it displays like this
[{"subnet": "10.0.0.0/24"}, {"subnet": "10.0.1.0/24"}
Is there any working pretty print module or in Ansible which would export my file in the same way as I see it in the debug messages?
Thanks!
Not exactly the output you get on screen, but you could use a template task to print the variable, after passing it through a to_nice_json filter. Example:
---
- hosts: localhost
gather_facts: false
vars:
my_ansible_facts:
routes:
- subnet: 10.0.0.0/24
- subnet: 10.0.1.0/24
tasks:
- template: src=nice_yaml_filter.j2 dest=/tmp/nice_yaml_filter.out
Please note that i am not using the same variable as you are, just a my_ansible_facts variable that i populated.
And the template file's contents, nice_yaml_filter.j2:
{{ my_ansible_facts | to_nice_json }}
Result:
[http_offline#greenhat-32 ANSIBLE_TESTS]$ cat /tmp/nice_yaml_filter.out
{
"routes": [
{
"subnet": "10.0.0.0/24"
},
{
"subnet": "10.0.1.0/24"
}
]
}[http_offline#greenhat-32 ANSIBLE_TESTS]$
cheers

best way to modify json in ansible

I have a variable (via set_fact) containing a json string:
{
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
What is the best way to insert more elements in the "Resource" array?
"arn:aws:iam::001122334455:role/Member1_Create_svcacct",
"arn:aws:iam::001122334455:role/Member1_Update_svcacct",
"arn:aws:iam::001122334455:role/Member1_Delete_svcacct",
"arn:aws:iam::001122334455:role/Member1_Full_svcacct"
I am exploring dumping the variable to a file and inserting the block I want with external shell tools, which does not seem to be elegant.
I don't know about the best way, but one option is to write a simple library module to handle the mechanics of the update for you. You could use the jsonpointer module as a way of locating the data you wish to modify, and then return the modified object to ansible. A starting point might look like:
#!/usr/bin/python
from ansible.module_utils.basic import AnsibleModule
import json
try:
import jsonpointer
except ImportError:
jsonpointer = None
def main():
module = AnsibleModule(
argument_spec=dict(
data=dict(required=True, type='dict'),
pointer=dict(required=True),
action=dict(required=True,
choices=['append', 'extend', 'update']),
update=dict(type='dict'),
extend=dict(type='list'),
append=dict(),
),
supports_check_mode=True,
)
if jsonpointer is None:
module.fail_json(msg='jsonpointer module is not available')
action = module.params['action']
data = module.params['data']
pointer = module.params['pointer']
if isinstance(data, str):
data = json.loads(str)
try:
res = jsonpointer.resolve_pointer(data, pointer)
except jsonpointer.JsonPointerException as err:
module.fail_json(msg=str(err))
if action == 'append':
res.append(module.params['append'])
if action == 'extend':
res.extend(module.params['extend'])
elif action == 'update':
res.update(module.params['update'])
module.exit_json(changed=True,
result=data)
if __name__ == '__main__':
main()
If you drop this into, e.g., library/json_modify.py, you can use it in a playbook like this:
- hosts: localhost
gather_facts: false
vars:
myvar: {
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
tasks:
- json_modify:
data: "{{ myvar }}"
pointer: "/PolicyVersion/Document/Statement/0/Resource"
action: extend
extend:
- "arn:aws:iam::001122334455:role/Member1_Create_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Update_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Delete_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Full_svcacct"
register: result
- debug:
var: result.result
The result of running this playbook and the proposed module is:
TASK [debug] *******************************************************************
ok: [localhost] => {
"result.result": {
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct",
"arn:aws:iam::001122334455:role/Member1_Create_svcacct",
"arn:aws:iam::001122334455:role/Member1_Update_svcacct",
"arn:aws:iam::001122334455:role/Member1_Delete_svcacct",
"arn:aws:iam::001122334455:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
}
Actually Ansible is natively able to read JSON files.
see this question:
add-a-new-key-value-to-a-json-file-using-ansible
This is a little bit old. I know but I felt it could be useful to many people so I'll post here my solution as "the best way to modify a json" that I could personally find.
First of all the JSON. In my use-case I had a bunch of aws snapshots that were encrypted with the wrong kms key and I had to recreate the AMI with the correct key.
So I had to :
get data of the old snapshots ( like size device_name etc.)
create the new snaps with the different key
re-register a new ami with the correct block_device_mapping
Here's the code
- name: get ami
amazon.aws.ec2_ami_info:
image_ids: ami-<id>
region: "{{ region }}"
register: ami
- name: save snapshot ids and device_name and volume_size
set_fact:
snapshot_ids: "{{ ami | json_query('images[].block_device_mappings[].ebs.snapshot_id') }}"
device_name: "{{ ami | json_query('images[].block_device_mappings[].device_name') }}"
volume_size: "{{ ami | json_query('images[].block_device_mappings[].ebs.volume_size') }}"
basically each of the above is a list of each of the 3 things (device_name, snap_id, volume_size) that I need (but it could be extended)
then:
- name: get kms arn
aws_kms_info:
filters:
alias: "{{ kms_keys.alias }}"
region: "{{ region }}"
register: aws_kms_facts_out
- debug:
var: aws_kms_facts_out
- set_fact:
kms_arn: "{{ aws_kms_facts_out['keys'][0].key_arn }}"
- name: copy snap with new encryption key
community.aws.ec2_snapshot_copy:
source_region: "{{ region }}"
region: "{{ region }}"
source_snapshot_id: "{{ item }}"
encrypted: yes
kms_key_id: "{{ kms_arn }}"
loop: "{{ snapshot_ids }}"
register: new_snapshots
and then here's the catch
- set_fact:
new_snapshot_ids: "{{ new_snapshots| json_query('snapshots[].snapshot_id') }}"
- name: creating the block_device_mappings structure (still a list of objects)
ansible.builtin.debug:
msg: '{
"device_name": "{{ item.2 }}",
"delete_on_termination": "true",
"snapshot_id": "{{ item.0 }}",
"volume_size": {{ item.1 }},
"volume_type": "gp2"
}'
loop: "{{ new_snapshot_ids|zip_longest(volume_size,device_name)|list }}"
register: block_device_mappings
- set_fact:
bdm: "{{ block_device_mappings | json_query('results[].msg') }}"
finally
- name: create new ami from newly created snapshots
amazon.aws.ec2_ami:
region: "{{ region }}"
name: "{{ instance.name }}-{{ ansible_date_time.date }}"
state: present
architecture: x86_64
virtualization_type: hvm
root_device_name: /dev/sda1
device_mapping: "{{ bdm }}"
This is how you can do it without requiring any additional trick.
Of course this is declined to my particular use case but you can adapt it to any circumstance, that do not require a complete disassemble and reassemble of the Json itself

ansible parse json with several keys with the same name to one list variable

I have an issue to parse a json using ansible
I have a task that connected to rancher and get a json file
task:
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: test
set_fact:
rancher_env_hosts: "{{ item.hostname }}"
#when: item.hostname == "*-i-*"
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
and I get the following json (after edit it to be more readable):
{
"json": {
"data": [
{
"hostname": "rancher-i-host-02",
"id": "adsfsa"
},
{
"hostname": "rancher-i-host-01",
"id": "gfdgfdg"
},
{
"hostname": "rancher-q-host-01",
"id": "dfgdg"
},
{
"hostname": "rancher-q-host-02",
"id": "dfgdg"
}
]
}
}
When I start the playbook I get only the last host name in the variable and not all the list of hostname. can I register all the list to the same variable?
In addition, I also added a line with the a comment "#" in order to get only the host names that match the string "-i-" bit it's not working. any idea?
This is what filters (and this) for:
- set_fact:
hosts_all: "{{ hosts_json.json.data | map(attribute='hostname') | list }}"
hosts_i: "{{ hosts_json.json.data | map(attribute='hostname') | map('regex_search','.*-i-.*') | select('string') | list }}"
host_all will contain all hostnames, host_i will contain only .*-i-.* matching hostnames.
Try this
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: init fact
set_fact:
rancher_env_hosts: "[]"
- name: test
set_fact:
rancher_env_hosts: "{{rancher_env_hosts}} + [ {{item.hostname}} ]"
when: item.hostname | search(".*-i-.*")
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
About search you can read here http://docs.ansible.com/ansible/playbooks_tests.html
UPD:
About adding values to array here: Is it possible to set a fact of an array in Ansible?