Ansible: extracting a dictionary from a complex JSON data structure - json

{
"vmware_dc": {
"changed": false,
"msg": "All items completed",
"results": [
{
"ansible_loop_var": "item",
"changed": false,
"datacenter_info": [
{
"config_status": "gray",
"moid": "datacenter-1146",
"name": "dc-1",
"overall_status": "gray"
}
],
"failed": false,
"invocation": {
"module_args": {
"datacenter": null,
"hostname": "vc-001",
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 443,
"properties": null,
"proxy_host": null,
"proxy_port": null,
"schema": "summary",
"show_tag": false,
"username": "",
"validate_certs": false
}
},
"item": "vc-001"
},
{
"ansible_loop_var": "item",
"changed": false,
"datacenter_info": [
{
"config_status": "gray",
"moid": "datacenter-424",
"name": "dc-2",
"overall_status": "gray"
},
{
"config_status": "gray",
"moid": "datacenter-2",
"name": "dc-3",
"overall_status": "gray"
}
],
"failed": false,
"invocation": {
"module_args": {
"datacenter": null,
"hostname": "vc-002",
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 443,
"properties": null,
"proxy_host": null,
"proxy_port": null,
"schema": "summary",
"show_tag": false,
"username": "",
"validate_certs": false
}
},
"item": "vc-002"
}
]
}
}
Please see JSON results above. I hope that someone can help me. I should take two fields, an item and datacenter_info.name
I can get values separate. I don't know how to create a dictionary because I have 2 items and 3 datacenter_info.name
enter - name: Set all_items
set_fact:
all_items: "{{ vmware_dc_info.results | flatten | map(attribute='item) | flatten }}"
- name: debug all_items
debug:
var: all_items
{
"all_items": [
"vc-001",
"vc-002"
]
- name: Set all dc
set_fact:
all_dc: "{{ vmware_dc_info.results | flatten | map(attribute='datacenter_info') | flatten | map(attribute='name') | flatten }}"
- name: debug all_dc
debug:
var: all_dc
"all_dc": [
"dc-1",
"dc-2",
"dc-3"
Dictionary should looks like below
vc-001: dc-1
vc-002: dc-2
vc-002: dc-3
Is it possible? What do you think?
Resolved with code below
- name: Set fact
set_fact:
dc_list: "{{ dc_list | default([]) + [{'vcenter': item[0].item, 'dc': item[1].name}] }}"
with_subelements:
- "{{ vmware_dc_info.results }}"
- datacenter_info
loop_control:
label: "{{ item[0].item }}"
- name: Debug
debug:
var: dc_list

For example
- set_fact:
hostnames: "{{ hostnames|d({})|combine({item: names}) }}"
loop: "{{ vmware_dc.results|map(attribute='item')|list }}"
vars:
_item: "{{ vmware_dc.results|selectattr('item', '==', item) }}"
names: "{{ _item.0.datacenter_info|map(attribute='name')|list }}"
gives you the expected result
hostnames:
vc-001:
- dc-1
vc-002:
- dc-2
- dc-3

Related

Need to replace value in dictionary in destination file with the value in the dictionary in source file

I have been trying to update the values in a dictionary in destination json with the values in the dictionary in source JSON file. Below is the example of source and destination JSON file:
Source file:
[
{
"key": "MYSQL",
"value": "456"
},
{
"key": "RDS",
"value": "123"
}
]
Destination File:
[
{
"key": "MYSQL",
"value": "100"
},
{
"key": "RDS",
"value": "111"
},
{
"key": "DB1",
"value": "TestDB"
},
{
"key": "OS",
"value": "EX1"
}
]
Expectation in destination file after running Ansible playbook:
[
{
"key": "MYSQL",
"value": "**456**"
},
{
"key": "RDS",
"value": "**123**"
},
{
"key": "DB1",
"value": "TestDB"
},
{
"key": "OS",
"value": "EX1"
}
]
Below is the playbook I have tried so far, but this only updates the value if it is hard coded:
- hosts: localhost
tasks:
- name: Parse JSON
shell: cat Source.json
register: result
- name: Save json data to a variable
set_fact:
jsondata: "{{result.stdout | from_json}}"
- name: Get key names
set_fact:
json_key: "{{ jsondata | map(attribute='key') | flatten }}"
- name: Get Values names
set_fact:
json_value: "{{ jsondata | map(attribute='value') | flatten }}"
# Trying to update the destination file with only the values provided in source.json
- name: Replace values in json
replace:
path: Destination.json
regexp: '"{{ item }}": "100"'
replace: '"{{ item }}": "456"'
loop:
- value
The main goal is to update the value in destination.json with the value provided in source.json.
In Ansible, the couple key/value tends to be handled with the filters dict2items and items2dict. And your use case can be handled by those filters.
Here would be the logic:
Read both files
Convert both files into dictionaries, with dict2items
Combine the two dictionaries, with the combine filter
Convert the dictionary back into a list with items2dict
Dump the result in JSON back into the file
Given the playbook:
- hosts: localhost
gather_facts: no
tasks:
- shell: cat Source.json
register: source
- shell: cat Destination.json
register: destination
- copy:
content: "{{
destination.stdout | from_json | items2dict |
combine(
source.stdout | from_json | items2dict
) | dict2items | to_nice_json
}}"
dest: Destination.json
We end up with Destination.json containing:
[
{
"key": "MYSQL",
"value": "456"
},
{
"key": "RDS",
"value": "123"
},
{
"key": "DB1",
"value": "TestDB"
},
{
"key": "OS",
"value": "EX1"
}
]
Without to knowing the structure of your destination file it's difficult to use a regex.
I suggest you to load your destination file in a variable, do the changes and save the content of variable to file.
This solution does the job:
- hosts: localhost
tasks:
- name: Parse JSON
set_fact:
source: "{{ lookup('file', 'source.json') | from_json }}"
destination: "{{ lookup('file', 'destination.json') | from_json }}"
- name: create new json
set_fact:
json_new: "{{ json_new | d([]) + ([item] if _rec == [] else [_rec]) | flatten }}"
loop: "{{ destination }}"
vars:
_rec: "{{ source | selectattr('key', 'equalto', item.key) }}"
- name: save new json
copy:
content: "{{ json_new | to_nice_json }}"
dest: dest_new.json
Result -> dest_new.json:
ok: [localhost] => {
"msg": [
{
"key": "MYSQL",
"value": "456"
},
{
"key": "RDS",
"value": "123"
},
{
"key": "DB1",
"value": "TestDB"
},
{
"key": "OS",
"value": "EX1"
}
]
}

How to read a value from json output

Basically, I need to read a value from a JSON output and use it in subsequent tasks. So, I tried with_items, loop, but nothing worked.
Sample JSON that I generated from a registered variable:
TASK [local_volume_mount : debug Info from device that is parted] **************
Monday 29 March 2021 21:33:39 +0000 (0:00:02.271) 0:00:02.417 **********
ok: [node1] => {
"partitioned_device_live_info": {
"changed": false,
"msg": "All items completed",
"results": [
{
"ansible_loop_var": "item",
"changed": false,
"disk": {
"dev": "/dev/nvme2n1",
"logical_block": 512,
"model": "SAMSUNG MZQLW960HMJP-00003",
"physical_block": 512,
"size": 937692504.0,
"table": "msdos",
"unit": "kib"
},
"failed": false,
"invocation": {
"module_args": {
"align": "optimal",
"device": "/dev/nvme2n1",
"flags": null,
"label": "msdos",
"name": null,
"number": 1,
"part_end": "100%",
"part_start": "0%",
"part_type": "primary",
"state": "present",
"unit": "KiB"
}
},
"item": [
{
"device": "/dev/nvme2n1",
"partitions": [
{
"end": "100%",
"number": 1,
"start": "0%",
"storage_class": "ssd-wkr-services"
}
]
},
{
"end": "100%",
"number": 1,
"start": "0%",
"storage_class": "ssd-wkr-services"
}
],
"partitions": [
{
"begin": 1024.0,
"end": 937692160.0,
"flags": [],
"fstype": "",
"name": "",
"num": 1,
"size": 937691136.0,
"unit": "kib"
}
],
"script": ""
}
]
}
}
So, from the above I need to read the partitions.num value and use it in the next task, but, I don't know how to do that.
Task:
- name: THIS IS BEING TESTED
debug:
var: "{{ item.partitions }}"
ignore_errors: no
#loop: "{{ partitioned_device_live_info.results }}"
with_items: "{{ partitioned_device_live_info.results }}"
Output of the above task is
'dict object' has no attribute 'partitions'
I want to store that value item.partitions.num in a variable and then use it in further tasks.
Figured out using msg, below is my change
- name: THIS IS BEING TESTED with MSG and with_items
debug:
msg: "{{ item.partitions[0].num }}"
with_items: "{{ partitioned_device_live_info.results }}"
You can always use regex in ansible to set facts. More details stackoverflow
Following are some use cases
- name: Long form task does not
ansible.builtin.replace:
path: /etc/hosts
regexp: '\b(localhost)(\d*)\b'
replace: '\1\2.localdomain\2 \1\2'
- name: Explicitly specifying positional matched groups in replacement
ansible.builtin.replace:
path: /etc/ssh/sshd_config
regexp: '^(ListenAddress[ ]+)[^\n]+$'
replace: '\g<1>0.0.0.0'
Setting fact
- name: set version in file after replacement
set_fact:
version_in_file: "{{ version_deployment_file | regex_search(docker_image_version) }}"
More details - ansible.builtin.replace

Filtering ansible output

I'm trying to filter a list of attributes from the below output. selection and value attributes need to be listed.
Output:
ok: [192.0.0.3] => {
"allow": "GET, PUT",
"attempts": 1,
"invocation": {
}
},
"json": {
"body": [
{
"selection": "fqdn",
"value": "airbus3.org"
},
{
"selection": "fqdn",
"value": "airbus4.org"
}
],
"key": "ntp_servers",
"meta": {
"transaction": "/api/transaction"
}
},
}
I'm trying to use the below task to filter but it's not working.
- set_fact:
ntp_details: "{{ item.selection }}"
with_items: "{{ reg_ntp.json.body | json_query('[*]') }}"
Please advise..
For example
- set_fact:
ntp_details: "{{ reg_ntp.json.body|json_query('[].selection') }}"
should give
ntp_details:
- fqdn
- fqdn

Extract value from Ansible task output and create a variable from it

I use elb_application_lb_info module to get info about my application load balancer. Here is the code I am using for it:
- name: Test playbook
hosts: tag_elastic_role_logstash
vars:
aws_access_key: AKIARWXXVHXJS5BOIQ6P
aws_secret_key: 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
tasks:
- name: Gather information about all ELBs
elb_application_lb_info:
aws_access_key: AKIXXXXXXXXXXXXXXXXXXX
aws_secret_key: gG6a586XXXXXXXXXXXXXXXXXX
region: ap-southeast-2
names:
- LoadBalancer
register: albinfo
- debug:
msg: "{{ albinfo }}"
This is working fine and I got the following output:
"load_balancers": [
{
"idle_timeout_timeout_seconds": "60",
"routing_http2_enabled": "true",
"created_time": "2021-01-26T23:58:27.890000+00:00",
"access_logs_s3_prefix": "",
"security_groups": [
"sg-094c894246db1bd92"
],
"waf_fail_open_enabled": "false",
"availability_zones": [
{
"subnet_id": "subnet-0195c9c0df024d221",
"zone_name": "ap-southeast-2b",
"load_balancer_addresses": []
},
{
"subnet_id": "subnet-071060fde585476e0",
"zone_name": "ap-southeast-2c",
"load_balancer_addresses": []
},
{
"subnet_id": "subnet-0d5f856afab8f0eec",
"zone_name": "ap-southeast-2a",
"load_balancer_addresses": []
}
],
"access_logs_s3_bucket": "",
"deletion_protection_enabled": "false",
"load_balancer_name": "LoadBalancer",
"state": {
"code": "active"
},
"scheme": "internet-facing",
"type": "application",
"load_balancer_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:loadbalancer/app/LoadBalancer/27cfc970d48501fd",
"access_logs_s3_enabled": "false",
"tags": {
"Name": "loadbalancer_test",
"srg:function": "Storage",
"srg:owner": "ISCloudPlatforms#superretailgroup.com",
"srg:cost-centre": "G110",
"srg:managed-by": "ISCloudPlatforms#superretailgroup.com",
"srg:environment": "TST"
},
"routing_http_desync_mitigation_mode": "defensive",
"canonical_hosted_zone_id": "Z1GM3OXH4ZPM65",
"dns_name": "LoadBalancer-203283612.ap-southeast-2.elb.amazonaws.com",
"ip_address_type": "ipv4",
"listeners": [
{
"default_actions": [
{
"target_group_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:targetgroup/test-ALBID-W04X8DBT450Q/c999ac1cda7b1d4a",
"type": "forward",
"forward_config": {
"target_group_stickiness_config": {
"enabled": false
},
"target_groups": [
{
"target_group_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:targetgroup/test-ALBID-W04X8DBT450Q/c999ac1cda7b1d4a",
"weight": 1
}
]
}
}
],
"protocol": "HTTP",
"rules": [
{
"priority": "default",
"is_default": true,
"rule_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:listener-rule/app/LoadBalancer/27cfc970d48501fd/671ad3428c35c834/5b5953a49a886c03",
"conditions": [],
"actions": [
{
"target_group_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:targetgroup/test-ALBID-W04X8DBT450Q/c999ac1cda7b1d4a",
"type": "forward",
"forward_config": {
"target_group_stickiness_config": {
"enabled": false
},
"target_groups": [
{
"target_group_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:targetgroup/test-ALBID-W04X8DBT450Q/c999ac1cda7b1d4a",
"weight": 1
}
]
}
}
]
}
],
"listener_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:listener/app/LoadBalancer/27cfc970d48501fd/671ad3428c35c834",
"load_balancer_arn": "arn:aws:elasticloadbalancing:ap-southeast-2:117557247443:loadbalancer/app/LoadBalancer/27cfc970d48501fd",
"port": 9200
}
],
"vpc_id": "vpc-0016dcdf5abe4fef0",
"routing_http_drop_invalid_header_fields_enabled": "false"
}
]
I need to fetch "dns_name" which is dns name of the load balancer and pass it in another play as a variable.
I tried with json_query but got the error. Here is the code:
- name: save the Json data to a Variable as a Fact
set_fact:
jsondata: "{{ albinfo.stdout | from_json }}"
- name: Get ALB dns name
set_fact:
dns_name: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'load_balancers.dns_name'
- debug:
msg: "{{ dns_name }}"
And here is the error:
"msg": "The task includes an option with an undefined variable. The error was: Unable to look up a name or access an attribute in template string ({{ albinfo.stdout | from_json }}).\nMake sure your variable name does not contain invalid characters like '-': the JSON object must be str, bytes or bytearray
Any idea how to extract "dns_name" from the json above?
Here is the way to get the dns_name from above json output:
- name: Get Application Load Balancer DNS Name
set_fact:
rezultat: "{{ albinfo | json_query('load_balancers[*].dns_name') }}"
- debug:
msg: "{{ rezultat }}"

Ansible: get specific attribute value from json output

I have the following Ansible task:
tasks:
- name: ensure instances are running
ec2:
aws_access_key: "{{aws_access_key}}"
aws_secret_key: "{{aws_secret_key}}"
...
user_data: "{{ lookup('template', 'userdata.txt.j2') }}"
register: ec2_result
- debug:
msg: "{{ ec2_result }}"
- set_fact:
win_instance_id: "{{ ec2_result | json_query('tagged_instances[*].id') }}"
The output:
TASK [debug] ***************
ok: [localhost] => {
"msg": {
"changed": false,
"failed": false,
"instance_ids": null,
"instances": [],
"tagged_instances": [
{
"ami_launch_index": "0",
"architecture": "x86_64",
"block_device_mapping": {
"/dev/sda1": {
"delete_on_termination": true,
"status": "attached",
"volume_id": "vol-01f217e489c681211"
}
},
"dns_name": "",
"ebs_optimized": false,
"groups": {
"sg-c63822ac": "WinRM RDP"
},
"hypervisor": "xen",
"id": "i-019c03c3e3929f76e",
"image_id": "ami-3204995d",
...
"tags": {
"Name": "Student01 _ Jumphost"
},
"tenancy": "default",
"virtualization_type": "hvm"
}
]
}
}
TASK [set_fact] ****************
ok: [localhost]
TASK [debug] ******************
ok: [localhost] => {
"msg": "The Windows Instance ID is: [u'i-019c03c3e3929f76e']"
}
As you can see, the instance ID is correct, but not well formated. Is there a way to convert this output into "human readable" output? Or is there any better way to parse the instance id from the ec2 task output?
Thanks!
It's not non-human readable format, but a list object in Python notation, because you query a list.
If you want a string, you should pass it through a first filter.
win_instance_id: "{{ ec2_result | json_query('tagged_instances[*].id') | first }}"
You can also access the value directly without json_query ([0] refers to the first element of a list):
win_instance_id: "{{ ec2_result.tagged_instances[0].id }}"