I have a variable of a large JSON data of students' data of different classes containing their names and numbers respectively. The variable is "data" and here is what the variable contains:
{
"Class001": [
{
"Jesse": 001
},
{
"Adam": 002
}
],
"Class002": [
{
"Jill": 111
},
{
"Ray": 112
}
],
.....
}
Using ansible, I want to separate the keys "Class00X" into some different files. For example, the Class001 file contain:
{
"Jesse": 001
},
{
"Adam": 002
},
.....
etc. How can i do this? thanks
i have created a file.json with:
{
"Class001": [
{
"Jesse": "001"
},
{
"Adam": "002"
}
],
"Class002": [
{
"Jill": "111"
},
{
"Ray": "112"
}
]
}
this playbook does the job:
- hosts: localhost
gather_facts: no
vars:
json: "{{ lookup('file', 'file.json') | from_json }}"
tasks:
- name: loop
copy:
dest: "files/{{ item.key }}.txt"
content: "{{ item.value | to_nice_json }}"
loop: "{{ json | dict2items }}"
here files files/class001.txt and files/class002.txt are created
A dictionary is a better structure to store students' names and numbers. Optionally, store the dictionaries instead of lists
- copy:
dest: "files/{{ item.key }}.yml"
content: "{{ item.value|combine|to_nice_yaml }}"
loop: "{{ json|dict2items }}"
gives
shell> cat files/Class001.yml
Adam: '002'
Jesse: '001'
shell> cat files/Class002.yml
Jill: '111'
Ray: '112
Related
I have this JSON output from Juniper switch where I need to get the remote system name associated with lldp_local_parent_interface_name having the value ae0.
So, I only know the value ae0 and I need to get the remote system name A_B_C_D in order to start targeting this host to execute some other tasks.
The data, inside the file juniper-data.json looks like this:
{
"output": {
"lldp_neighbors_information": [{
"lldp_neighbor_information": [{
"lldp_local_parent_interface_name": [{
"data": "ae0"
}],
"lldp_local_port_id": [{
"data": "xe_0/2/0"
}],
"lldp_remote_chassis_id": [{
"data": "00:00:00:00:00:00:00"
}],
"lldp_remote_chassis_id_subtype": [{
"data": "Mac address"
}],
"lldp_remote_port_description": [{
"data": "xe_1/0/1"
}],
"lldp_remote_system_name": [{
"data": "A_B_C_D"
}]
},
{
"lldp_local_parent_interface_name": [{
"data": "_"
}],
"lldp_local_port_id": [{
"data": "ge_0/0/23"
}],
"lldp_remote_chassis_id": [{
"data": "xx:xx:xx:xx:xx:xx"
}],
"lldp_remote_chassis_id_subtype": [{
"data": "Mac address"
}],
"lldp_remote_port_description": [{
"data": "bond0"
}]
}
]
}]
}
}
Here are my tasks:
- name: load data to var
set_fact:
remote_sys_name: "{{ lookup('file', 'juniper-data.json') | from_json }}"
- name: view loaded data
debug:
var: item
loop: "{{ remote_sys_name | community.general.json_query('output.lldp_neighbors_information[0].lldp_neighbor_information[*].[?lldp_local_parent_interface_name[0].data=='ae0'].lldp_remote_system_name[0].data') }}"
Expected results
"item": "A_B_C_D"
What I got is
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: expected token ',', got 'ae0'. String: {{ remote_sys_name | community.general.json_query('output.lldp_neighbors_information[0].lldp_neighbor_information[*].[?lldp_local_parent_interface_name[0].data=='ae0'].lldp_remote_system_name[0].data') }}"}
If you do not care about the exact hierarchy of nodes and their names beside lldp_neighbor_information, you can use a cohort of object projections and flatten projection to simplify it:
*.*[][].*[][?
lldp_local_parent_interface_name[0].data == 'ae0'
][].lldp_remote_system_name[].data
As for your current attempt, the issue lies in the fact that your condition is wrongly located in this bit:
lldp_neighbor_information[*].[?lldp_local_parent_interface_name[0].data=='ae0']
The condition should actual replace the star:
lldp_neighbor_information[?lldp_local_parent_interface_name[0].data=='ae0']
Ending up with the query:
output
.lldp_neighbors_information[0]
.lldp_neighbor_information[?
lldp_local_parent_interface_name[0].data=='ae0'
]
.lldp_remote_system_name[0]
.data
Given the task:
- debug:
var: item
loop: >-
{{
lookup('file', 'juniper-data.json')
| from_json
| json_query('
output
.lldp_neighbors_information[0]
.lldp_neighbor_information[?
lldp_local_parent_interface_name[0].data==`ae0`
]
.lldp_remote_system_name[0]
.data
')
}}
This yields:
ok: [localhost] => (item=A_B_C_D) =>
ansible_loop_var: item
item: A_B_C_D
Bonus: If know you before hand that you will only have one element, you can also end your query with | [0] to stop the projection and only get the first element.
This way, you can get rid of the loop:
- debug:
var: remote_sys_name
vars:
remote_sys_name: >-
{{
lookup('file', 'juniper-data.json')
| from_json
| json_query('
output
.lldp_neighbors_information[0]
.lldp_neighbor_information[?
lldp_local_parent_interface_name[0].data==`ae0`
]
.lldp_remote_system_name[0]
.data | [0]
')
}}
Does yields the name right away:
ok: [localhost] =>
remote_sys_name: A_B_C_D
Bonus bis: if your interface name is in a variable, here would be the task. Mind that it is in a variable local to the task here, but you might define it in any place your requirements call for.
- debug:
var: remote_sys_name
vars:
_interface_name: ae0
remote_sys_name: >-
{{
lookup('file', 'juniper-data.json')
| from_json
| json_query('
output
.lldp_neighbors_information[0]
.lldp_neighbor_information[?
lldp_local_parent_interface_name[0]
.data==`' ~ _interface_name ~ '`
]
.lldp_remote_system_name[0]
.data | [0]
')
}}
I have a json file with server information and the applications related.
Something like this:
{
"Apps": [
{
"AppOwner": "Me",
"AppNm": "Vacations"
}
],
"Hostnm": "some_server",
"Environment": "Prod",
"OSnm": "Windows Server 2008",
"OSManu": "Microsoft",
"SerialNum": "VMware-42 14 e1 37 7a 63 9b 0e-43 07 15 46 64 9c 3c 12"
}
I want to create a list with all the applications and the servers related
For example
{
"Vacations": [server1, server2]
},
{
"other_app": [server2, server3]
},
For that, I have been trying the following.
- set_fact:
apps: "{{ apps }} + {{ item.Apps | map(attribute='AppNm') | list}}"
loop: "{{ server_changes }}"
- set_fact:
apps_revised: "{{apps | unique}}"
- set_fact:
apps_server: "{{ apps_revised | default([]) + [{item : []}] }}"
with_items: "{{ apps_revised }}"
The first part of the code, will obtain all the applications with the duplicates from the json.
The second part of the code, will clean the apps with unique.
The last part its going to create a list of application with a nested list (of servers).
The problem is that I don't understand how to navigate the nested server_changes application list and then add the server to my new list.
Probably there is an easier way to do it, so any help will be appreciated.
Thanks
Given a test data
server_changes:
- "Apps":
- {"AppOwner": "Me", "AppNm": "Vacations"}
"Hostnm": "server1"
- "Apps":
- {"AppOwner": "Me", "AppNm": "Vacations"}
- {"AppOwner": "Me", "AppNm": "other_app"}
"Hostnm": "server2"
- "Apps":
- {"AppOwner": "Me", "AppNm": "other_app"}
"Hostnm": "server3"
Let's simplify the data. For example
- set_fact:
my_server_changes: "{{ my_server_changes|default([]) +
[{'hostname': item.Hostnm,
'applications': item.Apps|
json_query('[].AppNm')}] }}"
loop: "{{ server_changes }}"
- debug:
var: my_server_changes
gives
"my_server_changes": [
{
"applications": [
"Vacations"
],
"hostname": "server1"
},
{
"applications": [
"Vacations",
"other_app"
],
"hostname": "server2"
},
{
"applications": [
"other_app"
],
"hostname": "server3"
}
]
Create the list of the applications
- set_fact:
my_apps: "{{ server_changes|json_query('[].Apps[].AppNm')|unique }}"
- debug:
var: my_apps
gives
"my_apps": [
"Vacations",
"other_app"
]
Then loop the list of the applications, select records that contains the item and combine the dictionary
- set_fact:
apps_server: "{{ apps_server|default({})|
combine({item: my_server_changes|json_query(query)}) }}"
loop: "{{ my_apps }}"
vars:
query: "[?applications.contains(#, '{{ item }}')].hostname"
- debug:
var: apps_server
gives
"apps_server": {
"Vacations": [
"server1",
"server2"
],
"other_app": [
"server2",
"server3"
]
}
My data is in json file as shown below:
vmdata.json
{
"VMDetails":
[
{
"name": "Owner1",
"vms": [ "vm10", "vm11", "vm12", "vm13" ]
},
{
"name": "Owner2",
"vms": [ "vm20", "vm21", "vm22", "vm23" ]
},
{
"name": "Owner3",
"vms": [ "vm30", "vm31", "vm32", "vm33" ]
}
]
}
I need this json data converted to list of key-value pairs and later used for tagging the VMs on vcenter.
Owner1: vm11
Owner1: vm12
Owner1: vm13
Owner1: vm14
Owner2: vm21
Owner2: vm22
Owner2: vm23
Owner2: vm24
Owner3: vm31
Owner3: vm32
Owner3: vm33
Owner3: vm34
I have assigned the content from the data file to variable using :
vms_tobe_tagged: "{{ lookup ('file', 'vmtags.json')| from_json}}"
I query and get the list of owners using this and it works well:
- set_fact:
Owner: "{{ vms_tobe_tagged| json_query('OwnerDetails[*].name') }}"
- name: Test loop
debug:
msg: "{{ Owner }}"
Is it possible to generate the data using this?
loop: "{{ ['alice', 'bob'] |product(['clientdb', 'employeedb', 'providerdb'])|list }}"
Q: "I need this json data converted to a list of key-value pairs ..."
Owner1: vm11
Owner1: vm12
Owner1: vm13
Owner1: vm14
Owner2: vm21
...
Is it possible to generate the data using 'product' filter?
A: Yes. With the product filter it's possible to create a list that comprises all products. The list of the key-value pairs can be created in the next loop. For example the tasks below
- set_fact:
owner_list: "{{ owner_list|
default([]) +
[item.name]|product(item.vms)|list }}"
loop: "{{ vms_tobe_tagged.VMDetails }}"
- set_fact:
owner: "{{ owner|
default([]) +
[{item.0: item.1}] }}"
loop: "{{ owner_list }}"
- debug:
var: owner
give
"owner": [
{
"Owner1": "vm10"
},
{
"Owner1": "vm11"
},
{
"Owner1": "vm12"
},
{
"Owner1": "vm13"
},
{
"Owner2": "vm20"
...
This helped me. I used json_query, with_subelements to iterate.
- set_fact:
VMOwnerdetails: "{{ vms_tobe_tagged| json_query('VMDetails[*]') }}"
- name: Verbose updates
with_subelements:
- "{{VMOwnerdetails}}"
- vms
debug:
msg: "{{ item.0.name }} : {{ item.1 }}"
I have a variable (via set_fact) containing a json string:
{
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
What is the best way to insert more elements in the "Resource" array?
"arn:aws:iam::001122334455:role/Member1_Create_svcacct",
"arn:aws:iam::001122334455:role/Member1_Update_svcacct",
"arn:aws:iam::001122334455:role/Member1_Delete_svcacct",
"arn:aws:iam::001122334455:role/Member1_Full_svcacct"
I am exploring dumping the variable to a file and inserting the block I want with external shell tools, which does not seem to be elegant.
I don't know about the best way, but one option is to write a simple library module to handle the mechanics of the update for you. You could use the jsonpointer module as a way of locating the data you wish to modify, and then return the modified object to ansible. A starting point might look like:
#!/usr/bin/python
from ansible.module_utils.basic import AnsibleModule
import json
try:
import jsonpointer
except ImportError:
jsonpointer = None
def main():
module = AnsibleModule(
argument_spec=dict(
data=dict(required=True, type='dict'),
pointer=dict(required=True),
action=dict(required=True,
choices=['append', 'extend', 'update']),
update=dict(type='dict'),
extend=dict(type='list'),
append=dict(),
),
supports_check_mode=True,
)
if jsonpointer is None:
module.fail_json(msg='jsonpointer module is not available')
action = module.params['action']
data = module.params['data']
pointer = module.params['pointer']
if isinstance(data, str):
data = json.loads(str)
try:
res = jsonpointer.resolve_pointer(data, pointer)
except jsonpointer.JsonPointerException as err:
module.fail_json(msg=str(err))
if action == 'append':
res.append(module.params['append'])
if action == 'extend':
res.extend(module.params['extend'])
elif action == 'update':
res.update(module.params['update'])
module.exit_json(changed=True,
result=data)
if __name__ == '__main__':
main()
If you drop this into, e.g., library/json_modify.py, you can use it in a playbook like this:
- hosts: localhost
gather_facts: false
vars:
myvar: {
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
tasks:
- json_modify:
data: "{{ myvar }}"
pointer: "/PolicyVersion/Document/Statement/0/Resource"
action: extend
extend:
- "arn:aws:iam::001122334455:role/Member1_Create_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Update_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Delete_svcacct"
- "arn:aws:iam::001122334455:role/Member1_Full_svcacct"
register: result
- debug:
var: result.result
The result of running this playbook and the proposed module is:
TASK [debug] *******************************************************************
ok: [localhost] => {
"result.result": {
"PolicyVersion": {
"CreateDate": "2017-08-07T02:48:05Z",
"Document": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": [
"arn:aws:iam::123456789123:role/Root_Update_svcacct",
"arn:aws:iam::123456789123:role/Root_Delete_svcacct",
"arn:aws:iam::123456789123:role/Root_Create_svcacct",
"arn:aws:iam::123456789123:role/Root_Full_svcacct",
"arn:aws:iam::987654321987:role/Member1_Create_svcacct",
"arn:aws:iam::987654321987:role/Member1_Update_svcacct",
"arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
"arn:aws:iam::987654321987:role/Member1_Full_svcacct",
"arn:aws:iam::001122334455:role/Member1_Create_svcacct",
"arn:aws:iam::001122334455:role/Member1_Update_svcacct",
"arn:aws:iam::001122334455:role/Member1_Delete_svcacct",
"arn:aws:iam::001122334455:role/Member1_Full_svcacct"
]
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true,
"VersionId": "v2"
}
}
}
Actually Ansible is natively able to read JSON files.
see this question:
add-a-new-key-value-to-a-json-file-using-ansible
This is a little bit old. I know but I felt it could be useful to many people so I'll post here my solution as "the best way to modify a json" that I could personally find.
First of all the JSON. In my use-case I had a bunch of aws snapshots that were encrypted with the wrong kms key and I had to recreate the AMI with the correct key.
So I had to :
get data of the old snapshots ( like size device_name etc.)
create the new snaps with the different key
re-register a new ami with the correct block_device_mapping
Here's the code
- name: get ami
amazon.aws.ec2_ami_info:
image_ids: ami-<id>
region: "{{ region }}"
register: ami
- name: save snapshot ids and device_name and volume_size
set_fact:
snapshot_ids: "{{ ami | json_query('images[].block_device_mappings[].ebs.snapshot_id') }}"
device_name: "{{ ami | json_query('images[].block_device_mappings[].device_name') }}"
volume_size: "{{ ami | json_query('images[].block_device_mappings[].ebs.volume_size') }}"
basically each of the above is a list of each of the 3 things (device_name, snap_id, volume_size) that I need (but it could be extended)
then:
- name: get kms arn
aws_kms_info:
filters:
alias: "{{ kms_keys.alias }}"
region: "{{ region }}"
register: aws_kms_facts_out
- debug:
var: aws_kms_facts_out
- set_fact:
kms_arn: "{{ aws_kms_facts_out['keys'][0].key_arn }}"
- name: copy snap with new encryption key
community.aws.ec2_snapshot_copy:
source_region: "{{ region }}"
region: "{{ region }}"
source_snapshot_id: "{{ item }}"
encrypted: yes
kms_key_id: "{{ kms_arn }}"
loop: "{{ snapshot_ids }}"
register: new_snapshots
and then here's the catch
- set_fact:
new_snapshot_ids: "{{ new_snapshots| json_query('snapshots[].snapshot_id') }}"
- name: creating the block_device_mappings structure (still a list of objects)
ansible.builtin.debug:
msg: '{
"device_name": "{{ item.2 }}",
"delete_on_termination": "true",
"snapshot_id": "{{ item.0 }}",
"volume_size": {{ item.1 }},
"volume_type": "gp2"
}'
loop: "{{ new_snapshot_ids|zip_longest(volume_size,device_name)|list }}"
register: block_device_mappings
- set_fact:
bdm: "{{ block_device_mappings | json_query('results[].msg') }}"
finally
- name: create new ami from newly created snapshots
amazon.aws.ec2_ami:
region: "{{ region }}"
name: "{{ instance.name }}-{{ ansible_date_time.date }}"
state: present
architecture: x86_64
virtualization_type: hvm
root_device_name: /dev/sda1
device_mapping: "{{ bdm }}"
This is how you can do it without requiring any additional trick.
Of course this is declined to my particular use case but you can adapt it to any circumstance, that do not require a complete disassemble and reassemble of the Json itself
I have an issue to parse a json using ansible
I have a task that connected to rancher and get a json file
task:
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: test
set_fact:
rancher_env_hosts: "{{ item.hostname }}"
#when: item.hostname == "*-i-*"
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
and I get the following json (after edit it to be more readable):
{
"json": {
"data": [
{
"hostname": "rancher-i-host-02",
"id": "adsfsa"
},
{
"hostname": "rancher-i-host-01",
"id": "gfdgfdg"
},
{
"hostname": "rancher-q-host-01",
"id": "dfgdg"
},
{
"hostname": "rancher-q-host-02",
"id": "dfgdg"
}
]
}
}
When I start the playbook I get only the last host name in the variable and not all the list of hostname. can I register all the list to the same variable?
In addition, I also added a line with the a comment "#" in order to get only the host names that match the string "-i-" bit it's not working. any idea?
This is what filters (and this) for:
- set_fact:
hosts_all: "{{ hosts_json.json.data | map(attribute='hostname') | list }}"
hosts_i: "{{ hosts_json.json.data | map(attribute='hostname') | map('regex_search','.*-i-.*') | select('string') | list }}"
host_all will contain all hostnames, host_i will contain only .*-i-.* matching hostnames.
Try this
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: init fact
set_fact:
rancher_env_hosts: "[]"
- name: test
set_fact:
rancher_env_hosts: "{{rancher_env_hosts}} + [ {{item.hostname}} ]"
when: item.hostname | search(".*-i-.*")
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
About search you can read here http://docs.ansible.com/ansible/playbooks_tests.html
UPD:
About adding values to array here: Is it possible to set a fact of an array in Ansible?