Parsing JSON from Ansible - json

I'm looking on the web but I'm unable to find a solution, the scenario in quite complex.
I have one json array with values like these:
[
{
"IP1": "1.2.3.4"
"IP1_VLAN": "900 - CLOUD-DEV"
"IP1_Role": "Management"
"IP2": "4.5.6.7"
"IP2_VLAN": "901 - CLOUD-DEV"
"IP2_Role": "Production"
"IP2": "8.9.10.11"
"IP2_VLAN": "902 - CLOUD-DEV"
"IP2_Role": "Backup"
}
]
My goal is: "Select the IP with the role of Management and tell me the address", it should select IP*_Role == Management and tell me in this case 1.2.3.4
Do you have any idea how to deal with this?

Following my comment, my 2 cent answer: searching through your actual data structure is going to be hard and ugly. If you have the option to change it a bit, things can get much easier.
Here is an example using a list declared in yaml in the following demo playbook. Data is extracted using json_query with the relevent jmespath expression (i.e. the query variable in the debug task)
---
- hosts: localhost
gather_facts: False
vars:
my_ips:
- ip: "1.2.3.4"
vlan: "900 - CLOUD-DEV"
role: "Management"
- ip: "4.5.6.7"
vlan: "901 - CLOUD-DEV"
role: "Production"
- ip: "8.9.10.11"
vlan: "902 - CLOUD-DEV"
role: "Backup"
tasks:
- name: Get ips per role
vars:
query: >-
[?role=='{{ item }}']
debug:
msg: "List of ips for role {{ item }}: {{ my_ips | json_query(query) }}"
loop:
- Management
- Production
- Backup
Which gives
PLAY [localhost] ********************************************************************************************************************************************************************************************************************************************************
TASK [Get ips per role] *************************************************************************************************************************************************************************************************************************************************
Wednesday 23 October 2019 16:14:23 +0200 (0:00:00.043) 0:00:00.043 *****
ok: [localhost] => (item=Management) => {
"msg": "List of ips for role Management: [{'ip': '1.2.3.4', 'vlan': '900 - CLOUD-DEV', 'role': 'Management'}]"
}
ok: [localhost] => (item=Production) => {
"msg": "List of ips for role Production: [{'ip': '4.5.6.7', 'vlan': '901 - CLOUD-DEV', 'role': 'Production'}]"
}
ok: [localhost] => (item=Backup) => {
"msg": "List of ips for role Backup: [{'ip': '8.9.10.11', 'vlan': '902 - CLOUD-DEV', 'role': 'Backup'}]"
}
PLAY RECAP **************************************************************************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The equivalent variable declaration in plain json would be:
[
{
"ip": "1.2.3.4",
"vlan": "900 - CLOUD-DEV",
"role": "Management"
},
{
"ip": "4.5.6.7",
"vlan": "901 - CLOUD-DEV",
"role": "Production"
},
{
"ip": "8.9.10.11",
"vlan": "902 - CLOUD-DEV",
"role": "Backup"
}
]
If you have to load this from an external string (e.g. loading a file...), you can use the from_json filter in your playbook

Related

JMESPath filter for elements with nested values starting with certain string

I have the following JSON structure:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
},
{
"stack": [
"datasync"
],
"env": [
"val",
"prod"
],
"runOnBranch": [
"main"
]
}
]
And I would like to filter the list based on if a given string starts with one of the strings defined in the runOnBranch attribute.
My best guess so far doesn't work:
[?runOnBranch.starts_with(#, `feature/`) == `true`]
The error I get is:
"Search parse error": TypeError: starts_with() expected argument 1 to be type 2 but received type 3 instead.
The result I would like to get is:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
}
]
What am I missing?
In order to assess that there is one element in the array that starts with something you will need to:
assess each string in the array, effectively creating a list of boolean, so, with starts_with but targeting each string, not the whole array:
runOnBranch[].starts_with(#, `feature/`),
assess that there is at least one true value contained in the resulting array, with the help of the contains function
contains(runOnBranch[].starts_with(#, `feature/`), `true`)
and finally put all this in your filter
So, we end with the query:
[?contains(runOnBranch[].starts_with(#, `feature/`), `true`)]
Which yields:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
}
]
And to be more coherent in notation, this can also be written as:
[?(runOnBranch[].starts_with(#, `feature/`)).contains(#, `true`)]
Side note: simplify those kind of filter:
[?runOnBranch.starts_with(#, `feature/`) == `true`]
to
[?runOnBranch.starts_with(#, `feature/`)]
as starts_with already returns a boolean, as documented in the function signature:
boolean starts_with(string $subject, string $prefix)
Source: https://jmespath.org/specification.html#starts-with
Q: "Filter the list on attribute's runOnBranch any item starts with a given string."
A: Put the below declarations into the vars
_query: '[?runOnBranch[?starts_with(#, `feature/`)]]'
result: "{{ data|json_query(_query) }}"
gives
result:
- env: [dev, test]
runOnBranch: [feature/, bugfix/, develop]
stack: [datasync]
Example of a complete playbook for testing
- hosts: localhost
vars:
data:
- env: [dev, test]
runOnBranch: [feature/, bugfix/, develop]
stack: [datasync]
- env: [val, prod]
runOnBranch: [main]
stack: [datasync]
_query: '[?runOnBranch[?starts_with(#, `feature/`)]]'
result: "{{ data|json_query(_query) }}"
tasks:
- debug:
var: result|to_yaml
You can put the string into a variable. For example,
pattern: feature/
_query: '[?runOnBranch[?starts_with(#, `{{ pattern }}`)]]'
Then, you can override the string on the command line. For example,
shell> ansible-playbook playbook.yml -e pattern=ma
PLAY [localhost] *****************************************************************************
TASK [debug] *********************************************************************************
ok: [localhost] =>
feature|to_yaml: |-
- env: [val, prod]
runOnBranch: [main]
stack: [datasync]
PLAY RECAP ***********************************************************************************
localhost: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

JSON query in Ansible Playbook failing to select desired data

I am new to writing Ansible playbooks and have hit a roadblock. I am trying to use the Site24x7 API to schedule maintenance and need to get a specific ID from a list of Monitor Groups. I created the following:
- name: Download Monitor Groups from Site24x7
uri:
url: https://www.site24x7.com/api/monitor_groups
method: GET
return_content: yes
headers:
Accept: "application/json; version=2.1"
Authorization: "Zoho-oauthtoken {{authtoken.json.access_token}}"
body:
return_content: yes
register: monitor_groups
- name: Get Monitor Group ID
set_fact:
monitorGroupID_query: "[?display_name=='{{hostname.stdout}}'].group_id"
- name: Assign Monitor Group ID
set_fact:
monitorGroupID: "{{ monitor_groups.json | json_query(monitor_group_id_query) }}"
- debug:
var: monitorGroupID
My API call returns data that looks like the following
"monitor_groups.json": {
"code": 0,
"data": [
{
"description": "System Generated",
"display_name": "server1",
"group_id": "319283000000505864",
"group_type": 1,
"health_threshold_count": 1,
"monitors": [
"319283000000483017"
]
},
{
"display_name": "server2",
"group_id": "319283000004701003",
"group_type": 3,
"health_threshold_count": 1,
"monitors": [
"319283000003989345",
"319283000004061005"
]
}
],
"message": "success"
}
My query constantly returns an empty string.
TASK [Assign Monitor Group ID] *****************************************************************************************
ok: [server1.fdu.edu] => {"ansible_facts": {"monitorGroupID": ""}, "changed": false}
TASK [debug] *****************************************************************************************
ok: [server1.fdu.edu] => {
"monitorGroupID": ""
}
Thank you in advance for your help
Single quotes don't work in JMESPath the way it does in almost any other "query language" -- you'll want ` characters wrapped around the JSON literal values. Also, the data:[] in your response is not implied, so if you meant to use it you'll need to either .json.data | json_query or alter your JMESPath to add the data[?display... part
- name: Assign Monitor Group ID
set_fact:
monitorGroupID: "{{ monitor_groups.json | json_query( monitorGroupID_query ) }}"
vars:
monitorGroupID_query: 'data[?display_name==`"{{hostname.stdout}}"`].group_id'
given hostname.stdout=server2 yields
ok: [localhost] => {"ansible_facts": {"monitorGroupID": ["319283000004701003"]}, "changed": false}
For example, given the data below
monitor_groups:
json:
code: 0
data:
- description: System Generated
display_name: server1
group_id: '319283000000505864'
group_type: 1
health_threshold_count: 1
monitors:
- '319283000000483017'
- display_name: server2
group_id: '319283000004701003'
group_type: 3
health_threshold_count: 1
monitors:
- '319283000003989345'
- '319283000004061005'
message: success
Create a dictionary of the hostnames and their group_id. Use it to evaluate the variable monitorGroupID
name_id: "{{ monitor_groups.json.data|
items2dict(key_name='display_name', value_name='group_id') }}"
monitorGroupID: "{{ name_id[hostname.stdout] }}"
gives
name_id:
server1: '319283000000505864'
server2: '319283000004701003'
Then, the task below
- debug:
var: monitorGroupID
gives
TASK [debug] **********************************************************
ok: [server1] =>
monitorGroupID: '319283000000505864'
ok: [server2] =>
monitorGroupID: '319283000004701003'

How to append JSON array with specific data

I am trying to run :
- name: Describe config aggregator
shell: >
aws configservice describe-configuration-aggregators --configuration-aggregator-name test-config
register: config_ouput
below is the data generated.
{
"ConfigurationAggregators": [
{
"ConfigurationAggregatorName": "test-config",
"ConfigurationAggregatorArn": "arn:aws:config:us-east-1:4567:config-aggregator/config-aggregator-uw2o9pzf",
"AccountAggregationSources": [
{
"AccountIds": [
"895677"
],
"AllAwsRegions": true
}
],
"CreationTime": 1624454176.124,
"LastUpdatedTime": 1626426755.504
}
]
}
Now I want to append the accountIds above with any new account say 1234567 which should give me result such as
{
"ConfigurationAggregators": [
{
"ConfigurationAggregatorName": "test-config",
"ConfigurationAggregatorArn": "arn:aws:config:us-east-1:8778:config-aggregator/test-config-pzf",
"AccountAggregationSources": [
{
"AccountIds": [
"895677,1234567"
],
"AllAwsRegions": true
}
],
"CreationTime": 1624454176.124,
"LastUpdatedTime": 1626426755.504
}
]
}
I am trying to do is:
- name: Export results to JSON
set_fact:
config_ouput_json: "{{ config_ouput + [{"AccountIds": "1234567","AllAwsRegions": true}]}}"
but this doesn't work, please let me know the right syntax.
Basically you require bit of JSON manipulation to achieve your task.
Steps :
Store output of first command in some json file. In your case you can keep that as registered variable of ansible.
Get existing account_ids in some variable.
Create a list of new accounts as variables in ansible.
Iterate over new account_ids and add to existing account_ids.
Update the aws config command.
Sample Code :
- name: initial validation
hosts: localhost
connection: local
vars:
newAccountIds:
- "123456"
- "566544"
- "555445"
tasks:
- name: register json file
include_vars:
file: 'abc.json'
name: bundle
- name: set value
set_fact:
values: "{{ bundle['ConfigurationAggregators'][0]['AccountAggregationSources'][0]['AccountIds'] }}"
- set_fact:
values: "{{ (values | default([])) + [item] }}"
with_items: "{{ newAccountIds }}"
- debug:
msg: "{{ values }}"
- debug:
msg: '"aws configservice put-configuration-aggregator --configuration-aggregator-name test-config --account-aggregation-sources "[{"AccountIds": {{ values | to_json }},"AwsRegions": ["us-east-1"]}]\""'
Sample Output :
PLAY [initial validation] ********************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************
ok: [localhost]
TASK [register json file] ********************************************************************************************
ok: [localhost]
TASK [set value] *****************************************************************************************************
ok: [localhost]
TASK [set_fact] ******************************************************************************************************
ok: [localhost] => (item=123456)
ok: [localhost] => (item=566544)
ok: [localhost] => (item=555445)
TASK [debug] *********************************************************************************************************
ok: [localhost] => {
"msg": [
"895677",
"123456",
"566544",
"555445"
]
}
TASK [debug] *********************************************************************************************************
ok: [localhost] => {
"msg": "\"aws configservice put-configuration-aggregator --configuration-aggregator-name test-config --account-aggregation-sources \"[{\"AccountIds\": [\"895677\", \"123456\", \"566544\", \"555445\"],\"AwsRegions\": [\"us-east-1\"]}]\\\"\""}
PLAY RECAP ***********************************************************************************************************
localhost : ok=6 changed=0 unreachable=0 failed=0

Parsing json output retrieved from an API using Ansbile

I am new to Ansible. I am trying to fetch some API info from a centralized server in the json format, parse the json and set "certificate" to "yes" for each and every server and make a PUT operation back. GET and PUT operations in my code are working fine.
I am not able to find correct syntax to parse listofservers.json.
What is the syntax to display a specific json element??
Can you help me on how to parse the json to change "Certificate" value to Yes for all the servers?
My script
- hosts: serverlist
vars_prompt:
- name: "USER"
prompt: "Enter Admin Username"
private: no
- name: "PASS"
prompt: "Enter Admin Password"
private: yes
tasks:
- name: "Get the current values of all Servers"
delegate_to: localhost
register: listofservers
check_mode: no
uri:
validate_certs: "no"
user: "{{USER}}"
password: "{{PASS}}"
url: "https://console.exmaple.com/adminapi/serverlist"
method: "GET"
body_format: "json"
return_content: "no"
status_code: "200"
- name: Print json dictionary
debug:
var: listofservers.json
Output
TASK [Print json dictionary] *****************************************************************************************
ok: [centalServer.example.com] =>
{
"listofservers.json": {
"serverlist": [
{
"id": 1,
"servername": "redhat.example.com",
"apachestatus": "running",
"certificate": "no"
},
{
"id": 2,
"servername": "solaris.example.com",
"apachestatus": "down",
"certificate": "yes"
} ] } }
Q: "Change "certificate" value to "yes" for all servers."
A: The task below does the job
- set_fact:
listofservers: "{{ {'json':
{'serverlist': serverlist}} }}"
vars:
serverlist: "{{ listofservers.json.serverlist|
map('combine', {'certificate': 'yes'})|
list }}"
- debug:
var: listofservers.json
give
{
"listofservers.json": {
"serverlist": [
{
"apachestatus": "running",
"certificate": "yes",
"id": 1,
"servername": "redhat.example.com"
},
{
"apachestatus": "down",
"certificate": "yes",
"id": 2,
"servername": "solaris.example.com"
}
]
}
}
Q: "What is the syntax to display a specific JSON element?"
A: There are two main options 1) Ansible filters and 2) json_query, and many variations which depend on the structure of the data and requested query
For example, display first servername where id is equal to 1
- debug:
msg: "{{ listofservers.json.serverlist|
selectattr('id', 'eq', 1)|
map(attribute='servername')|
first }}"
gives
"msg": "redhat.example.com"
The same result will give json_query below
- debug:
msg: "{{ listofservers.json.serverlist|
json_query('[?id == `1`].servername')|
first }}"

Ansible | Process data which can be either JSON or YAML

I'm using Ansible to read a config, which can be either JSON or YAML and extract values from some of the nodes in the file.
I know I can use from_json or from_yaml to process it in Ansible, but since I don't know which format the config will be in, I'm having difficulty making it work.
The file is Kubernetes' Kubeconfig. Examples below:
in YAML
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://my-k8s-cluster.com
name: k8s-clstr-master
contexts:
- context:
cluster: k8s-clstr-master
namespace: kube-system
user: k8s-clstr-master-admin
name: k8s-clstr-master
current-context: k8s-clstr-master
kind: Config
preferences: {}
users:
- name: k8s-clstr-master-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
in JSON
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "k8s-clstr-master",
"cluster": {
"server": "https://my-k8s-cluster.com",
"certificate-authority-data": "REDACTED"
}
}
],
"users": [
{
"name": "k8s-clstr-master-admin",
"user": {
"client-certificate-data": "REDACTED",
"client-key-data": "REDACTED"
}
}
],
"contexts": [
{
"name": "k8s-clstr-master",
"context": {
"cluster": "k8s-clstr-master",
"user": "k8s-clstr-master-admin",
"namespace": "kube-system"
}
}
],
"current-context": "k8s-clstr-master"
}
Ansible I'm using:
vars:
kubeconfig: "{{ lookup('hashivault', '/kubeconfig/admin', 'config') }}"
tasks:
- name: Find cluster server name
shell: "echo {{ kubeconfig.clusters[0].cluster.server }}"
Above Ansible block will work okay if kubeconfig is retrieved in JSON format, but it will fail if it's retrieved as in YAML format.
I might be able to make a task with |from yaml and then add ignore_errors: true, but that just doesn't feel like right way of doing it.
Anyone has any tips for me on how I can approach this problem?
There are some built-in tests in Jinja2.
The way Ansible templator works if you have JSON string inside {{...}} expression, it is automatically converted to object. So if you fetch JSON from your vault, kubeconfig becomes object, otherwise it is a string.
Here's a recipe for you:
vars:
kubeconfig_raw: "{{ lookup('hashivault', '/kubeconfig/admin', 'config') }}"
kubeconfig: "{{ kubeconfig_raw if kubeconfig_raw is mapping else kubeconfig_raw | from_yaml }}"
tasks:
- name: Find cluster server name
shell: "echo {{ kubeconfig.clusters[0].cluster.server }}"
If you use the include_vars task, it does not matter which format you provide. The task accepts both.
---
- hosts: localhost
connection: local
tasks:
- include_vars:
file: config
name: kubeconfig
- debug: var=kubeconfig