Parsing json output retrieved from an API using Ansbile - json

I am new to Ansible. I am trying to fetch some API info from a centralized server in the json format, parse the json and set "certificate" to "yes" for each and every server and make a PUT operation back. GET and PUT operations in my code are working fine.
I am not able to find correct syntax to parse listofservers.json.
What is the syntax to display a specific json element??
Can you help me on how to parse the json to change "Certificate" value to Yes for all the servers?
My script
- hosts: serverlist
vars_prompt:
- name: "USER"
prompt: "Enter Admin Username"
private: no
- name: "PASS"
prompt: "Enter Admin Password"
private: yes
tasks:
- name: "Get the current values of all Servers"
delegate_to: localhost
register: listofservers
check_mode: no
uri:
validate_certs: "no"
user: "{{USER}}"
password: "{{PASS}}"
url: "https://console.exmaple.com/adminapi/serverlist"
method: "GET"
body_format: "json"
return_content: "no"
status_code: "200"
- name: Print json dictionary
debug:
var: listofservers.json
Output
TASK [Print json dictionary] *****************************************************************************************
ok: [centalServer.example.com] =>
{
"listofservers.json": {
"serverlist": [
{
"id": 1,
"servername": "redhat.example.com",
"apachestatus": "running",
"certificate": "no"
},
{
"id": 2,
"servername": "solaris.example.com",
"apachestatus": "down",
"certificate": "yes"
} ] } }

Q: "Change "certificate" value to "yes" for all servers."
A: The task below does the job
- set_fact:
listofservers: "{{ {'json':
{'serverlist': serverlist}} }}"
vars:
serverlist: "{{ listofservers.json.serverlist|
map('combine', {'certificate': 'yes'})|
list }}"
- debug:
var: listofservers.json
give
{
"listofservers.json": {
"serverlist": [
{
"apachestatus": "running",
"certificate": "yes",
"id": 1,
"servername": "redhat.example.com"
},
{
"apachestatus": "down",
"certificate": "yes",
"id": 2,
"servername": "solaris.example.com"
}
]
}
}
Q: "What is the syntax to display a specific JSON element?"
A: There are two main options 1) Ansible filters and 2) json_query, and many variations which depend on the structure of the data and requested query
For example, display first servername where id is equal to 1
- debug:
msg: "{{ listofservers.json.serverlist|
selectattr('id', 'eq', 1)|
map(attribute='servername')|
first }}"
gives
"msg": "redhat.example.com"
The same result will give json_query below
- debug:
msg: "{{ listofservers.json.serverlist|
json_query('[?id == `1`].servername')|
first }}"

Related

JSON query in Ansible Playbook failing to select desired data

I am new to writing Ansible playbooks and have hit a roadblock. I am trying to use the Site24x7 API to schedule maintenance and need to get a specific ID from a list of Monitor Groups. I created the following:
- name: Download Monitor Groups from Site24x7
uri:
url: https://www.site24x7.com/api/monitor_groups
method: GET
return_content: yes
headers:
Accept: "application/json; version=2.1"
Authorization: "Zoho-oauthtoken {{authtoken.json.access_token}}"
body:
return_content: yes
register: monitor_groups
- name: Get Monitor Group ID
set_fact:
monitorGroupID_query: "[?display_name=='{{hostname.stdout}}'].group_id"
- name: Assign Monitor Group ID
set_fact:
monitorGroupID: "{{ monitor_groups.json | json_query(monitor_group_id_query) }}"
- debug:
var: monitorGroupID
My API call returns data that looks like the following
"monitor_groups.json": {
"code": 0,
"data": [
{
"description": "System Generated",
"display_name": "server1",
"group_id": "319283000000505864",
"group_type": 1,
"health_threshold_count": 1,
"monitors": [
"319283000000483017"
]
},
{
"display_name": "server2",
"group_id": "319283000004701003",
"group_type": 3,
"health_threshold_count": 1,
"monitors": [
"319283000003989345",
"319283000004061005"
]
}
],
"message": "success"
}
My query constantly returns an empty string.
TASK [Assign Monitor Group ID] *****************************************************************************************
ok: [server1.fdu.edu] => {"ansible_facts": {"monitorGroupID": ""}, "changed": false}
TASK [debug] *****************************************************************************************
ok: [server1.fdu.edu] => {
"monitorGroupID": ""
}
Thank you in advance for your help
Single quotes don't work in JMESPath the way it does in almost any other "query language" -- you'll want ` characters wrapped around the JSON literal values. Also, the data:[] in your response is not implied, so if you meant to use it you'll need to either .json.data | json_query or alter your JMESPath to add the data[?display... part
- name: Assign Monitor Group ID
set_fact:
monitorGroupID: "{{ monitor_groups.json | json_query( monitorGroupID_query ) }}"
vars:
monitorGroupID_query: 'data[?display_name==`"{{hostname.stdout}}"`].group_id'
given hostname.stdout=server2 yields
ok: [localhost] => {"ansible_facts": {"monitorGroupID": ["319283000004701003"]}, "changed": false}
For example, given the data below
monitor_groups:
json:
code: 0
data:
- description: System Generated
display_name: server1
group_id: '319283000000505864'
group_type: 1
health_threshold_count: 1
monitors:
- '319283000000483017'
- display_name: server2
group_id: '319283000004701003'
group_type: 3
health_threshold_count: 1
monitors:
- '319283000003989345'
- '319283000004061005'
message: success
Create a dictionary of the hostnames and their group_id. Use it to evaluate the variable monitorGroupID
name_id: "{{ monitor_groups.json.data|
items2dict(key_name='display_name', value_name='group_id') }}"
monitorGroupID: "{{ name_id[hostname.stdout] }}"
gives
name_id:
server1: '319283000000505864'
server2: '319283000004701003'
Then, the task below
- debug:
var: monitorGroupID
gives
TASK [debug] **********************************************************
ok: [server1] =>
monitorGroupID: '319283000000505864'
ok: [server2] =>
monitorGroupID: '319283000004701003'

ansible obtain values from changing key

I would like to get value from json, but one of the key can be differet.
here is example json
{
"json": {
"id": "9758b1e5-442e-4545-9364-45f28477edfb",
"results": [{
"code": 200,
"host": "localhost",
"message": "no change",
"runTime": 1233,
"tenant": "http-validate-2.usa-dc.com"
}],
"traces": {
"http-validate-2.usa-dc.comCurrent": {
"/Common/10.10.100.10": {
"command": "ltm node"
},
"http-validate-2.usa-dc.comDiff": [{
"command": "ltm virtual",
"kind": "D",
"lhs": {
"default": "yes"
},
"lhsCommand": "ltm virtual",
"path": [
"/http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80",
"properties",
"persist",
"/Common/cookie"
],
"rhsCommand": "ltm virtual",
"tags": [
"tmsh"
]
}]
}
}
}
}
my ansible playbook
tasks:
- name : deploy json file AS3 to F5
debug:
msg: "{{ lookup('file', 'parse.json') }}"
register: atc_AS3_status
- name: debug
debug:
msg: "{{ atc_AS3_status.msg.json['traces']['.*Diff']}}"
I would like to reach key "path" but the key above "http-validate-2.usa-dc.comDiff" can be different like "http-validate-3.can-dc.comDiff" but always finish with Diff
Use json_query, e.g.
- debug:
msg: "{{ json.traces|json_query('*.*[][].path') }}"
should give the list of the paths (there might be more of them)
msg:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie
Q: "I need path only in the key which ends Diff."
A: JMESPath is not able to search key wildcards, AFAIK. Instead, use select and create the list of the nested keys that match the regex, e.g.
- debug:
msg: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
gives
msg:
- http-validate-2.usa-dc.comDiff
Then iterate this list, select path and concatenate the list paths, e.g.
- set_fact:
paths: "{{ paths|d([]) + json.traces|json_query(query) }}"
loop: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
vars:
query: '*."{{ item }}"[].path'
gives the list of paths for the keys that match the regex
paths:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie

Ansible - issue with loop and json_query's

Question
I need to get the ID from the GET because it's needed in the URL in the PUT task to edit a specific "input" entry. I'm using the Ansible URI to talk to a REST API to manage this.
playbook
*host_vars/host.yml
*
---
inputs:
- title: "test_input_api"
type: "org.graylog2.inputs.syslog.udp.SyslogUDPInput"
global: false
configuration:
allow_override_date: false
bind_address: "0.0.0.0"
expand_structured_data: false
force_rdns: false
number_worker_threads: 8
override_source: null
port: 5999
recv_buffer_size: null
store_full_message: true
- title: "test_input_api_2"
type: "org.graylog2.inputs.syslog.udp.SyslogUDPInput"
global: false
configuration:
allow_override_date: false
bind_address: "0.0.0.0"
expand_structured_data: false
force_rdns: false
number_worker_threads: 8
override_source: null
port: 5998
recv_buffer_size: null
store_full_message: true
playbook.yml
---
- name: Configure system
hosts: graylog
connection: local
gather_facts: no
roles:
- graylog/inputs
roles/graylog/inputs/tasks/main.yml
---
- include_tasks: get_inputs.yml
- include_tasks: put_inputs.yml
roles/graylog/inputs/tasks/get_inputs.yml
---
- name: "API GET System Inputs"
uri:
url: http://{{ ansible_host }}:9000/api/system/inputs
url_username : "{{ system.users.triple_admin.api_token }}"
url_password: token
method: GET
return_content: yes
register: get_graylog_inputs
- name: Set Fact
set_fact:
get_input_id: "{{ get_graylog_inputs.content | from_json | json_query('inputs[?title == `{}`] | [0].id '.format(input.title)) }}"
loop: "{{ inputs }}"
loop_control:
loop_var: input
The registered var from the get show's the following
{
"json": {
"inputs": [
{
"attributes": {
"allow_override_date": "False",
"bind_address": "0.0.0.0",
"expand_structured_data": "False",
"force_rdns": "False",
"number_worker_threads": 8,
"override_source": "",
"port": 5999,
"recv_buffer_size": "",
"store_full_message": "True"
},
"content_pack": null,
"created_at": "2021-07-30T15:21:47.590Z",
"creator_user_id": "triple_admin",
"global": false,
"id": "6104170beca15547502665d6",
"name": "Syslog UDP",
"node": "ba52ad48-0b13-419d-b957-d47d8911b413",
"static_fields": {},
"title": "test_input_api",
"type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput"
},
roles/graylog/inputs/tasks/put_inputs.yml
---
- name: "API PUT System Inputs"
uri:
url: http://{{ ansible_host }}:9000/api/system/inputs/{{ get_input_id }}
url_username : "{{ system.users.triple_admin.api_token }}"
url_password: token
headers:
X-Requested-By: X-Ansible
method: PUT
body_format: json
body: "{{ lookup('template', 'templates/post_template.j2') }}"
status_code: 201
return_content: yes
loop: "{{ inputs }}"
loop_control:
loop_var: input
"ansible_facts": {
"get_input_id": "61015085eca1554750236084",
"get_input_titles": "test_input_api"
},
"ansible_facts": {
"get_input_id": "610282d0eca155475024ac91",
"get_input_titles": "test_input_api_2"
Results of running the play
loop 1 - this needs to be matched to the title and therefor get id "61015085eca1554750236084"
"title": "test_input_api",
"url": "http://192.168.21.82:9000/api/system/inputs/610282d0eca155475024ac91",
loop 2
"title": "test_input_api_2",
"url": "http://192.168.21.82:9000/api/system/inputs/610282d0eca155475024ac91",
All help is welcome !
(not related) You don't need to json_decode the result get_graylog_inputs.content. If the server on the over side sends the correct Content-type: application/json header, you should have a get_graylog_inputs.json entry containing the already decoded json result.
You don't need to loop twice. Remove the set_fact loop (which is not correct anyway) in your first file and use the value from your register directly in the second loop.
You did not show any example of your input data so I have to guess a bit here from your jmespath expression... but you basically don't need json_query at all and can stick to generic core ansible filters.
Here is how I see the solution in the second file once you cleaned-up the first:
---
- name: "API PUT System Inputs"
vars:
get_input_id: "{{ get_graylog_inputs.json.inputs | selectattr('title', '==', input.title) | map(attribute='id') | first }}"
uri:
url: http://{{ ansible_host }}:9000/api/system/inputs/{{ get_input_id }}
url_username : "{{ system.users.triple_admin.api_token }}"
url_password: token
headers:
X-Requested-By: X-Ansible
method: PUT
body_format: json
body: "{{ lookup('template', 'templates/post_template.j2') }}"
status_code: 201
return_content: yes
loop: "{{ inputs }}"
loop_control:
loop_var: input
You will probably have to debug and tune the expression to get the input id as I could not do it myself against an example data structure.

ansible.builtin.uri module - Read and format JSON file content as a string to use in payload

I am trying to use the ansible.builtin.lookup plugin in order to read a JSON file from the local directory and then pass it as the payload to the ansible.builtin.uri module to send a POST message to a URI endpoint.
Following are the contents of my JSON file (config.json):
{
"Configuration": {
"Components": [
{
"Name": "A",
"Attributes": [
{
"Name": "A1",
"Value": "1",
"Set On Import": "True",
"Comment": "Read and Write"
},
{
"Name": "A2",
"Value": "2",
"Set On Import": "True",
"Comment": "Read and Write"
}
]
}
]
}
}
I need to send the above JSON content as the below string in the payload to ansible.builtin.uri module:
"{\"Configuration\": {\"Components\": [{\"Name\": \"A\", \"Attributes\": [{\"Name\": \"A1\", \"Value\": \"1\", \"Set On Import\": \"True\", \"Comment\": \"Read and Write\"}, {\"Name\": \"A2\", \"Value\": \"2\", \"Set On Import\": \"True\", \"Comment\": \"Read and Write\"}]}]}}"
I am trying to use the lookup plugin with the to_json filter to read and format the JSON content. Following is my playbook:
- name: import scp
ansible.builtin.uri:
url: "https://{{ inventory_hostname }}/api/config/actions/import"
user: "{{ user }}"
password: "{{ password }}"
method: POST
headers:
Accept: "application/json"
Content-Type: "application/json"
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | to_json }}"
body_format: json
status_code: 202
validate_certs: no
force_basic_auth: yes
However, the uri module double escapes all the new-line and tab characters. Following is the how the payload is sent when I run the playbook:
"invocation": {
"module_args": {
"attributes": null,
"body": {
"Buffer": "\"{\\n\\t\\\"Configuration\\\": {\\n\\t\\t\\\"Components\\\": [\\n\\t\\t\\t{\\n\\t\\t\\t\\t\\\"Name\\\": \\\"A\\\",\\n\\t\\t\\t\\t\\\"Attributes\\\": [\\n\\t\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\t\\\"Name\\\": \\\"A1\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Value\\\": \\\"1\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Set On Import\\\": \\\"True\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Comment\\\": \\\"Read and Write\\\"\\n\\t\\t\\t\\t\\t},\\n\\t\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\t\\\"Name\\\": \\\"A2\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Value\\\": \\\"2\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Set On Import\\\": \\\"True\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Comment\\\": \\\"Read and Write\\\"\\n\\t\\t\\t\\t\\t}\\n\\t\\t\\t\\t]\\n\\t\\t\\t}\\n\\t\\t]\\n\\t}\\n}\"",
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
"body_format": "json",
...
},
Could you please let me know how I can format the payload with uri module? Appreciate any help.
Edited (5/11/2021):
I made the changes as suggested by #mdaniel in his response and used string filter instead of to_json. With the suggested change, I can see the JSON being formatted properly into a string with newline ('\n') and tab ('\t') characters. I tried to use the replace filter to remove the \n and \t characters. However, now the whole string is converted back into the JSON.
Following is the playbook and the output when using the string filter alone:
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string }}"
$ ansible-playbook import_file.yml -i hosts --tags
...
"body": {
"HostPowerState": "On",
"Buffer": "{\n\t\"Configuration\": {\n\t\t\"Components\": [\n\t\t\t{\n\t\t\t\t\"Name\": \"A\",\n\t\t\t\t\"Attributes\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"A1\",\n\t\t\t\t\t\t\"Value\": \"1\",\n\t\t\t\t\t\t\"Set On Import\": \"True\",\n\t\t\t\t\t\t\"Comment\": \"Read and Write\"\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"A2\",\n\t\t\t\t\t\t\"Value\": \"2\",\n\t\t\t\t\t\t\"Set On Import\": \"True\",\n\t\t\t\t\t\t\"Comment\": \"Read and Write\"\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t]\n\t}\n}",
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
Following is the playbook and the output when using replace filter in conjunction with string filter:
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string | replace('\n', '') | replace('\t', '') }}"
...
$ ansible-playbook import_file.yml -i hosts --tags
...
"body": {
"Buffer": {
"Configuration": {
"Components": [
{
"Attributes": [
{
"Comment": "Read and Write",
"Name": "A1",
"Set On Import": "True",
"Value": "1"
},
{
"Comment": "Read and Write",
"Name": "A2",
"Set On Import": "True",
"Value": "2"
}
],
"Name": "A"
}
]
}
},
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
...
Any pointers on how I remove the \n and \t characters from the string?
You have used to_json on a dict value that is, itself, going to be to_json-ed; ansible cannot transmit a python dict over HTTP, so any yaml structure that is not already a string needs to be converted into one first
What you'll want is just that lookup result (which will return a str, not a dict) and then ansible will apply to_json to the whole body: value for the aforementioned reason
However, because ansible is trying to be "helpful", it will auto-coerce a yaml value that it finds starting with { back into a dict -- that's why you just need to send the result of lookup through the | string filter to reinforce to ansible that yes, you really do want it to remain a str in that context
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string }}"
updated answer approach
In light of the comment discussion that the dict coercion was continuing to be a problem, and the leading space concerned the OP, the alternative approach is to build up the actual payload structure completely, and only "JSON-ify" it before transmission, to keep ansible and jinja on the same page about the data types:
- name: import scp
vars:
body_dict:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
# this will be filled in before submission
# Buffer:
whitespace_free_config_json: >-
{{ lookup('file', 'config.json')
| regex_replace('[\t\n]', '')
| string
}}
ansible.builtin.uri:
...
body: >-
{{ body_dict
| combine({"Buffer": whitespace_free_config_json})
| to_json }}
body_format: json
status_code: 202

ansible parse json with several keys with the same name to one list variable

I have an issue to parse a json using ansible
I have a task that connected to rancher and get a json file
task:
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: test
set_fact:
rancher_env_hosts: "{{ item.hostname }}"
#when: item.hostname == "*-i-*"
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
and I get the following json (after edit it to be more readable):
{
"json": {
"data": [
{
"hostname": "rancher-i-host-02",
"id": "adsfsa"
},
{
"hostname": "rancher-i-host-01",
"id": "gfdgfdg"
},
{
"hostname": "rancher-q-host-01",
"id": "dfgdg"
},
{
"hostname": "rancher-q-host-02",
"id": "dfgdg"
}
]
}
}
When I start the playbook I get only the last host name in the variable and not all the list of hostname. can I register all the list to the same variable?
In addition, I also added a line with the a comment "#" in order to get only the host names that match the string "-i-" bit it's not working. any idea?
This is what filters (and this) for:
- set_fact:
hosts_all: "{{ hosts_json.json.data | map(attribute='hostname') | list }}"
hosts_i: "{{ hosts_json.json.data | map(attribute='hostname') | map('regex_search','.*-i-.*') | select('string') | list }}"
host_all will contain all hostnames, host_i will contain only .*-i-.* matching hostnames.
Try this
- uri:
url: http://rancher.local:8080/v1/hosts
method: GET
user: ##################
password: ################
body_format: json
register: hosts_json
- name: init fact
set_fact:
rancher_env_hosts: "[]"
- name: test
set_fact:
rancher_env_hosts: "{{rancher_env_hosts}} + [ {{item.hostname}} ]"
when: item.hostname | search(".*-i-.*")
with_items: "{{hosts_json.json.data}}"
- name: output
debug:
msg: "hosts: {{rancher_env_hosts}}"
About search you can read here http://docs.ansible.com/ansible/playbooks_tests.html
UPD:
About adding values to array here: Is it possible to set a fact of an array in Ansible?