jmespath Select arrays only if they contain specific value - json

I have this JSON as input:
{
"users": {
"alpha": [
"read",
"annotate",
"write",
"delete",
"manage"
],
"beta": [
"read",
"annotate",
"write",
"delete",
"manage"
],
"gamma": [
"read",
"annotate",
"write",
"delete"
],
"delta": [
"read",
"annotate",
"write",
"delete",
"manage"
]
}
}
And I've been asked to return the lists (users) only if they contain the element manage. Since this has to be processed with Ansible filter json_query, it should use only the JMESPath query language.
Expected result is something like:
["alpha", "beta", "delta"]

Try this
managers: "{{ users|dict2items|json_query(_query) }}"
_query: "[?contains(value, 'manage')].key"
It's an analogy to (credit #Zeitounator)
managers: "{{ users|dict2items|selectattr('value', 'contains', 'manage')|
map(attribute='key') }}"
Q: "Do you always need a list?"
A: In most cases yes, but not always. It depends on what you want to achieve. For example, the dictionary is fine if you want to find out whether all users contain the element manage or not
all_users_manage: "{{ users|json_query(_query2) }}"
_query2: "*.contains(#, 'manage')"
gives
all_users_manage:
- true
- true
- false
- true
Test the list
- debug:
msg: Not all users are managers.
when: not all_users_manage is all
Or, you can find out whether any user contains the element manage
- debug:
msg: There are managers among the users.
when: all_users_manage is any

Related

JMESPath filter for elements with nested values starting with certain string

I have the following JSON structure:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
},
{
"stack": [
"datasync"
],
"env": [
"val",
"prod"
],
"runOnBranch": [
"main"
]
}
]
And I would like to filter the list based on if a given string starts with one of the strings defined in the runOnBranch attribute.
My best guess so far doesn't work:
[?runOnBranch.starts_with(#, `feature/`) == `true`]
The error I get is:
"Search parse error": TypeError: starts_with() expected argument 1 to be type 2 but received type 3 instead.
The result I would like to get is:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
}
]
What am I missing?
In order to assess that there is one element in the array that starts with something you will need to:
assess each string in the array, effectively creating a list of boolean, so, with starts_with but targeting each string, not the whole array:
runOnBranch[].starts_with(#, `feature/`),
assess that there is at least one true value contained in the resulting array, with the help of the contains function
contains(runOnBranch[].starts_with(#, `feature/`), `true`)
and finally put all this in your filter
So, we end with the query:
[?contains(runOnBranch[].starts_with(#, `feature/`), `true`)]
Which yields:
[
{
"stack": [
"datasync"
],
"env": [
"dev",
"test"
],
"runOnBranch": [
"feature/",
"bugfix/",
"develop"
]
}
]
And to be more coherent in notation, this can also be written as:
[?(runOnBranch[].starts_with(#, `feature/`)).contains(#, `true`)]
Side note: simplify those kind of filter:
[?runOnBranch.starts_with(#, `feature/`) == `true`]
to
[?runOnBranch.starts_with(#, `feature/`)]
as starts_with already returns a boolean, as documented in the function signature:
boolean starts_with(string $subject, string $prefix)
Source: https://jmespath.org/specification.html#starts-with
Q: "Filter the list on attribute's runOnBranch any item starts with a given string."
A: Put the below declarations into the vars
_query: '[?runOnBranch[?starts_with(#, `feature/`)]]'
result: "{{ data|json_query(_query) }}"
gives
result:
- env: [dev, test]
runOnBranch: [feature/, bugfix/, develop]
stack: [datasync]
Example of a complete playbook for testing
- hosts: localhost
vars:
data:
- env: [dev, test]
runOnBranch: [feature/, bugfix/, develop]
stack: [datasync]
- env: [val, prod]
runOnBranch: [main]
stack: [datasync]
_query: '[?runOnBranch[?starts_with(#, `feature/`)]]'
result: "{{ data|json_query(_query) }}"
tasks:
- debug:
var: result|to_yaml
You can put the string into a variable. For example,
pattern: feature/
_query: '[?runOnBranch[?starts_with(#, `{{ pattern }}`)]]'
Then, you can override the string on the command line. For example,
shell> ansible-playbook playbook.yml -e pattern=ma
PLAY [localhost] *****************************************************************************
TASK [debug] *********************************************************************************
ok: [localhost] =>
feature|to_yaml: |-
- env: [val, prod]
runOnBranch: [main]
stack: [datasync]
PLAY RECAP ***********************************************************************************
localhost: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

ansible obtain values from changing key

I would like to get value from json, but one of the key can be differet.
here is example json
{
"json": {
"id": "9758b1e5-442e-4545-9364-45f28477edfb",
"results": [{
"code": 200,
"host": "localhost",
"message": "no change",
"runTime": 1233,
"tenant": "http-validate-2.usa-dc.com"
}],
"traces": {
"http-validate-2.usa-dc.comCurrent": {
"/Common/10.10.100.10": {
"command": "ltm node"
},
"http-validate-2.usa-dc.comDiff": [{
"command": "ltm virtual",
"kind": "D",
"lhs": {
"default": "yes"
},
"lhsCommand": "ltm virtual",
"path": [
"/http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80",
"properties",
"persist",
"/Common/cookie"
],
"rhsCommand": "ltm virtual",
"tags": [
"tmsh"
]
}]
}
}
}
}
my ansible playbook
tasks:
- name : deploy json file AS3 to F5
debug:
msg: "{{ lookup('file', 'parse.json') }}"
register: atc_AS3_status
- name: debug
debug:
msg: "{{ atc_AS3_status.msg.json['traces']['.*Diff']}}"
I would like to reach key "path" but the key above "http-validate-2.usa-dc.comDiff" can be different like "http-validate-3.can-dc.comDiff" but always finish with Diff
Use json_query, e.g.
- debug:
msg: "{{ json.traces|json_query('*.*[][].path') }}"
should give the list of the paths (there might be more of them)
msg:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie
Q: "I need path only in the key which ends Diff."
A: JMESPath is not able to search key wildcards, AFAIK. Instead, use select and create the list of the nested keys that match the regex, e.g.
- debug:
msg: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
gives
msg:
- http-validate-2.usa-dc.comDiff
Then iterate this list, select path and concatenate the list paths, e.g.
- set_fact:
paths: "{{ paths|d([]) + json.traces|json_query(query) }}"
loop: "{{ json.traces|json_query('*.keys(#)')|flatten|
select('match', '^.*Diff$')|list }}"
vars:
query: '*."{{ item }}"[].path'
gives the list of paths for the keys that match the regex
paths:
- - /http-validate-2.usa-dc.com/app/vs_http-validate-2.usa-dc.com_80
- properties
- persist
- /Common/cookie

ansible json-query trying to select "id" by content file wildcard

Very new to JSON.
I'm trying to extract 2 variables from this json file.
It has many files and id's but I only want the file & id if it contains es7.x86_64
When done my desired variables would be:
id=13140
file=NessusAgent-8.3.0-es7.x86_64.rpm
{
"banners": [],
"containsRequiredAuth": true,
"created_at": "2017-10-13T00:53:32.137Z",
"description": "Download Nessus Agents for use with Tenable.io and Nessus Manager",
"documentation_link": null,
"downloads": [
{
"created_at": "2021-06-29T19:06:41.776Z",
"description": "Red Hat ES 7 (64-bit) / CentOS 7 / Oracle Linux 7 (including Unbreakable Enterprise Kernel)",
"file": "NessusAgent-8.3.0-es7.x86_64.rpm",
"id": 13140,
"meta_data": {
"md5": "f67a2bdd2a7180f66b75f319439d56d5",
"product": "Nessus Agents - 8.3.0",
"product_notes": null,
"product_release_date": "06/29/2021",
"product_type": "default",
"release_date": "06/03/2021",
"sha256": "8a6452086ce0a7193e0f24b1f2adbff3aa6bd0f4ac519384e8453bb68bae0460",
"version": "8.3.0"
},
"name": "NessusAgent-8.3.0-es7.x86_64.rpm",
"page_id": 61,
"publish": true,
"required_auth": false,
"size": 16375828,
"sort_order": null,
"type": "download",
"updated_at": "2021-06-29T19:08:47.628Z"
},
My utterly failed attempt to assign file & id variables that have es7.x86_64.
- name: Convert agent_tempfile to json and register result
shell: python -m json.tool "{{ agent_tempfile }}"
register: result
- name: Extract file & id for es7.x86_64 rpm's
set_fact:
agent_id: "{{ result | json_query('downloads[*es7.x86_64*].id') | first }}"
agent_file: "{{ result | json_query('downloads[*es7.x86_64*].file') | first }}"
I have a feeling I'm going to be doing a lot more of these types of queries soon. Can some one also direct me to a good guide that details parsing specific values from JSON output? The stuff I've found so far just lists arrays but I really want to know how to pull specific data out.
First, there are some great tools out there for playing with JMESPath syntax (the syntax used by the json_query filter). The examples in the JMESPath tutorial are all "live": you can paste your own data into the text fields, and then experiment with filters and check the result.
The jpterm command is a terminal tool for experimenting with JMESPath queries. This is my personal favorite.
To look for items that contain a specific substring (like es7.x86_64), you can use the contains operator, like this:
json_query("downloads[?contains(name, 'es7.x86_64')]")
To make this work for your code, we first need to deal with the fact
that the result of your first task is going to be a string, rather
than a dictionary. We'll need to pass the standard output through the
from_json filter.
We can also avoid having two almost identical json_query expression
by moving the bulk of the expression into a task-local variable.
This gives us something like:
- hosts: localhost
gather_facts: false
tasks:
- command: cat data.json
register: result
- set_fact:
agent_id: "{{ selected[0].id }}"
agent_file: "{{ selected[0].file }}"
vars:
selected: >-
{{
result.stdout |
from_json |
json_query("downloads[?contains(name, 'es7.x86_64')]")
}}
- debug:
msg:
- "ID: {{ agent_id }}"
- "FILE: {{ agent_file }}"
When that task runs, the value of selected will be something like:
[
{
"file": "NessusAgent-8.3.0-es7.x86_64.rpm",
"id": 13140,
"name": "NessusAgent-8.3.0-es7.x86_64.rpm",
"page_id": 61,
"publish": true,
"required_auth": false,
"size": 16375828,
"sort_order": null,
"type": "download",
"updated_at": "2021-06-29T19:08:47.628Z"
}
]
This assumes you're only expecting a single result, so we can just ask
for selected[0] to get at that dictionary, and then it's a simple
matter of getting at the .id and .file attributes.
Running the above playbook produces:
TASK [debug] *********************************************************************************
ok: [localhost] => {
"msg": [
"ID: 13140",
"FILE: NessusAgent-8.3.0-es7.x86_64.rpm"
]
}

Parsing json output retrieved from an API using Ansbile

I am new to Ansible. I am trying to fetch some API info from a centralized server in the json format, parse the json and set "certificate" to "yes" for each and every server and make a PUT operation back. GET and PUT operations in my code are working fine.
I am not able to find correct syntax to parse listofservers.json.
What is the syntax to display a specific json element??
Can you help me on how to parse the json to change "Certificate" value to Yes for all the servers?
My script
- hosts: serverlist
vars_prompt:
- name: "USER"
prompt: "Enter Admin Username"
private: no
- name: "PASS"
prompt: "Enter Admin Password"
private: yes
tasks:
- name: "Get the current values of all Servers"
delegate_to: localhost
register: listofservers
check_mode: no
uri:
validate_certs: "no"
user: "{{USER}}"
password: "{{PASS}}"
url: "https://console.exmaple.com/adminapi/serverlist"
method: "GET"
body_format: "json"
return_content: "no"
status_code: "200"
- name: Print json dictionary
debug:
var: listofservers.json
Output
TASK [Print json dictionary] *****************************************************************************************
ok: [centalServer.example.com] =>
{
"listofservers.json": {
"serverlist": [
{
"id": 1,
"servername": "redhat.example.com",
"apachestatus": "running",
"certificate": "no"
},
{
"id": 2,
"servername": "solaris.example.com",
"apachestatus": "down",
"certificate": "yes"
} ] } }
Q: "Change "certificate" value to "yes" for all servers."
A: The task below does the job
- set_fact:
listofservers: "{{ {'json':
{'serverlist': serverlist}} }}"
vars:
serverlist: "{{ listofservers.json.serverlist|
map('combine', {'certificate': 'yes'})|
list }}"
- debug:
var: listofservers.json
give
{
"listofservers.json": {
"serverlist": [
{
"apachestatus": "running",
"certificate": "yes",
"id": 1,
"servername": "redhat.example.com"
},
{
"apachestatus": "down",
"certificate": "yes",
"id": 2,
"servername": "solaris.example.com"
}
]
}
}
Q: "What is the syntax to display a specific JSON element?"
A: There are two main options 1) Ansible filters and 2) json_query, and many variations which depend on the structure of the data and requested query
For example, display first servername where id is equal to 1
- debug:
msg: "{{ listofservers.json.serverlist|
selectattr('id', 'eq', 1)|
map(attribute='servername')|
first }}"
gives
"msg": "redhat.example.com"
The same result will give json_query below
- debug:
msg: "{{ listofservers.json.serverlist|
json_query('[?id == `1`].servername')|
first }}"

Extracting data from json within ansible

I have an ansible playbook that creates some IAM users in AWS.
I'd like the playbook to return the username and access keys details for each account created.
Registering the output from the task is fairly straightforward:
- name: Create IAM users
iam:
iam_type: user
name: "{{ item }}"
state: present
access_key_state: create
with_items:
- "user1"
- "someotheruser"
- "user3"
register: users
I'm finding doing something with that output tricky.
The Json it produces is as follows: (slightly truncated to reduce the length here)
ok: [localhost] => {
"users": {
"changed": true,
"msg": "All items completed",
"results": [
"user_meta": {
"access_keys": [
{
"access_key_id": "key1",
"access_key_selector": "HMAC",
"create_date": "2016-05-19T08:37:11.007Z",
"secret_access_key": "secretkey1",
"status": "Active",
"user_name": "user1"
}
],
}
},
{
"user_meta": {
"access_keys": [
{
"access_key_id": "key2",
"access_key_selector": "HMAC",
"create_date": "2016-05-19T08:37:13.391Z",
"secret_access_key": "secretkey2",
"status": "Active",
"user_name": "someotheruser"
}
],
}
},
{
"user_meta": {
"access_keys": [
{
"access_key_id": "key3",
"access_key_selector": "HMAC",
"create_date": "2016-05-19T08:37:16.243Z",
"secret_access_key": "secretkey3",
"status": "Active",
"user_name": "user3"
}
],
}
}
]
}
}
I've want to display this neatly on completion of the playbook, the closest I've come to getting what I'm after is use debug as follows:
- debug: var="users.results[0].user_meta.access_keys"
- debug: var="users.results[1].user_meta.access_keys"
- debug: var="users.results[2].user_meta.access_keys"
However that feels lame. What if I add user 4 and 5?
I'd like to be able to refer to the entire results array so that it will work no matter how many users I add to the playbook.
Is there a way to achieve this?
As a side note, I've also tried to use from_json, e.g.
- debug: msg="{{ (users.stdout|from_json).results }}"
but I don't know if I'm hitting a bug with ansible 2.0.1.0 or if I'm using it wrong but I could only ever get "type error. expected string or buffer"
Any thoughts gladly received.
You can iterate through the registered variable as well.
Something like this should work:
- name : debug access keys
debug : var="{{ item.user_meta.access_keys }}"
with_items: users.results
In case someone else wants to do the same thing...
I found the best solution for me was to count the number of objects in the array using 'length' and the then set the range that debug should iterate over.
I subtract "1" from the value returned since the first object in the array is referenced by "0".
- name: Record Users Access Keys
debug: var=users.results[{{ item }}].user_meta.access_keys
with_sequence: start=0 end={{users.results|length -1}}
Hope that helps someone looking to do something similar.
Ansiballer's answer worked for me, but if anyone else is looking for something simpler you can use json_query:
- name: show access keys
debug:
var: item
loop: "{{ users | json_query('results[*].user_meta.access_keys') }}"
Ansible docs on json_query
You may have to install jmespath for this to work (apt-get install python-jmespath or similar)