Extract part of JSON in ansible playbook - json

I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"

There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params

Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"

Related

ansible - parsing json w jmesquery

I'm querying Splunk via the uri module w a certain search:
- name: splunk query
uri:
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=json
method: POST
user: ...
password: ...
validate_certs: false
body_format: form-urlencoded
return_content: true
headers:
Content-Type: application/json
body:
- [ search, "{{ splunk_search }}" ]
vars:
- splunk_search: '| ...'
register: me
Splunk returns a 200, w content:
TASK [debug] ********************************************************************************************************************************************************************************************
ok: [host1] => {
"msg": "{\n \"title\": \"clientMessageId\",\n \"app\": \"whatever\"\n}\n{\n \"title\": \"a_title\",\n \"app\": \"some_app\"\n}\n{\n \"title\": \"another_title\",\n \"app\": \"my_app\"\n}\n{\n \"title\": \"title_for_app\",\n \"app\": \"another_app\"\n}"
}
However, I can't properly parse the output, I've tried:
- name: query result from content
debug:
msg: "{{ me.content | json_query('title') }}"
In this case, Ansible will return an empty string. I assume something is wrong with the formatting of the output.
Q: How can I configure Ansible to properly parse the json output, so that a new list is created based on the output.
Example:
my_list:
- title: clientMessageId
app: whatever
- title: a_title
app: some_app
...
What I did was change the url output_mode in which Splunk returns the content from json to csv.
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=csv
Then, I wrote the output to a file and read it using the ansible read_csv module.
- name: write received content to csv file on controller
copy:
content: "{{ me.content }}"
dest: "./file.csv"
mode: '0775'
delegate_to: localhost
- name: read csv file
read_csv:
path: ./file.csv
delegate_to: localhost
register: read_csv
- name: set fact to create list
set_fact:
the_output: "{{ read_csv.list }}"

Ansible: How to parse looped registered variable?

I'm working on a playbook that needs to make some webrequests which is getting back JSON output. This works awesome for single requests, but I need to loop through a list to get a list of docker images.
Here is what I got so far:
- name: get all repositories from project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories?page=1&page_size=50"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
register: repositories
- name: temp to variable
set_fact:
repo_list_tmp: "{{ repositories.json | json_query('[*].name') | list }}"
- name: set repos as a list and trim names
set_fact:
repo_list: "{{ repo_list|default([]) + [ item.split(seperator)[1] ] }}"
loop: "{{ repo_list_tmp }}"
- name: get all images from a project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories/{{ item }}/artifacts?page=1&page_size=50&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: "application/json"
loop: "{{ repo_list }}"
register: images
The last task loops through the repo_list variable and register it's output to "images", as Ansible works the register in a loop always creates a dictionary and here's the proble, I got no clue on how to get specific values with a json query or something else out of it.
As the output is too big for stackoverflow I added it to a pastebin:
https://pastebin.com/GzQJfBUL
This is the whole output from the URI task. What I need is to extract the following:
item (in this example myimage or myotherimage) -> tags -> name
For example, inside the output is this entry:
"tags": [
{
"artifact_id": 4776,
"id": 2373,
"immutable": false,
"name": "MYTAG_ELASTICSEARCH",
"pull_time": "0001-01-01T00:00:00.000Z",
"push_time": "2021-09-21T11:33:47.687Z",
"repository_id": 303,
"signed": false
}
How do I extract values from a looped registerd value?
Thank you
Dan
According to mdaniel's comment below the first post, this did the trick:
{{ images | json_query('results[*].json[*].tags[*].name') | flatten }}
Thank you!

Why is Ansible unable to read unicode string as JSON?

Summary
When retrieving data using the uri module in Ansible, I am unable to parse a section of it as JSON to retrieve a nested value.
The desired value is the ci field inside the content.data or json.data field (see output below).
Steps to Reproduce
site.yml
---
- hosts: localhost
gather_facts: false
tasks:
- name: Get String
uri:
url: "http://localhost/get-data"
method: POST
body_format: json
body: "{ \"kong-jid\": \"run-sn-discovery\" }"
return_content: yes
register: output
- set_fact:
ci: "{{ output.json.data.ci }}"
- debug:
msg: "{{ ci }}"
The {{ output }} variable
{
u'status': 200,
u'cookies': {},
u'url': u'http://kong-demo:8000/get-data',
u'transfer_encoding': u'chunked',
u'changed': False,
u'connection': u'close',
u'server': u'kong/0.34-1-enterprise-edition',
u'content':
u'{"data":"\\"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}\\""}',
'failed': False,
u'json': {u'data': u'"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}"'},
u'content_type': u'application/json',
u'date': u'Thu, 18 Apr 2019 15:50:25 GMT',
u'redirected': False,
u'msg': u'OK (unknown bytes)'
}
Result
[user#localhost]$ ansible-playbook site.yml
[WARNING]: Could not match supplied host pattern, ignoring: all
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [localhost] ***************************************************************************************************************
TASK [Pass Redis data to next task as output] **********************************************************************************
ok: [localhost]
TASK [set_fact] ****************************************************************************************************************
fatal: [localhost]: FAILED! => {}
MSG:
The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
The error appears to have been in 'site.yml': line 19, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- set_fact:
^ here
exception type: <class 'ansible.errors.AnsibleUndefinedVariable'>
exception: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
Important Troubleshooting Information
It appears the root issue is related to which Ansible type being interpreted. I desire to parse ci from the output in one task.
The two-task solution shown below works, but this leads me to believe this should be possible in one line...
Two-Task Solution
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci['ci'] }}"
But the ci fact set from {{ output.json.data | from_json }} reports a different TYPE than the inline type...
Unicode or Dict?
- debug:
msg: "{{ output.json.data | from_json | type_debug }}" # returns unicode
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci | type_debug }}" # returns dict
Why isn't {{ output.json.data | from_json | type_debug }}
the same as {{ ci | type_debug }}?
Although json and data are keys in their resp objects, ci is just part of a larger string (which happens to look like a JSON object
If the relevant line in your datastructure would be:
u'json': {u'data': {'ci': u'3bb8d625dbac3700e4f07b6e0f96195b'}},
then you could expect to use "{{ output.json.data.ci }}" but not when the .ci part is just a normal part of a string.

Ansible Extract JSON Tag

I'm trying to work with Infoblox API, and it's responses. I would need to extract values of tags from the response, that seems to be in JSON format, but I cannot find the way to do it.
Here is my playbook:
- name: "Checking _node_exporter Service Record for {{ inventory_hostname }}"
local_action:
module: uri
url: "{{ infobloxapiurl }}record:srv?name=_node_exporter.domain.com&target={{ inventory_hostname }}"
force_basic_auth: yes
user: "{{ infobloxuser }}"
password: "{{ infobloxpassword }}"
validate_certs: no
return_content: yes
register: _infoblox_results
- debug:
var: _infoblox_results.json
The _infoblox_results.json variable looks like this:
TASK [prometheus : debug] *******************************************************************************************************************************************************************************************
task path: /ansible/roles/tasks/task.yml:38
ok: [server.domain.com] => {
"_infoblox_results.json": [
{
"_ref": "record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default",
"name": "_node_exporter.domain.com",
"port": 9100,
"priority": 0,
"target": "server.domain.com",
"view": "default",
"weight": 0
}
]
}
I want to use the data of _ref from _infoblox_results.json, but I wasn't able to extract it with regex_replace (it drops back the full _infoblox_results.json):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | regex_replace('record:srv.*\\/default,', '\\1') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Neither with json_query (it drops back nothing):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | json_query('_ref') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Can someone please point me into the right direction?
You have already an object in the memory, so simply refer to its value: _infoblox_results.json[0]._ref contains the string record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default.
With that you can split the string and select the second element:
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json[0]._ref.split('/')[1] }}"

Ansible parse json and read result into different variables

I've set up a task which queries the github api meta endpoint and returns the following
{
"verifiable_password_authentication": true,
"github_services_sha": "f9e3a6b98d76d9964a6613d581164039b8d54d89",
"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
],
"git": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20",
"13.229.188.59/32",
"13.250.177.223/32",
"18.194.104.89/32",
"18.195.85.27/32",
"35.159.8.160/32",
"52.74.223.119/32"
],
"pages": [
"192.30.252.153/32",
"192.30.252.154/32",
"185.199.108.153/32",
"185.199.109.153/32",
"185.199.110.153/32",
"185.199.111.153/32"
],
"importer": [
"54.87.5.173",
"54.166.52.62",
"23.20.92.3"
]
}
What I need to do is get the 3 hook IPs and read them each into their own variable.
I've tried a couple of solutions i've found around but nothing is seeming to work for me.
I've got as far as drilling down into the json so i'm being returned only the 3 IPs, but how do I get them out and into variables individually?
i gave it a shot using j2 syntax in the variable name part, and - TIL - looks like the jinja2 syntax is allowed in that part as well!
please see playbook to process the hooks list variable and assign to variables variable_1, variable_2, variable_3 and so on:
- hosts: localhost
gather_facts: false
vars:
counter: 1
hooks:
- 192.30.252.0/22
- 185.199.108.0/22
- 140.82.112.0/20
tasks:
- name: populate vars
set_fact:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ hooks }}"
- name: print vars
debug:
msg: "variable_1: {{variable_1}}, variable_2: {{variable_2}}, variable_3: {{variable_3}}"
and the output:
[root#optima-ansible ILIAS]# ansible-playbook 50257063.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [populate vars] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.30.252.0/22)
ok: [localhost] => (item=185.199.108.0/22)
ok: [localhost] => (item=140.82.112.0/20)
TASK [print vars] **********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "variable_1: 192.30.252.0/22, variable_2: 185.199.108.0/22, variable_3: 140.82.112.0/20"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[root#optima-ansible ILIAS]#
hope it helps
UPDATE:
something weird i noticed - also TIL - is that if you reverse the lines:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
to:
counter: "{{ counter | int + 1 }}"
variable_{{counter}}: "{{ item }}"
you still end up with the same variable names, _1 to _3, while i would expect to get _2 to _4.
I guess ansible loops behave differently than expected from other programming languages.
---
- name: Query Github Meta API and get Hook Ips
hosts: local
connection: local
vars:
counter: 1
tasks:
- name: Query API
uri:
url: https://api.github.com/meta
return_content: yes
register: response
- name: Populate Hook Variables
set_fact:
webhook_ip_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ response['json']['hooks'] }}"
- name: print vars
debug:
msg: "Variable_1: {{ webhook_ip_1 }}, Variable_2: {{ webhook_ip_2 }}, Variable_3: {{ webhook_ip_3 }}"
Works with GitHub Webhook IPs in a loop
- name: get request to github
uri:
url: "https://api.github.com/meta"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
#X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: GitHub webhook IPs
debug:
msg: "{{ item }}"
with_items: "{{ (json_response.content | from_json).hooks }}"