Ansible, error when reading values from JSON response - json

In my Ansible-Code I have a task which makes some requests to check if a Cluster is created. Response of the Request is a JSON file which contains the values "healthy" and "status" and some more.
This is my Code:
- name: Wait for Elastic Cluster to be ready
uri:
url: https://abcdefbla.{{ lookup('env','ENV') }}.some.url.com/api/v1/clusters/elasticsearch/{{elasticClusterDetails.elasticsearchId}}
method: GET
user: admin
password: "{{rootpw.stdout}}"
force_basic_auth: yes
return_content: yes
validate_certs: no
register: result
until: "{{ (result.stdout | from_json | first).healthy }}" == false and "{{ (result.stdout | from_json | first).status }}" == "started"
retries: 60
delay: 10
The from_json filter helps parsing the response, and the "first" tells that it should use the first key with that name.
When I run it, I get the following error:
ERROR! Syntax Error while loading YAML.
did not find expected key.
The error appears to have been in '/home/ubuntu/MLAAS-V2_USETHIS/mlaas-v2-aws/mls_ece_core/play.yml': line 295, column 65, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
register: result
until: "{{ (result.stdout | from_json | first).healthy }}" == false and "{{ (result.stdout | from_json | first).status }}" == "started"
^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value. For instance:
with_items:
- {{ foo }}
Should be written as:
with_items:
- "{{ foo }}"
As you see, I quoted the line as suggested. Any Ansible-Expert out there who can tell me what I am missing?!

I found the solution. There was an error with spacing for the URL object. Later, an error with JSON parsing came up which I also fixed. The following code worked for me:
- name: Wait for Elastic Cluster to be ready
uri:
url: https://abcdefbla.{{ lookup('env','ENV') }}.some.url.com/api/v1/clusters/elasticsearch/{{elasticClusterDetails.elasticsearchId}}
method: GET
user: admin
password: "{{rootpw.stdout}}"
force_basic_auth: yes
return_content: yes
validate_certs: no
register: result
until: resultA.json.healthy == false and "started" == "{{ resultA.json.status|lower }}"
retries: 60
delay: 10

Related

Ansible: How to parse looped registered variable?

I'm working on a playbook that needs to make some webrequests which is getting back JSON output. This works awesome for single requests, but I need to loop through a list to get a list of docker images.
Here is what I got so far:
- name: get all repositories from project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories?page=1&page_size=50"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
register: repositories
- name: temp to variable
set_fact:
repo_list_tmp: "{{ repositories.json | json_query('[*].name') | list }}"
- name: set repos as a list and trim names
set_fact:
repo_list: "{{ repo_list|default([]) + [ item.split(seperator)[1] ] }}"
loop: "{{ repo_list_tmp }}"
- name: get all images from a project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories/{{ item }}/artifacts?page=1&page_size=50&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: "application/json"
loop: "{{ repo_list }}"
register: images
The last task loops through the repo_list variable and register it's output to "images", as Ansible works the register in a loop always creates a dictionary and here's the proble, I got no clue on how to get specific values with a json query or something else out of it.
As the output is too big for stackoverflow I added it to a pastebin:
https://pastebin.com/GzQJfBUL
This is the whole output from the URI task. What I need is to extract the following:
item (in this example myimage or myotherimage) -> tags -> name
For example, inside the output is this entry:
"tags": [
{
"artifact_id": 4776,
"id": 2373,
"immutable": false,
"name": "MYTAG_ELASTICSEARCH",
"pull_time": "0001-01-01T00:00:00.000Z",
"push_time": "2021-09-21T11:33:47.687Z",
"repository_id": 303,
"signed": false
}
How do I extract values from a looped registerd value?
Thank you
Dan
According to mdaniel's comment below the first post, this did the trick:
{{ images | json_query('results[*].json[*].tags[*].name') | flatten }}
Thank you!

Extract part of JSON in ansible playbook

I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"
There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params
Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"

How to use a url from a json formatted answer with the get_url module?

I am requesting a REST API to provide me with a download url for a specific item (this is solved by the uri-module). I then try to use the url in the get_url module but it fails with the following message
FAILED! => {"msg": "template error while templating string: expected name or number. String: {{ cora_response.json.['#odata.mediaReadLink'].stdout | from_json }}"}
How am i suppose to put the variable into the url parameter?
My task looks currently like this:
- name: Download Link from REST
debug: var=cora_response.json['#odata.mediaReadLink']
- name: Download the latest Release
get_url:
url: "{{ cora_response.json.['#odata.mediaReadLink'].stdout | from_json }}"
dest: "{{ installation_path }}/install/19.3.zip"
mode: 660
My debug task shows me the url i am trying to use, however the get_url module fails.
Please advise and thank you in advance.
Edit:
Tried the following suggested by #al76
- name: Download the latest CH Release
get_url:
url: "{{ cora_response.json['#odata.mediaReadLink'].stdout | from_json }}"
dest: "{{ installation_path }}/install/CH19.3.zip"
mode: 660
Error message:
fatal: [ln-lnxcelcon01.owo.company]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ cora_response.json['#odata.mediaReadLink'].stdout | from_json }}): expected string or buffer"}
try
url: "{{ cora_response.json['#odata.mediaReadLink'].stdout | from_json }}"
Blind luck took me out of my misery
url: "{{ cora_response.json['#odata.mediaReadLink'] }}"

Why is Ansible unable to read unicode string as JSON?

Summary
When retrieving data using the uri module in Ansible, I am unable to parse a section of it as JSON to retrieve a nested value.
The desired value is the ci field inside the content.data or json.data field (see output below).
Steps to Reproduce
site.yml
---
- hosts: localhost
gather_facts: false
tasks:
- name: Get String
uri:
url: "http://localhost/get-data"
method: POST
body_format: json
body: "{ \"kong-jid\": \"run-sn-discovery\" }"
return_content: yes
register: output
- set_fact:
ci: "{{ output.json.data.ci }}"
- debug:
msg: "{{ ci }}"
The {{ output }} variable
{
u'status': 200,
u'cookies': {},
u'url': u'http://kong-demo:8000/get-data',
u'transfer_encoding': u'chunked',
u'changed': False,
u'connection': u'close',
u'server': u'kong/0.34-1-enterprise-edition',
u'content':
u'{"data":"\\"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}\\""}',
'failed': False,
u'json': {u'data': u'"{u\'ci\': u\'3bb8d625dbac3700e4f07b6e0f96195b\'}"'},
u'content_type': u'application/json',
u'date': u'Thu, 18 Apr 2019 15:50:25 GMT',
u'redirected': False,
u'msg': u'OK (unknown bytes)'
}
Result
[user#localhost]$ ansible-playbook site.yml
[WARNING]: Could not match supplied host pattern, ignoring: all
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [localhost] ***************************************************************************************************************
TASK [Pass Redis data to next task as output] **********************************************************************************
ok: [localhost]
TASK [set_fact] ****************************************************************************************************************
fatal: [localhost]: FAILED! => {}
MSG:
The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
The error appears to have been in 'site.yml': line 19, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- set_fact:
^ here
exception type: <class 'ansible.errors.AnsibleUndefinedVariable'>
exception: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ci'
Important Troubleshooting Information
It appears the root issue is related to which Ansible type being interpreted. I desire to parse ci from the output in one task.
The two-task solution shown below works, but this leads me to believe this should be possible in one line...
Two-Task Solution
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci['ci'] }}"
But the ci fact set from {{ output.json.data | from_json }} reports a different TYPE than the inline type...
Unicode or Dict?
- debug:
msg: "{{ output.json.data | from_json | type_debug }}" # returns unicode
- set_fact:
ci: "{{ output.json.data | from_json }}"
- debug:
msg: "{{ ci | type_debug }}" # returns dict
Why isn't {{ output.json.data | from_json | type_debug }}
the same as {{ ci | type_debug }}?
Although json and data are keys in their resp objects, ci is just part of a larger string (which happens to look like a JSON object
If the relevant line in your datastructure would be:
u'json': {u'data': {'ci': u'3bb8d625dbac3700e4f07b6e0f96195b'}},
then you could expect to use "{{ output.json.data.ci }}" but not when the .ci part is just a normal part of a string.

Ansible parse json and read result into different variables

I've set up a task which queries the github api meta endpoint and returns the following
{
"verifiable_password_authentication": true,
"github_services_sha": "f9e3a6b98d76d9964a6613d581164039b8d54d89",
"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
],
"git": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20",
"13.229.188.59/32",
"13.250.177.223/32",
"18.194.104.89/32",
"18.195.85.27/32",
"35.159.8.160/32",
"52.74.223.119/32"
],
"pages": [
"192.30.252.153/32",
"192.30.252.154/32",
"185.199.108.153/32",
"185.199.109.153/32",
"185.199.110.153/32",
"185.199.111.153/32"
],
"importer": [
"54.87.5.173",
"54.166.52.62",
"23.20.92.3"
]
}
What I need to do is get the 3 hook IPs and read them each into their own variable.
I've tried a couple of solutions i've found around but nothing is seeming to work for me.
I've got as far as drilling down into the json so i'm being returned only the 3 IPs, but how do I get them out and into variables individually?
i gave it a shot using j2 syntax in the variable name part, and - TIL - looks like the jinja2 syntax is allowed in that part as well!
please see playbook to process the hooks list variable and assign to variables variable_1, variable_2, variable_3 and so on:
- hosts: localhost
gather_facts: false
vars:
counter: 1
hooks:
- 192.30.252.0/22
- 185.199.108.0/22
- 140.82.112.0/20
tasks:
- name: populate vars
set_fact:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ hooks }}"
- name: print vars
debug:
msg: "variable_1: {{variable_1}}, variable_2: {{variable_2}}, variable_3: {{variable_3}}"
and the output:
[root#optima-ansible ILIAS]# ansible-playbook 50257063.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [populate vars] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.30.252.0/22)
ok: [localhost] => (item=185.199.108.0/22)
ok: [localhost] => (item=140.82.112.0/20)
TASK [print vars] **********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "variable_1: 192.30.252.0/22, variable_2: 185.199.108.0/22, variable_3: 140.82.112.0/20"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[root#optima-ansible ILIAS]#
hope it helps
UPDATE:
something weird i noticed - also TIL - is that if you reverse the lines:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
to:
counter: "{{ counter | int + 1 }}"
variable_{{counter}}: "{{ item }}"
you still end up with the same variable names, _1 to _3, while i would expect to get _2 to _4.
I guess ansible loops behave differently than expected from other programming languages.
---
- name: Query Github Meta API and get Hook Ips
hosts: local
connection: local
vars:
counter: 1
tasks:
- name: Query API
uri:
url: https://api.github.com/meta
return_content: yes
register: response
- name: Populate Hook Variables
set_fact:
webhook_ip_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ response['json']['hooks'] }}"
- name: print vars
debug:
msg: "Variable_1: {{ webhook_ip_1 }}, Variable_2: {{ webhook_ip_2 }}, Variable_3: {{ webhook_ip_3 }}"
Works with GitHub Webhook IPs in a loop
- name: get request to github
uri:
url: "https://api.github.com/meta"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
#X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: GitHub webhook IPs
debug:
msg: "{{ item }}"
with_items: "{{ (json_response.content | from_json).hooks }}"