I'm working on a playbook and I'm waiting for this playbook to collect web requests and generate an output. Since I am making requests to multiple interfaces I need to split that. How could I do that?
tasks:
- name: Collecting Endpoints
uri:
url: "https://applicationserver.com/api/app_set"
method: GET
return_content: yes
validate_certs : no
body_format: json
register: Result
- name: Temp to variable.
set_fact:
Endpoint: "{{ Result.json | json_query('[].name[]') }}"
Here is my response from the first task https://paste.chapril.org/?142535f7072d47ed#4TahPDaDGPkP1NxxB3gzZP86SkusJ9Kk6jSfqmSy7wK7
According to the data I have collected from the first task, I want to request each one-by-one filter data from within.
This is the JSON response of any of my endpoints. https://paste.chapril.org/?44e8f5462b105285#E2J7LzRwWSCJXR8BetzB1pewnV2ye1Fr1g741VacwBJb
- name: Get Responses All.
uri:
url: "https://applicationserver.com/api/app_set/{{ item }}"
method: GET
return_content: yes
body_format: json
validate_certs : no
register: Resultsofall
loop: "{{ Endpoint }}"
After this task, I am receiving all Json datas of all endpoints.
How can I filter the data from the whole result and loop like that?
app_sets
Index 1 (Which is counting by endpoint)
AppSetName (Getting from the first task (json_query('[].name[]')
StatefulAppSetName (Getting from the first task( ('[].labels[].AttachedToStatefulAppSet[]'))
Apps (Getting from the second task json_query('[].json.pods.app')
Index 2 (Which is counting by endpoint)
AppSetName (Getting from the first task (json_query('[].name[]')
StatefulAppSetName (Getting from the first task( ('[].labels[].AttachedToStatefulAppSet[]'))
Apps (Getting from the second task json_query('[].json.pods.app')
You need to get your json_query right, which is based on jmespath. Assuming you're interested on the name of the your services in your json your query should be:
- name: Temp to variable.
set_fact:
Endpoint: "{{ Result.json | json_query('services[].app') }}"
If you are interested in services and pods you could de something like
- name: Temp to variable.
set_fact:
services_apps: "{{ Result.json | json_query('services[].app') }}"
pods_apps: "{{ Result.json | json_query('services[].app') }}"
- name: Get Responses All.
uri:
url: "https://applicationserver.com/api/app_set/{{ item }}"
method: GET
return_content: yes
body_format: json
validate_certs : no
register: Resultsofall
loop: "{{ (services_apps + pods_apps )| unique | select | list }}"
I recommend using https://jmespath.org/ to develop and test you queries.
Related
I'm working on a playbook that needs to make some webrequests which is getting back JSON output. This works awesome for single requests, but I need to loop through a list to get a list of docker images.
Here is what I got so far:
- name: get all repositories from project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories?page=1&page_size=50"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
register: repositories
- name: temp to variable
set_fact:
repo_list_tmp: "{{ repositories.json | json_query('[*].name') | list }}"
- name: set repos as a list and trim names
set_fact:
repo_list: "{{ repo_list|default([]) + [ item.split(seperator)[1] ] }}"
loop: "{{ repo_list_tmp }}"
- name: get all images from a project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories/{{ item }}/artifacts?page=1&page_size=50&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: "application/json"
loop: "{{ repo_list }}"
register: images
The last task loops through the repo_list variable and register it's output to "images", as Ansible works the register in a loop always creates a dictionary and here's the proble, I got no clue on how to get specific values with a json query or something else out of it.
As the output is too big for stackoverflow I added it to a pastebin:
https://pastebin.com/GzQJfBUL
This is the whole output from the URI task. What I need is to extract the following:
item (in this example myimage or myotherimage) -> tags -> name
For example, inside the output is this entry:
"tags": [
{
"artifact_id": 4776,
"id": 2373,
"immutable": false,
"name": "MYTAG_ELASTICSEARCH",
"pull_time": "0001-01-01T00:00:00.000Z",
"push_time": "2021-09-21T11:33:47.687Z",
"repository_id": 303,
"signed": false
}
How do I extract values from a looped registerd value?
Thank you
Dan
According to mdaniel's comment below the first post, this did the trick:
{{ images | json_query('results[*].json[*].tags[*].name') | flatten }}
Thank you!
I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"
There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params
Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"
I have some ansible code involving a json_query in a loop that I'm trying to make conditional. But I think there is something I'm not understanding about when clauses and loops.
The following code works fine, when the condition is met (IaC.status == 400), the problem is that when the condition is not met, the 2nd task still runs, the loop tries to process IaC2, which doesn't exist, and the task fails with: "Invalid data passed to 'loop', it requires a list, got this instead:
I think this is actually expected behaviour for conditional statements and loops: (https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#using-conditionals-in-loops), but the solution described to skip the entire task is to use an empty iterator, but I have no idea how to merge that in with the json_query() statement.
Does anyone know how I can truly make the 2nd task below conditional?
# The preceding code has made a RESTful API call to create a gitlab group, if the group
# exists the return status is 400, so I need to look it up instead. In which case the
# following code works fine. The problem is when IaC.status = 201 for some reason the loop
# in the 2nd task below tries to run, despite the when clause, and fails
# because the variable IaC2 doesn't exist
- name: or if IaC already exists
when: IaC.status == 400
uri:
url: https://{{new_hostname}}/api/v4/groups
method: GET
headers:
Authorization: "Bearer {{token.json.access_token}}"
body_format: json
body:
name: "IaC"
top_level_only: true
status_code: 200
validate_certs: yes
register: IaC2
- name: json_query to find the IaC group id when status = "{{IaC.status}}"
when: IaC.status == 400
debug:
var: item
loop: "{{ IaC2 | community.general.json_query(jmesquery) }}"
vars:
jmesquery: "json[?name=='IaC'].id"
register: group_id
You should condition your second task on the first task:
- name: json_query to find the IaC group id when status = "{{IaC.status}}"
when: IaC2 is defined
debug:
var: item
loop: "{{ IaC2 | community.general.json_query(jmesquery) }}"
vars:
jmesquery: "json[?name=='IaC'].id"
register: group_id
I cannot reproduce your case on ansible 2.9.10 where the task is skipped as expected. But you can also help with a default. I guess this would work:
- name: json_query to find the IaC group id when status = "{{IaC.status}}"
when: IaC2 is defined
debug:
var: item
loop: "{{ IaC2 | community.general.json_query(jmesquery) | default([]) }}"
vars:
jmesquery: "json[?name=='IaC'].id"
register: group_id
I have an answer of sorts. Firstly there were two things I didn't fully understand;
the register var (IaC2) is always created even if the uri task is skipped. And IaC2 has a totally different structure depending upon whether this task is executed or skipped.
The loop in the following task is always executed, so the when clause on the second task is pretty meaningless.
The actual problem was the jmesquery looking for the IaC2 variable to have a particular json structure, which it doesn't have in the case where the uri task is skipped.
So the solution that seems to work is checking to see if the uri task was skipped (when: IaC2.skipped is defined), and if it was skipped re-defining IaC2 in a empty list form that doesn't break the loop.
Code examples shown below.
So, it works, but I can't help thinking there must be an easier way of doing this?
- name: If IaC already exists (IaC.status == 400) then get the list of existing groups.
when: IaC.status == 400
uri:
url: https://{{new_hostname}}/api/v4/groups
method: GET
headers:
Authorization: "Bearer {{token.json.access_token}}"
body_format: json
body:
name: "IaC"
top_level_only: true
status_code: 200
validate_certs: no
register: IaC2
# To avoid breaking the following loop that searches the groups in the case that
# the group was created, re-define IaC2 into a form that won't break it.
- name: Not created group "IaC", so redefine IaC2 var so as not to break the following loop
when: IaC2.skipped is defined
set_fact:
IaC2: '{{ {"json": []} }}'
- name: json_query to find the IaC group id when status = "{{IaC.status}}"
#when: IaC.status == 400 << Leave in or comment out, makes no difference
debug:
var: item
loop: "{{ (IaC2 | community.general.json_query(jmesquery)) }}"
vars:
jmesquery: "json[?name=='IaC'].id"
register: group_id
- name: record the IaC group's id (400)
when: IaC.status == 400
set_fact:
IaC_group_id: "{{group_id.results[0].item}}"
Something is wrong with the way I am substituting values and constructing a json template in ansible to be sent via the uri module to a webserver. Can anyone assist?
It seems like ansible is converting all the double brackets in the json to single brackets when it is running the replace command
Full Example:
I have a variable file where I am storing secrets:
ie secrets.yml
---
username: "admin"
password: "admin"
I am reading these in and replace the values in a json template:
ie template.json
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}
the playbook has tasks structured thus
- name: load template
set_fact:
loginTemplate: "{{ lookup('file', 'template.json') }}"
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) }}"
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate }}"
body_template: json
If I put the values in the json and just submit without the extra set_fact step to fill in the values it works.
Try converting the request body to json content using to_json filter in set_fact.
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) | to_json }}"
or in uri module,
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate | to_json }}"
body_template: json
Also, are you missing : in the json file like?
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}
I have a service call that returns system status in json format. I want to use the ansible URI module to make the call and then inspect the response to decide whether the system is up or down
{"id":"20161024140306","version":"5.6.1","status":"UP"}
This would be the json that is returned
This is the ansible task that makes a call:
- name: check sonar web is up
uri:
url: http://sonarhost:9000/sonar/api/system/status
method: GET
return_content: yes
status_code: 200
body_format: json
register: data
Question is how can I access data and inspect it as per ansible documentation this is how we store results of a call. I am not sure of the final step which is to check the status.
This works for me.
- name: check sonar web is up
uri:
url: http://sonarhost:9000/sonar/api/system/status
method: GET
return_content: yes
status_code: 200
body_format: json
register: result
until: result.json.status == "UP"
retries: 10
delay: 30
Notice that result is a ansible dictionary and when you set return_content=yes the response is added to this dictionary and is accessible using json key
Also ensure you have indented the task properly as shown above.
You've made the right first step by saving the output into a variable.
The next step is to use either when: or failed_when: statement in your next task, which will then switch based on the contents of the variable. There are a whole powerful set of statements for use in these, the Jinja2 builtin filters, but they are not really linked well into the Ansible documentation, or summarised nicely.
I use super explicitly named output variables, so they make sense to me later in the playbook :) I would probably write yours something like:
- name: check sonar web is up
uri:
url: http://sonarhost:9000/sonar/api/system/status
method: GET
return_content: yes
status_code: 200
body_format: json
register: sonar_web_api_status_output
- name: do this thing if it is NOT up
shell: echo "OMG it's not working!"
when: sonar_web_api_status_output.stdout.find('UP') == -1
That is, the text "UP" is not found in the variable's stdout.
Other Jinja2 builtin filters I've used are:
changed_when: "'<some text>' not in your_variable_name.stderr"
when: some_number_of_files_changed.stdout|int > 0
The Ansible "Conditionals" docs page has some of this info. This blog post was also very informative.
As per documentation at https://docs.ansible.com/ansible/latest/modules/uri_module.html
Whether or not to return the body of the response as a "content" key in the dictionary result. Independently of this option, if the reported Content-type is "application/json", then the JSON is always loaded into a key called json in the dictionary results.
---
- name: Example of JSON body parsing with uri module
connection: local
gather_facts: true
hosts: localhost
tasks:
- name: Example of JSON body parsing with uri module
uri:
url: https://jsonplaceholder.typicode.com/users
method: GET
return_content: yes
status_code: 200
body_format: json
register: data
# failed_when: <optional condition based on JSON returned content>
- name: Print returned json dictionary
debug:
var: data.json
- name: Print certain element
debug:
var: data.json[0].address.city