Ansible using variables properly for uri request - json

Something is wrong with the way I am substituting values and constructing a json template in ansible to be sent via the uri module to a webserver. Can anyone assist?
It seems like ansible is converting all the double brackets in the json to single brackets when it is running the replace command
Full Example:
I have a variable file where I am storing secrets:
ie secrets.yml
---
username: "admin"
password: "admin"
I am reading these in and replace the values in a json template:
ie template.json
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}
the playbook has tasks structured thus
- name: load template
set_fact:
loginTemplate: "{{ lookup('file', 'template.json') }}"
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) }}"
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate }}"
body_template: json
If I put the values in the json and just submit without the extra set_fact step to fill in the values it works.

Try converting the request body to json content using to_json filter in set_fact.
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) | to_json }}"
or in uri module,
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate | to_json }}"
body_template: json
Also, are you missing : in the json file like?
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}

Related

How to Loop Parsed Json Response on Ansible

I'm working on a playbook and I'm waiting for this playbook to collect web requests and generate an output. Since I am making requests to multiple interfaces I need to split that. How could I do that?
tasks:
- name: Collecting Endpoints
uri:
url: "https://applicationserver.com/api/app_set"
method: GET
return_content: yes
validate_certs : no
body_format: json
register: Result
- name: Temp to variable.
set_fact:
Endpoint: "{{ Result.json | json_query('[].name[]') }}"
Here is my response from the first task https://paste.chapril.org/?142535f7072d47ed#4TahPDaDGPkP1NxxB3gzZP86SkusJ9Kk6jSfqmSy7wK7
According to the data I have collected from the first task, I want to request each one-by-one filter data from within.
This is the JSON response of any of my endpoints. https://paste.chapril.org/?44e8f5462b105285#E2J7LzRwWSCJXR8BetzB1pewnV2ye1Fr1g741VacwBJb
- name: Get Responses All.
uri:
url: "https://applicationserver.com/api/app_set/{{ item }}"
method: GET
return_content: yes
body_format: json
validate_certs : no
register: Resultsofall
loop: "{{ Endpoint }}"
After this task, I am receiving all Json datas of all endpoints.
How can I filter the data from the whole result and loop like that?
app_sets
Index 1 (Which is counting by endpoint)
AppSetName (Getting from the first task (json_query('[].name[]')
StatefulAppSetName (Getting from the first task( ('[].labels[].AttachedToStatefulAppSet[]'))
Apps (Getting from the second task json_query('[].json.pods.app')
Index 2 (Which is counting by endpoint)
AppSetName (Getting from the first task (json_query('[].name[]')
StatefulAppSetName (Getting from the first task( ('[].labels[].AttachedToStatefulAppSet[]'))
Apps (Getting from the second task json_query('[].json.pods.app')
You need to get your json_query right, which is based on jmespath. Assuming you're interested on the name of the your services in your json your query should be:
- name: Temp to variable.
set_fact:
Endpoint: "{{ Result.json | json_query('services[].app') }}"
If you are interested in services and pods you could de something like
- name: Temp to variable.
set_fact:
services_apps: "{{ Result.json | json_query('services[].app') }}"
pods_apps: "{{ Result.json | json_query('services[].app') }}"
- name: Get Responses All.
uri:
url: "https://applicationserver.com/api/app_set/{{ item }}"
method: GET
return_content: yes
body_format: json
validate_certs : no
register: Resultsofall
loop: "{{ (services_apps + pods_apps )| unique | select | list }}"
I recommend using https://jmespath.org/ to develop and test you queries.

ansible - parsing json w jmesquery

I'm querying Splunk via the uri module w a certain search:
- name: splunk query
uri:
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=json
method: POST
user: ...
password: ...
validate_certs: false
body_format: form-urlencoded
return_content: true
headers:
Content-Type: application/json
body:
- [ search, "{{ splunk_search }}" ]
vars:
- splunk_search: '| ...'
register: me
Splunk returns a 200, w content:
TASK [debug] ********************************************************************************************************************************************************************************************
ok: [host1] => {
"msg": "{\n \"title\": \"clientMessageId\",\n \"app\": \"whatever\"\n}\n{\n \"title\": \"a_title\",\n \"app\": \"some_app\"\n}\n{\n \"title\": \"another_title\",\n \"app\": \"my_app\"\n}\n{\n \"title\": \"title_for_app\",\n \"app\": \"another_app\"\n}"
}
However, I can't properly parse the output, I've tried:
- name: query result from content
debug:
msg: "{{ me.content | json_query('title') }}"
In this case, Ansible will return an empty string. I assume something is wrong with the formatting of the output.
Q: How can I configure Ansible to properly parse the json output, so that a new list is created based on the output.
Example:
my_list:
- title: clientMessageId
app: whatever
- title: a_title
app: some_app
...
What I did was change the url output_mode in which Splunk returns the content from json to csv.
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=csv
Then, I wrote the output to a file and read it using the ansible read_csv module.
- name: write received content to csv file on controller
copy:
content: "{{ me.content }}"
dest: "./file.csv"
mode: '0775'
delegate_to: localhost
- name: read csv file
read_csv:
path: ./file.csv
delegate_to: localhost
register: read_csv
- name: set fact to create list
set_fact:
the_output: "{{ read_csv.list }}"

Ansible: How to parse looped registered variable?

I'm working on a playbook that needs to make some webrequests which is getting back JSON output. This works awesome for single requests, but I need to loop through a list to get a list of docker images.
Here is what I got so far:
- name: get all repositories from project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories?page=1&page_size=50"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
register: repositories
- name: temp to variable
set_fact:
repo_list_tmp: "{{ repositories.json | json_query('[*].name') | list }}"
- name: set repos as a list and trim names
set_fact:
repo_list: "{{ repo_list|default([]) + [ item.split(seperator)[1] ] }}"
loop: "{{ repo_list_tmp }}"
- name: get all images from a project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories/{{ item }}/artifacts?page=1&page_size=50&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: "application/json"
loop: "{{ repo_list }}"
register: images
The last task loops through the repo_list variable and register it's output to "images", as Ansible works the register in a loop always creates a dictionary and here's the proble, I got no clue on how to get specific values with a json query or something else out of it.
As the output is too big for stackoverflow I added it to a pastebin:
https://pastebin.com/GzQJfBUL
This is the whole output from the URI task. What I need is to extract the following:
item (in this example myimage or myotherimage) -> tags -> name
For example, inside the output is this entry:
"tags": [
{
"artifact_id": 4776,
"id": 2373,
"immutable": false,
"name": "MYTAG_ELASTICSEARCH",
"pull_time": "0001-01-01T00:00:00.000Z",
"push_time": "2021-09-21T11:33:47.687Z",
"repository_id": 303,
"signed": false
}
How do I extract values from a looped registerd value?
Thank you
Dan
According to mdaniel's comment below the first post, this did the trick:
{{ images | json_query('results[*].json[*].tags[*].name') | flatten }}
Thank you!

Extract part of JSON in ansible playbook

I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"
There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params
Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"

How to POST Json file with arguments using Ansible

All
I am using Ansible to POST data on a website. I have taken a very simple example of the JSON content
{
"test2":"{{test}}"
}
Below is snippet for ansible task responsible to POST data:
vars:
test: "somerandomtest"
tasks:
- name: Test
register: json_output
uri:
url: "http://localhost:3000"
validate_certs: no
follow_redirects: none
headers:
Authorization: Bearer 12344
Content-Type: application/json
method: POST
body_format: json
body: " {{ lookup('file','test.json') | from_json }} "
#body: '{"test2":" {{test}} "}'
timeout: 600
But on the web service I see the below data, where "test" is not replaced with "somerandomtest".
{'test2': '{{test}}'}
Not sure what I am doing wrong, or maybe I am using lookup the wrong way. If I replace the body with the content everything is fine, but the real JSON I work with is 400 lines, and I want to keep it separate.
Can anyone please let me know what can I do to change the JSON data using Ansible?
The file lookup doesn't evaluate file content for Jinja template markers (which would lead to unexpected behavior if a document just happened to contain {{).
If you want to read in a template, use the template lookup instead:
- hosts: localhost
gather_facts: false
tasks:
vars:
test: "somerandomtest"
tasks:
- name: Test
register: json_output
uri:
url: "https://httpbin.org/post"
validate_certs: no
follow_redirects: none
headers:
Content-Type: application/json
method: POST
body_format: json
body: " {{ lookup('template','test.json') }} "
return_content: true
timeout: 600
- debug:
var: json_output
This shows that the content sent to the remote server is:
{'test2': 'somerandomtest'}
Put the hash into a dictionary and format it to_json. For example
- hosts: localhost
vars:
test: somerandomtest
body:
test2: "{{ test }}"
tasks:
- debug:
msg: "{{ body|to_json }}"
gives the JSON
msg: '{"test2": "somerandomtest"}'
The template gives the same result
shell> cat test.json
{
"test2": "{{ test }}"
}
- debug:
msg: "{{ lookup('template','test.json')|to_json }}"