Ansible: How to parse looped registered variable? - json

I'm working on a playbook that needs to make some webrequests which is getting back JSON output. This works awesome for single requests, but I need to loop through a list to get a list of docker images.
Here is what I got so far:
- name: get all repositories from project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories?page=1&page_size=50"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
register: repositories
- name: temp to variable
set_fact:
repo_list_tmp: "{{ repositories.json | json_query('[*].name') | list }}"
- name: set repos as a list and trim names
set_fact:
repo_list: "{{ repo_list|default([]) + [ item.split(seperator)[1] ] }}"
loop: "{{ repo_list_tmp }}"
- name: get all images from a project
uri:
url: "https://{{ harbor_url }}/api/v2.0/projects/{{ harbor_project }}/repositories/{{ item }}/artifacts?page=1&page_size=50&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false"
return_content: yes
validate_certs: no
url_username: "{{ username }}"
url_password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: "application/json"
loop: "{{ repo_list }}"
register: images
The last task loops through the repo_list variable and register it's output to "images", as Ansible works the register in a loop always creates a dictionary and here's the proble, I got no clue on how to get specific values with a json query or something else out of it.
As the output is too big for stackoverflow I added it to a pastebin:
https://pastebin.com/GzQJfBUL
This is the whole output from the URI task. What I need is to extract the following:
item (in this example myimage or myotherimage) -> tags -> name
For example, inside the output is this entry:
"tags": [
{
"artifact_id": 4776,
"id": 2373,
"immutable": false,
"name": "MYTAG_ELASTICSEARCH",
"pull_time": "0001-01-01T00:00:00.000Z",
"push_time": "2021-09-21T11:33:47.687Z",
"repository_id": 303,
"signed": false
}
How do I extract values from a looped registerd value?
Thank you
Dan

According to mdaniel's comment below the first post, this did the trick:
{{ images | json_query('results[*].json[*].tags[*].name') | flatten }}
Thank you!

Related

Extract part of JSON in ansible playbook

I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"
There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params
Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"

Ansible using variables properly for uri request

Something is wrong with the way I am substituting values and constructing a json template in ansible to be sent via the uri module to a webserver. Can anyone assist?
It seems like ansible is converting all the double brackets in the json to single brackets when it is running the replace command
Full Example:
I have a variable file where I am storing secrets:
ie secrets.yml
---
username: "admin"
password: "admin"
I am reading these in and replace the values in a json template:
ie template.json
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}
the playbook has tasks structured thus
- name: load template
set_fact:
loginTemplate: "{{ lookup('file', 'template.json') }}"
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) }}"
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate }}"
body_template: json
If I put the values in the json and just submit without the extra set_fact step to fill in the values it works.
Try converting the request body to json content using to_json filter in set_fact.
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) | to_json }}"
or in uri module,
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate | to_json }}"
body_template: json
Also, are you missing : in the json file like?
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}

Substitution of variables from JSON response (from uri) in ansible playbook to create local users account

I have JSON as output for query into CMDB, which contains informations about local users like UID, GID etc. And I would like to substitute them in ansible playbook to create those account. I know how to use it as one variable. But I would like to use it for more variables as parameters - one value from key for UID, GID, ssh public key etc. Example of JSON is attached.
Any suggestion or help?
JSON answer
$ curl -s -k "https://cmdb.test.loc/api/getusers.php" | jq -r '.'
{
"users": [
{
"login": "johnd",
"description": "John Doe",
"uid": "9004",
"sshpubkey": "ssh-rsa AAsQxHUC4mshf+mXy3+Evtd1l9/x/DF5tR1AMI9i0CQoggxX6GbxRb+lYVySUmCgnMqT/dVcOlL3ETzSnubeoC3o4Yjv9Q1NLrw== johnd#lin.x"
},
{
"login": "thomasm",
"description": "Thomas Mann",
"uid": "9003",
"sshpubkey": "ssh-rsa AAAdUG4PPLzrJo/M5StBXsgUcUQAPuU4y1vXIJIw+o5mx/IkxpZVvbWTqx6HbLJD8CL5jp4wpXb1UGjyYorvumrGyArYWAXWOQZNdzsIeTp1Z9tZb0qoVkGX1aR1HGst36FbcYDTB3yEmliYHSNQ== thomasm#lin.x"
},
{
"login": "trib",
"description": "Three Be",
"uid": "9002",
"sshpubkey": "ssh-rsa AAAAB3NzpyNyRsdvthGYzkm0rvEug1RRwURWZ9d++TzJtoWZrCKw7Ec8PJGgNJvHRAvFdSJ8mKXAK3U0WIc46P2Ij7BJzCR4o900Sad/qeYSlY7xL+oW+nXWPUuxCVT25pHAeTJDjHeHJmpqDpHGg8sLmQMXtwCb5r6YxFUOUSlM1L2Q2iVedC/IXKMTFP+7Wp9YvPX75MVjMWc6L1xCZMsBk+dQ== 3be#lin.x"
}
]
}
Begin of playbook.yml
---
- hosts: all
tasks:
- name: Take information about actual users from CMDB
uri:
url: "https://cmdb.test.loc/api/getusers.php"
validate_certs: false
method: GET
status_code: 200,400
body_format: json
return_content: yes
register: cmdb_users
- name: Filter and rename variables from source list
debug:
msg: "{{ cmdb_users.json | json_query('users[*].{login: login, description: description, uid: uid, sshpubkey: sshpubkey}') }}"
- set_fact:
users: "{{ cmdb_users.json | json_query('users[*].{login: login, description: description, uid: uid, sshpubkey: sshpubkey}') }}"
...
I would like have any suggestion for next steps to use it for account creating:
- name : Create local users
user:
name : {{ ??? }}
uid : {{ ??? }}
comment : {{ ??? }}
I know that authorized-keys file have to be created by other way, but I think, that it can be created as shell "echo" to specific file to specific path composed from variables.
The loop below should do the job
- name: Create local users
user:
name: "{{ item.login }}"
uid: "{{ item.uid|int }}"
comment: "{{ item.description }}"
loop: "{{ users }}"
Next loop will store the public keys to the files in pub_key_dir
- name: Store public keys to files
copy:
dest: "{{ pub_key_dir }}/{{ item.login }}.sshpubkey"
content: |
{{ item.sshpubkey }}
loop: "{{ users }}"
Notes
1) The correct name of the module is user
2) In the playbook, the registered variable is cmdb_users. The correct list is probably
loop: "{{ cmdb_users.json.users }}"

Ansible Extract JSON Tag

I'm trying to work with Infoblox API, and it's responses. I would need to extract values of tags from the response, that seems to be in JSON format, but I cannot find the way to do it.
Here is my playbook:
- name: "Checking _node_exporter Service Record for {{ inventory_hostname }}"
local_action:
module: uri
url: "{{ infobloxapiurl }}record:srv?name=_node_exporter.domain.com&target={{ inventory_hostname }}"
force_basic_auth: yes
user: "{{ infobloxuser }}"
password: "{{ infobloxpassword }}"
validate_certs: no
return_content: yes
register: _infoblox_results
- debug:
var: _infoblox_results.json
The _infoblox_results.json variable looks like this:
TASK [prometheus : debug] *******************************************************************************************************************************************************************************************
task path: /ansible/roles/tasks/task.yml:38
ok: [server.domain.com] => {
"_infoblox_results.json": [
{
"_ref": "record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default",
"name": "_node_exporter.domain.com",
"port": 9100,
"priority": 0,
"target": "server.domain.com",
"view": "default",
"weight": 0
}
]
}
I want to use the data of _ref from _infoblox_results.json, but I wasn't able to extract it with regex_replace (it drops back the full _infoblox_results.json):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | regex_replace('record:srv.*\\/default,', '\\1') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Neither with json_query (it drops back nothing):
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json | json_query('_ref') }}"
- debug:
var: _rcdid
when: _infoblox_results.json != []
Can someone please point me into the right direction?
You have already an object in the memory, so simply refer to its value: _infoblox_results.json[0]._ref contains the string record:srv/ZG5zLmJpbmRfc3J2JC5fZGVmYXVsdC5jb20udmNpbnQuZXcxL19ub2RlX2V4cG9ydGVyLzAvMC85MTAwL3Zhcm5pc2g3MDJ0c3QuZXcxLnZjaW50LmNvbQ:_node_exporter.domain.com/default.
With that you can split the string and select the second element:
- name: Get Record ID
set_fact:
_rcdid: "{{ _infoblox_results.json[0]._ref.split('/')[1] }}"

Ansible parse json and read result into different variables

I've set up a task which queries the github api meta endpoint and returns the following
{
"verifiable_password_authentication": true,
"github_services_sha": "f9e3a6b98d76d9964a6613d581164039b8d54d89",
"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
],
"git": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20",
"13.229.188.59/32",
"13.250.177.223/32",
"18.194.104.89/32",
"18.195.85.27/32",
"35.159.8.160/32",
"52.74.223.119/32"
],
"pages": [
"192.30.252.153/32",
"192.30.252.154/32",
"185.199.108.153/32",
"185.199.109.153/32",
"185.199.110.153/32",
"185.199.111.153/32"
],
"importer": [
"54.87.5.173",
"54.166.52.62",
"23.20.92.3"
]
}
What I need to do is get the 3 hook IPs and read them each into their own variable.
I've tried a couple of solutions i've found around but nothing is seeming to work for me.
I've got as far as drilling down into the json so i'm being returned only the 3 IPs, but how do I get them out and into variables individually?
i gave it a shot using j2 syntax in the variable name part, and - TIL - looks like the jinja2 syntax is allowed in that part as well!
please see playbook to process the hooks list variable and assign to variables variable_1, variable_2, variable_3 and so on:
- hosts: localhost
gather_facts: false
vars:
counter: 1
hooks:
- 192.30.252.0/22
- 185.199.108.0/22
- 140.82.112.0/20
tasks:
- name: populate vars
set_fact:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ hooks }}"
- name: print vars
debug:
msg: "variable_1: {{variable_1}}, variable_2: {{variable_2}}, variable_3: {{variable_3}}"
and the output:
[root#optima-ansible ILIAS]# ansible-playbook 50257063.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [populate vars] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.30.252.0/22)
ok: [localhost] => (item=185.199.108.0/22)
ok: [localhost] => (item=140.82.112.0/20)
TASK [print vars] **********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "variable_1: 192.30.252.0/22, variable_2: 185.199.108.0/22, variable_3: 140.82.112.0/20"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[root#optima-ansible ILIAS]#
hope it helps
UPDATE:
something weird i noticed - also TIL - is that if you reverse the lines:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
to:
counter: "{{ counter | int + 1 }}"
variable_{{counter}}: "{{ item }}"
you still end up with the same variable names, _1 to _3, while i would expect to get _2 to _4.
I guess ansible loops behave differently than expected from other programming languages.
---
- name: Query Github Meta API and get Hook Ips
hosts: local
connection: local
vars:
counter: 1
tasks:
- name: Query API
uri:
url: https://api.github.com/meta
return_content: yes
register: response
- name: Populate Hook Variables
set_fact:
webhook_ip_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ response['json']['hooks'] }}"
- name: print vars
debug:
msg: "Variable_1: {{ webhook_ip_1 }}, Variable_2: {{ webhook_ip_2 }}, Variable_3: {{ webhook_ip_3 }}"
Works with GitHub Webhook IPs in a loop
- name: get request to github
uri:
url: "https://api.github.com/meta"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
#X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: GitHub webhook IPs
debug:
msg: "{{ item }}"
with_items: "{{ (json_response.content | from_json).hooks }}"