converting string to integer in ansible - json

In ansible i am running a role to create retention rules for some projects. using PUT method i stored results in register module, from that i am trying to read retention_id: "59" and project_id: "80" i need this as integer like 59 and 80 , but it is taking as string to the field but i need it as integer. i tried to change that by using "{{ item.project_id | int }}"
and {{ item.metadata.retention_id | int}}
it was not changed to integer, i tried to converted the yaml body to JSON for luck. still i am getting the error.
i got struck with this task since one week . please provide any solution to get out of this.
your response will be appreciated. Thank you
actual playbook:
---
# List of projects to check retention and project ID's
- name: get projects
uri:
url: "https://{{ ansible_host }}/api/v2.0/projects"
method: GET
headers:
Authorization: "{{ admin_passwd }}"
Accept: application/json
validate_certs: no
register: projects_result
- debug:
var: projects_result.json | flatten(levels=1)
# Create retention rules for the projects
- name: creating retention rules
uri:
url: "https://{{ ansible_host }}/api/v2.0/retentions/{{ item.metadata.retention_id | int}}"
method: PUT
body_format: json
headers:
Content-Type: application/json
Authorization: "{{ admin_passwd }}"
Accept: 'application/json, text/plain, */*'
body:
algorithm: "or"
id: "{{ item.metadata.retention_id | int }}"
rules:
- action: "retain"
params:
latestPushedK: 3
scope_selectors:
repository:
- decoration: "repoMatches"
kind: "doublestar"
pattern: "**"
tag_selectors:
- decoration: "matches"
extras: "{\"untagged\":true}"
kind: "doublestar"
pattern: "**"
template: latestPushedK
scope:
level: "project"
ref: "{{ item.project_id | int }}"
trigger:
kind: "Schedule"
settings:
cron: "0 0 * * * *"
validate_certs: no
status_code: 200
when: "item.registry_id is defined"
loop: "{{ projects_result.json | flatten(levels=1) }}"
ignore_errors: yes
with this i was getting this error:
$ ansible-playbook -i ansible/inventories/shiplab ansible/playbooks/harbor-configurations.yaml --ask-vault-pass
Vault password:
PLAY [harbor] ***********************************************************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************************************************
ok: [harbor_shiplab_test1e]
TASK [../roles/harbor-retention-rules : get projects] *******************************************************************************************************
ok: [harbor_shiplab_test1e]
TASK [../roles/harbor-retention-rules : debug] **************************************************************************************************************
ok: [harbor_shiplab_test1e] => {
"projects_result.json | flatten(levels=1)": [
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:28.228Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 56,
"items": [],
"project_id": 75,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "54"
},
"name": "harbor-esl-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 75,
"registry_id": 101,
"repo_count": 0,
"update_time": "2021-10-21T07:35:28.228Z"
},
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:43.098Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 58,
"items": [],
"project_id": 77,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "56"
},
"name": "harbor-library-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 77,
"registry_id": 103,
"repo_count": 0,
"update_time": "2021-10-21T07:35:43.098Z"
},
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:13.930Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 54,
"items": [],
"project_id": 73,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "52"
},
"name": "harbor-megatron-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 73,
"registry_id": 104,
"repo_count": 0,
"update_time": "2021-10-21T07:35:13.930Z"
},
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:50.271Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 59,
"items": [],
"project_id": 78,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "57"
},
"name": "harbor-migration-poc-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 78,
"registry_id": 105,
"repo_count": 0,
"update_time": "2021-10-21T07:35:50.271Z"
},
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:57.671Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 60,
"items": [],
"project_id": 79,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "58"
},
"name": "harbor-payment-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 79,
"registry_id": 106,
"repo_count": 0,
"update_time": "2021-10-21T07:35:57.671Z"
},
{
"chart_count": 0,
"creation_time": "2021-10-21T07:36:05.861Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 61,
"items": [],
"project_id": 80,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "59"
},
"name": "harbor-platform-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 80,
"registry_id": 107,
"repo_count": 0,
"update_time": "2021-10-21T07:36:05.861Z"
},
{
"chart_count": 1,
"creation_time": "2021-08-19T15:50:23.772Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 1,
"items": [],
"project_id": 1,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true"
},
"name": "library",
"owner_id": 1,
"owner_name": "admin",
"project_id": 1,
"repo_count": 3,
"update_time": "2021-08-19T15:50:23.772Z"
},
{
"chart_count": 2,
"creation_time": "2021-08-24T16:45:19.230Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 2,
"items": [],
"project_id": 2,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true"
},
"name": "platform",
"owner_id": 1,
"owner_name": "admin",
"project_id": 2,
"repo_count": 11,
"update_time": "2021-08-24T16:45:19.230Z"
}
]
}
TASK [../roles/harbor-retention-rules : creating retention rules] *******************************************************************************************
failed: [harbor_shiplab_test1e] (item={'chart_count': 0, 'creation_time': '2021-10-21T07:35:28.228Z',
'current_user_role_id': 1, 'current_user_role_ids': [1], 'cve_allowlist': {'creation_time': '0001-01-01T00:00:00.000Z',
'id': 56, 'items': [], 'project_id': 75, 'update_time': '0001-01-01T00:00:00.000Z'},
'metadata': {'public': 'true', 'retention_id': '54'}, 'name': 'harbor-esl-proxy', 'owner_id': 1, 'owner_name': 'admin', 'project_id': 75,
'registry_id': 101, 'repo_count': 0, 'update_time': '2021-10-21T07:35:28.228Z'}) => {"ansible_loop_var": "item", "changed": false,
"connection": "close", "content": "{\"errors\":[{\"code\":\"UNPROCESSABLE_ENTITY\",\"message\":\"validation failure
list:\\nparsing policy body from \\\"\\\" failed, because json: cannot unmarshal string into Go struct field RetentionPolicy.id of type int64\"}]}\n",
"content_length": "213", "content_type": "application/json; charset=utf-8", "date": "Fri, 22 Oct 2021 12:26:56 GMT", "elapsed": 1,
"item": {"chart_count": 0, "creation_time": "2021-10-21T07:35:28.228Z", "current_user_role_id": 1, "current_user_role_ids": [1],
"cve_allowlist": {"creation_time": "0001-01-01T00:00:00.000Z", "id": 56, "items": [], "project_id": 75, "update_time": "0001-01-01T00:00:00.000Z"},
"metadata": {"public": "true", "retention_id": "54"}, "name": "harbor-esl-proxy", "owner_id": 1, "owner_name": "admin", "project_id": 75,
"registry_id": 101, "repo_count": 0, "update_time": "2021-10-21T07:35:28.228Z"}, "json": {"errors": [{"code": "UNPROCESSABLE_ENTITY",
"message": "validation failure list:\nparsing policy body from \"\" failed, because json: cannot unmarshal string into Go struct field RetentionPolicy.id of type int64"}]},
"msg": "Status code was 422 and not [200]: HTTP Error 422: Unprocessable Entity", "redirected": false, "set_cookie": "sid=2e5f911867c675528b5e23c74f803240; Path=/; HttpOnly",
"status": 422, "url": "https://harbor.tst1e.k8s.shiplab.ss.acl.com/api/v2.0/retentions/54", "vary": "Accept-Encoding", "x_request_id": "334cb06f-c161-4085-8df2-a448b7776819"}
after this i changed body to json :
json converted:
- name: creating retention rules
uri:
url: "https://{{ ansible_host }}/api/v2.0/retentions/{{ item.metadata.retention_id | int }}"
method: PUT
body_format: json
headers:
Content-Type: application/json
Authorization: "{{ admin_passwd }}"
Accept: 'application/json, text/plain, */*'
body:
'{ "algorithm": "or",
"id": {{ item.metadata.retention_id | int }},
"rules": [{
"action": "retain",
"params": {
"latestPushedK": 3 },
"scope_selectors":{
"repository": [{
"decoration": "repoMatches",
"kind": "doublestar",
"pattern": "**" } ] },
"tag_selectors": [{
"decoration": "matches",
"extras": "{\"untagged\":true}",
"kind": "doublestar",
"pattern": "**" }],
"template": latestPushedK } ],
"scope": {
"level": "project",
"ref": {{ item.project_id | int }} },
"trigger": {
"kind": "Schedule",
"settings":{
"cron": "0 0 * * * *" }}}'
validate_certs: no
status_code: 200
when: "item.registry_id is defined"
loop: "{{ projects_result.json | flatten(levels=1) }}"
ignore_errors: yes
with this also i am getting the same error. please help me out on this.
verbosity information:
ansible-playbook -i ansible/inventories/shiplab ansible/playbooks/harbor-configurations.yaml --ask-vault-pass -vvv
ansible-playbook -i ansible/inventories/shiplab ansible/playbooks/harbor-configurations.yaml --ask-vault-pass -vvv
ansible-playbook 2.8.4
config file = /home/user1/repos/harbor/konvoy/ansible.cfg
configured module search path = ['/home/user1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.7/site-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 3.7.10 (default, May 5 2021, 11:43:58) [GCC 10.2.0]
Using /home/user1/repos/harbor/konvoy/ansible.cfg as config file
Vault password:
host_list declined parsing /home/user1/repos/harbor/konvoy/ansible/inventories/shiplab/test1e as it did not pass it's verify_file() method
auto declined parsing /home/user1/repos/harbor/konvoy/ansible/inventories/shiplab/test1e as it did not pass it's verify_file() method
Parsed /home/user1/repos/harbor/konvoy/ansible/inventories/shiplab/test1e inventory source with ini plugin
PLAYBOOK: harbor-configurations.yaml ************************************************************************************************************************
1 plays in ansible/playbooks/harbor-configurations.yaml
PLAY [harbor] ***********************************************************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************************************************
task path: /home/user1/repos/harbor/konvoy/ansible/playbooks/harbor-configurations.yaml:2
<harbor.tst1e.k8s.shiplab.ss.acl.com> ESTABLISH LOCAL CONNECTION FOR USER: MEA+user1
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'echo '"'"'~MEA+user1'"'"' && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129 `" && echo ansible-tmp-1634916091.044559-272844769197129="` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129 `" ) && sleep 0'
<harbor_shiplab_test1e> Attempting python interpreter discovery
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<harbor_shiplab_test1e> Python interpreter discovery fallback (unsupported platform for extended discovery: cygwin_nt-10.0)
Using module file /usr/lib/python3.7/site-packages/ansible/modules/system/setup.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> PUT /home/user1/.ansible/tmp/ansible-local-1736l28mza02/tmp5ucaxth_ TO /home/user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129/AnsiballZ_setup.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129/ /home/user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129/AnsiballZ_setup.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '/usr/bin/python /home/user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129/AnsiballZ_setup.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1634916091.044559-272844769197129/ > /dev/null 2>&1 && sleep 0'
ok: [harbor_shiplab_test1e]
META: ran handlers
TASK [../roles/harbor-retention-rules : get projects] *******************************************************************************************************
task path: /home/user1/repos/harbor/konvoy/ansible/roles/harbor-retention-rules/tasks/main.yml:3
<harbor.tst1e.k8s.shiplab.ss.acl.com> ESTABLISH LOCAL CONNECTION FOR USER: MEA+user1
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'echo '"'"'~MEA+user1'"'"' && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630 `" && echo ansible-tmp-1634916112.1313658-277975019551630="` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630 `" ) && sleep 0'
Using module file /usr/lib/python3.7/site-packages/ansible/modules/net_tools/basics/uri.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> PUT /home/user1/.ansible/tmp/ansible-local-1736l28mza02/tmpsm6sxu8_ TO /home/user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630/AnsiballZ_uri.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630/ /home/user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630/AnsiballZ_uri.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '/usr/bin/python /home/user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630/AnsiballZ_uri.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1634916112.1313658-277975019551630/ > /dev/null 2>&1 && sleep 0'
ok: [harbor_shiplab_test1e] => {
"changed": false,
"connection": "close",
"content_type": "application/json",
"cookies": {
"sid": "033af4ae913d69a8314a4edb84512aa6"
},
"cookies_string": "sid=033af4ae913d69a8314a4edb84512aa6",
"date": "Fri, 22 Oct 2021 15:22:01 GMT",
"elapsed": 1,
"invocation": {
"module_args": {
"attributes": null,
"backup": null,
"body": null,
"body_format": "raw",
"client_cert": null,
"client_key": null,
"content": null,
"creates": null,
"delimiter": null,
"dest": null,
"directory_mode": null,
"follow": false,
"follow_redirects": "safe",
"force": false,
"force_basic_auth": false,
"group": null,
"headers": {
"Accept": "application/json",
"Authorization": "Basic YWRtaW46SGFyYm9yMTIzNDU="
},
"http_agent": "ansible-httpget",
"method": "GET",
"mode": null,
"owner": null,
"regexp": null,
"remote_src": null,
"removes": null,
"return_content": false,
"selevel": null,
"serole": null,
"setype": null,
"seuser": null,
"src": null,
"status_code": [
200
],
"timeout": 30,
"unix_socket": null,
"unsafe_writes": null,
"url": "https://harbor.tst1e.k8s.shiplab.ss.acl.com/api/v2.0/projects",
"url_password": null,
"url_username": null,
"use_proxy": true,
"validate_certs": false
}
},
"json": [
{
"chart_count": 0,
"creation_time": "2021-10-21T07:35:28.228Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 56,
"items": [],
"project_id": 75,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "54"
},
"name": "harbor-esl-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 75,
"registry_id": 101,
"repo_count": 0,
"update_time": "2021-10-21T07:35:28.228Z"
}
],
"msg": "OK (unknown bytes)",
"redirected": false,
"set_cookie": "sid=033af4ae913d69a8314a4edb84512aa6; Path=/; HttpOnly",
"status": 200,
"transfer_encoding": "chunked",
"url": "https://harbor.tst1e.k8s.shiplab.ss.acl.com/api/v2.0/projects",
"vary": "Accept-Encoding",
"x_request_id": "c1d907c3-e324-49a5-a582-82f9672d1f8d",
"x_total_count": "8"
}
TASK [../roles/harbor-retention-rules : creating retention rules] *******************************************************************************************
task path: /home/user1/repos/harbor/konvoy/ansible/roles/harbor-retention-rules/tasks/main.yml:36
<harbor.tst1e.k8s.shiplab.ss.acl.com> ESTABLISH LOCAL CONNECTION FOR USER: MEA+user1
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'echo '"'"'~MEA+user1'"'"' && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702 `" && echo ansible-tmp-1634916125.1009495-142461137289702="` echo ~MEA+user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702 `" ) && sleep 0'
Using module file /usr/lib/python3.7/site-packages/ansible/modules/net_tools/basics/uri.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> PUT /home/user1/.ansible/tmp/ansible-local-1736l28mza02/tmpy38u0lww TO /home/user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702/AnsiballZ_uri.py
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702/ /home/user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702/AnsiballZ_uri.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c '/usr/bin/python /home/user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702/AnsiballZ_uri.py && sleep 0'
<harbor.tst1e.k8s.shiplab.ss.acl.com> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1634916125.1009495-142461137289702/ > /dev/null 2>&1 && sleep 0'
failed: [harbor_shiplab_test1e] (item={'chart_count': 0, 'creation_time': '2021-10-21T07:35:28.228Z', 'current_user_role_id': 1, 'current_user_role_ids': [1], 'cve_allowlist': {'creation_time': '0001-01-01T00:00:00.000Z', 'id': 56, 'items': [], 'project_id': 75, 'update_time': '0001-01-01T00:00:00.000Z'}, 'metadata': {'public': 'true', 'retention_id': '54'}, 'name': 'harbor-esl-proxy', 'owner_id': 1, 'owner_name': 'admin', 'project_id': 75, 'registry_id': 101, 'repo_count': 0, 'update_time': '2021-10-21T07:35:28.228Z'}) => {
"ansible_loop_var": "item",
"changed": false,
"connection": "close",
"content": "{\"errors\":[{\"code\":\"UNPROCESSABLE_ENTITY\",\"message\":\"validation failure list:\\nparsing policy body from \\\"\\\" failed, because json: cannot unmarshal string into Go struct field RetentionPolicy.id of type int64\"}]}\n",
"content_length": "213",
"content_type": "application/json; charset=utf-8",
"date": "Fri, 22 Oct 2021 15:22:17 GMT",
"elapsed": 3,
"invocation": {
"module_args": {
"attributes": null,
"backup": null,
"body": {
"algorithm": "or",
"id": "54",
"rules": [
{
"action": "retain",
"params": {
"latestPushedK": 3
},
"scope_selectors": {
"repository": [
{
"decoration": "repoMatches",
"kind": "doublestar",
"pattern": "**"
}
]
},
"tag_selectors": [
{
"decoration": "matches",
"extras": "{\"untagged\":true}",
"kind": "doublestar",
"pattern": "**"
}
],
"template": "latestPushedK"
}
],
"scope": {
"level": "project",
"ref": "75"
},
"trigger": {
"kind": "Schedule",
"settings": {
"cron": "0 0 * * * *"
}
}
},
"body_format": "json",
"client_cert": null,
"client_key": null,
"content": null,
"creates": null,
"delimiter": null,
"dest": null,
"directory_mode": null,
"follow": false,
"follow_redirects": "safe",
"force": false,
"force_basic_auth": false,
"group": null,
"headers": {
"Accept": "application/json, text/plain, */*",
"Authorization": "Basic YWRtaW46SGFyYm9yMTIzNDU=",
"Content-Type": "application/json"
},
"http_agent": "ansible-httpget",
"method": "PUT",
"mode": null,
"owner": null,
"regexp": null,
"remote_src": null,
"removes": null,
"return_content": false,
"selevel": null,
"serole": null,
"setype": null,
"seuser": null,
"src": null,
"status_code": [
"200"
],
"timeout": 30,
"unix_socket": null,
"unsafe_writes": null,
"url": "https://harbor.tst1e.k8s.shiplab.ss.acl.com/api/v2.0/retentions/54",
"url_password": null,
"url_username": null,
"use_proxy": true,
"validate_certs": false
}
},
"item": {
"chart_count": 0,
"creation_time": "2021-10-21T07:35:28.228Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 56,
"items": [],
"project_id": 75,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "true",
"retention_id": "54"
},
"name": "harbor-esl-proxy",
"owner_id": 1,
"owner_name": "admin",
"project_id": 75,
"registry_id": 101,
"repo_count": 0,
"update_time": "2021-10-21T07:35:28.228Z"
},
"json": {
"errors": [
{
"code": "UNPROCESSABLE_ENTITY",
"message": "validation failure list:\nparsing policy body from \"\" failed, because json: cannot unmarshal string into Go struct field RetentionPolicy.id of type int64"
}
]
},
"msg": "Status code was 422 and not [200]: HTTP Error 422: Unprocessable Entity",
"redirected": false,
"set_cookie": "sid=5ffe84980b5cb188e7821ae838e4fc33; Path=/; HttpOnly",
"status": 422,
"url": "https://harbor.tst1e.k8s.shiplab.ss.acl.com/api/v2.0/retentions/54",
"vary": "Accept-Encoding",
"x_request_id": "d7a855cf-3de6-47ab-8b47-d275cda5274b"
}
"skip_reason": "Conditional result was False"
}
...ignoring
META: ran handlers
META: ran handlers
PLAY RECAP **************************************************************************************************************************************************
harbor_shiplab_test1e : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1

Wow, what a fascinating rabbit hole :-(
So, there is an existing issue reporting that behavior, but it was closed "works-as-designed," along with some nonsense about configuring jinja2 in some incompatible way
As best I can tell from running a bunch of experiments, manually crafting that JSON payload is the only safe way:
body_format: json
body: >-
{{
(
{
"algorithm": "or",
"id": item.metadata.retention_id | int,
"rules": [
{
"action": "retain",
"params": {
"latestPushedK": 3
},
"scope_selectors": {
"repository": [
{
"decoration": "repoMatches",
"kind": "doublestar",
"pattern": "**"
}
]
},
"tag_selectors": [
{
"decoration": "matches",
"kind": "doublestar",
"pattern": "**"
}
],
"template": "latestPushedK"
}
],
"scope": {
"level": "project",
"ref": item.project_id | int
},
"trigger": {
"kind": "Schedule",
"settings": {
"cron": "0 0 * * * *"
}
}
}
) | to_json }}
when: "item.registry_id is defined"
I wish I had a better explanation as to what is going on, but given the response to that issue, I'm not super sure they even consider your experience to be a bug
But for clarity, what is happening here is that those leading brace characters inside the () are python dict literals, which is why the "id": item.metadata.retention_id | int, doesn't need any special consideration -- both sides of that are python (err, jinja2, but same-same)
The () are likely not strictly necessary, but do make it super obvious what is being fed into the | to_json filter
Then, the outer {{ are the jinja2 delimiters that you're used to, in order to have all that inner jinja2 code run, and the body: >- is to get us out of yaml quoting hell

Related

json jq request format

I need help to extract data from this JSON file using jq.
The app flv then verify the streamname mystrame is active and extract meta.video.height
I did try lot of queries without success and my jq knowledge is poor.
{
"port": 1935,
"server_index": 0,
"applications": [{
"name": "hls",
"live": {
"streams": [{
"name": "donbosco",
"time": 2380739,
"bw_in": 2112440,
"bytes_in": 541618713,
"bw_out": 0,
"bytes_out": 0,
"bw_audio": 35544,
"bw_video": 2076888,
"clients": [{
"id": 453,
"address": "127.0.0.1",
"time": 2380959,
"flashver": "FMLE/3.0 (compatible; Lavf57.83.100)",
"dropped": 0,
"avsync": 28,
"timestamp": 2382635,
"publishing": true,
"active": true
}],
"records": [],
"meta": {
"video": {
"width": 1168,
"height": 720,
"frame_rate": 25,
"codec": "H264",
"profile": "High",
"level": 3.1
},
"audio": {
"codec": "AAC",
"profile": "LC",
"channels": 2,
"sample_rate": 16000
}
},
"nclients": 1,
"publishing": true,
"active": true
}],
"nclients": 1
},
"recorders": {
"count": 0,
"lists": []
}
},
{
"name": "flv",
"live": {
"streams": [{
"name": "mystream",
"time": 2382811,
"bw_in": 2059096,
"bytes_in": 541841549,
"bw_out": 2059096,
"bytes_out": 543351459,
"bw_audio": 35472,
"bw_video": 2023624,
"clients": [{
"id": 452,
"address": "127.0.0.1",
"time": 2382727,
"flashver": "LNX 9,0,124,2",
"dropped": 0,
"avsync": -12,
"timestamp": 2384520,
"publishing": false,
"active": true
},
{
"id": 451,
"address": "127.0.0.1",
"time": 2383031,
"flashver": "FMLE/3.0 (compatible; Lavf58.74.100)",
"dropped": 0,
"avsync": -12,
"timestamp": 2384520,
"publishing": true,
"active": true
}
],
"records": [],
"meta": {
"video": {
"width": 1168,
"height": 720,
"frame_rate": 25,
"codec": "H264",
"profile": "High",
"level": 3.1
},
"audio": {
"codec": "AAC",
"profile": "LC",
"channels": 2,
"sample_rate": 16000
}
},
"nclients": 2,
"publishing": true,
"active": true
}],
"nclients": 2
},
"recorders": {
"count": 0,
"lists": []
}
}
]
}
Are you asking for help with the command-line JSON processor jq or the JavaScript library jQuery? They are not the same. For jq, try
jq '
.applications
| map(select(.name == "flv"))[].live.streams
| map(select(.name == "mystream" and .active))[].meta.video.height
'
720
Demo

How to fetch multiple values from a nested JSON like output

I am looking forward a way to list the value from a nested variable (which has dict and list values) ie register which is response_find and I have that in mostly json format and most of the values inside the json are nested within [] list like construct.
It works somehow if I get individual values from it like below:
var=response_find['json']['results'][0]['content_facet_attributes']
or
var=response_find['json']['results'][0]['certname']
But did not work when I did below:
var=response_find['json']['results'][0]['content_view_id']
Below are few values which I'm looking forward to fetch from this nested output:
"architecture_name": "x86_64",
"name": "satcap.rest.example.com",
"url": "https://satcap.rest.example.com:9090"
"content_source_name": "satcap.rest.example.com",
"name": "ccv-azure-infra-rhel7"
"name": "Prod"
"id": 33485,
"ip": "192.168.88.88",
"name": "invwharn108.test.exampl.com",
"registered_through": "satcap.rest.example.com"
Nested Json output:
{
"response_find": {
"apipie_checksum": "7533ab625c45a3819647f4fb4c9394c2832c0180",
"cookies": {
"_session_id": "25872833d9e2723073797fbd8cfa2d63"
},
"cookies_string": "_session_id=25872833d9e2723073797fbd8cfa2d63",
"foreman_version": "1.24.1.21",
"json": {
"page": 1,
"per_page": 20,
"results": [
{
"architecture_id": 1,
"architecture_name": "x86_64",
"build": false,
"capabilities": [
"build"
],
"certname": "invwharn108.test.exampl.com",
"comment": null,
"content_facet_attributes": {
"applicable_module_stream_count": 0,
"applicable_package_count": 0,
"content_source": {
"id": 17,
"name": "satcap.rest.example.com",
"url": "https://satcap.rest.example.com:9090"
},
"content_source_id": 17,
"content_source_name": "satcap.rest.example.com",
"content_view": {
"id": 67,
"name": "ccv-azure-infra-rhel7"
},
"content_view_id": 67,
"content_view_name": "ccv-azure-infra-rhel7",
"errata_counts": {
"bugfix": 0,
"enhancement": 0,
"security": 0,
"total": 0
},
"id": 32255,
"kickstart_repository": {
"id": 5772,
"name": "Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.8"
},
"kickstart_repository_id": 5772,
"kickstart_repository_name": "5772",
"lifecycle_environment": {
"id": 5,
"name": "Prod"
},
"lifecycle_environment_id": 5,
"lifecycle_environment_name": "Prod",
"upgradable_module_stream_count": 0,
"upgradable_package_count": 0,
"uuid": "dab2d66b-1a73-490a-aa9d-3f036658980a"
},
"created_at": "2020-12-23 13:19:35 UTC",
"disk": null,
"domain_id": 25,
"domain_name": "test.example-aws.example.com",
"enabled": true,
"environment_id": null,
"environment_name": null,
"errata_status": 1,
"errata_status_label": "Could not calculate errata status, ensure host is registered and the katello-host-tools package is installed",
"global_status": 1,
"global_status_label": "Warning",
"hostgroup_id": 982,
"hostgroup_name": "infra",
"hostgroup_title": "NXDI-hg-aws/west-europe/infra",
"id": 33485,
"image_file": "",
"image_id": null,
"image_name": null,
"installed_at": null,
"ip": "192.168.88.88",
"ip6": null,
"last_compile": "2020-12-23 13:19:38 UTC",
"last_report": null,
"location_id": 37,
"location_name": "west-europe",
"mac": "02:78:43:60:ee:fb",
"managed": false,
"medium_id": null,
"medium_name": null,
"model_id": 25,
"model_name": "HVM domU",
"name": "invwharn108.test.exampl.com",
"openscap_proxy": null,
"openscap_proxy_id": null,
"openscap_proxy_name": null,
"operatingsystem_id": 21,
"operatingsystem_name": "RedHat 7.8",
"organization_id": 3,
"pxe_loader": "PXELinux BIOS",
"subnet_id": 65,
"subnet_name": "aws-west-europe-infra",
"subscription_facet_attributes": {
"autoheal": true,
"hypervisor": false,
"id": 33593,
"last_checkin": "2020-12-23 13:19:42 UTC",
"purpose_addons": [],
"purpose_role": "",
"purpose_usage": "",
"registered_at": "2020-12-23 13:19:35 UTC",
"registered_through": "satcap.rest.example.com",
},
"subscription_global_status": 1,
"subscription_status": 1,
"subscription_status_label": "Partially entitled",
}
],
"search": "name=invwharn108.test.exampl.com",
"sort": {
"by": null,
"order": null
},
"subtotal": 1,
"total": 13121
},
"msg": "OK (unknown bytes)",
"x_request_id": "e02dd9b3-b2f8-4c39-8bf5-1ce1897d9548",
}
}
My Play:
---
- hosts: localhost
tasks:
- include_vars: vaults/aws_secrets.yml
no_log: true
- include_vars: requirements.yml
no_log: true
- name: Find the ID of Cloud VM in Satellite
uri:
url: "https://{{ satserver }}/api/v2/hosts?search=name={{ aws_instance_name }}"
method: GET
user: "{{ aws_satuser }}"
password: "{{ aws_satpw }}"
force_basic_auth: yes
validate_certs: no
register: response_find
delegate_to: localhost
changed_when: false
- debug:
var=response_find['json']['results'][0]['content_facet_attributes']
#var=response_find.json.results.0.organization_name
#var=response_find['json']['results'][0]['certname']
Result:
"content_facet_attributes": {
"applicable_module_stream_count": 0,
"applicable_package_count": 0,
"content_source": {
"id": 17,
"name": "satcap.rest.example.com",
"url": "https://satcap.rest.example.com:9090"
},
"content_source_id": 17,
"content_source_name": "satcap.rest.example.com",
"content_view": {
"id": 67,
"name": "ccv-azure-infra-rhel7"
},
"content_view_id": 67,
"content_view_name": "ccv-azure-infra-rhel7",
"errata_counts": {
"bugfix": 0,
"enhancement": 0,
"security": 0,
"total": 0
},
"id": 32255,
"kickstart_repository": {
"id": 5772,
"name": "Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.8"
},
"kickstart_repository_id": 5772,
"kickstart_repository_name": "5772",
"lifecycle_environment": {
"id": 5,
"name": "Prod"
},
"lifecycle_environment_id": 5,
"lifecycle_environment_name": "Prod",
"upgradable_module_stream_count": 0,
"upgradable_package_count": 0,
"uuid": "dab2d66b-1a73-490a-aa9d-3f036658980a"
},
If you want to do a dictionary out of those data you listed, you could use the filter json_query, which is using JMESPath to parse and process JSON.
In order to extract a dictionary looking like:
{
"architecture_name": "x86_64",
"content_source_name": "satcap.rest.example.com",
"content_source_url": "https://satcap.rest.example.com:9090",
"content_view_name": "ccv-azure-infra-rhel7",
"egistered_through": "satcap.rest.example.com",
"id": 33485,
"ip": "192.168.88.88",
"lifecycle_environment_name": "Prod",
"name": "invwharn108.test.exampl.com"
}
You could use JMESPath to filter multiselect hashes.
Here is a JMESPath query that would give this result:
json.results[*].{
"architecture_name": architecture_name,
"content_source_name": content_facet_attributes.content_source.name,
"content_source_url": content_facet_attributes.content_source.url,
"content_view_name": content_facet_attributes.content_view.name,
"lifecycle_environment_name": content_facet_attributes.lifecycle_environment.name,
"id": id,
"ip": ip,
"name": name,
"egistered_through": subscription_facet_attributes.registered_through
}
Here is an example playbook using this:
- hosts: localhost
gather_facts: no
tasks:
- debug:
msg: >-
{{
(
response_find | json_query('
json.results[*].{
"architecture_name": architecture_name,
"content_source_name": content_facet_attributes.content_source.name,
"content_source_url": content_facet_attributes.content_source.url,
"content_view_name": content_facet_attributes.content_view.name,
"lifecycle_environment_name": content_facet_attributes.lifecycle_environment.name,
"id": id,
"ip": ip,
"name": name,
"egistered_through": subscription_facet_attributes.registered_through
}
')
).0
}}
vars:
{
"response_find": {
"apipie_checksum": "7533ab625c45a3819647f4fb4c9394c2832c0180",
"cookies": {
"_session_id": "25872833d9e2723073797fbd8cfa2d63"
},
"cookies_string": "_session_id=25872833d9e2723073797fbd8cfa2d63",
"foreman_version": "1.24.1.21",
"json": {
"page": 1,
"per_page": 20,
"results": [{
"architecture_id": 1,
"architecture_name": "x86_64",
"build": false,
"capabilities": [
"build"
],
"certname": "invwharn108.test.exampl.com",
"comment": null,
"content_facet_attributes": {
"applicable_module_stream_count": 0,
"applicable_package_count": 0,
"content_source": {
"id": 17,
"name": "satcap.rest.example.com",
"url": "https://satcap.rest.example.com:9090"
},
"content_source_id": 17,
"content_source_name": "satcap.rest.example.com",
"content_view": {
"id": 67,
"name": "ccv-azure-infra-rhel7"
},
"content_view_id": 67,
"content_view_name": "ccv-azure-infra-rhel7",
"errata_counts": {
"bugfix": 0,
"enhancement": 0,
"security": 0,
"total": 0
},
"id": 32255,
"kickstart_repository": {
"id": 5772,
"name": "Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.8"
},
"kickstart_repository_id": 5772,
"kickstart_repository_name": "5772",
"lifecycle_environment": {
"id": 5,
"name": "Prod"
},
"lifecycle_environment_id": 5,
"lifecycle_environment_name": "Prod",
"upgradable_module_stream_count": 0,
"upgradable_package_count": 0,
"uuid": "dab2d66b-1a73-490a-aa9d-3f036658980a"
},
"created_at": "2020-12-23 13:19:35 UTC",
"disk": null,
"domain_id": 25,
"domain_name": "test.example-aws.example.com",
"enabled": true,
"environment_id": null,
"environment_name": null,
"errata_status": 1,
"errata_status_label": "Could not calculate errata status, ensure host is registered and the katello-host-tools package is installed",
"global_status": 1,
"global_status_label": "Warning",
"hostgroup_id": 982,
"hostgroup_name": "infra",
"hostgroup_title": "NXDI-hg-aws/west-europe/infra",
"id": 33485,
"image_file": "",
"image_id": null,
"image_name": null,
"installed_at": null,
"ip": "192.168.88.88",
"ip6": null,
"last_compile": "2020-12-23 13:19:38 UTC",
"last_report": null,
"location_id": 37,
"location_name": "west-europe",
"mac": "02:78:43:60:ee:fb",
"managed": false,
"medium_id": null,
"medium_name": null,
"model_id": 25,
"model_name": "HVM domU",
"name": "invwharn108.test.exampl.com",
"openscap_proxy": null,
"openscap_proxy_id": null,
"openscap_proxy_name": null,
"operatingsystem_id": 21,
"operatingsystem_name": "RedHat 7.8",
"organization_id": 3,
"pxe_loader": "PXELinux BIOS",
"subnet_id": 65,
"subnet_name": "aws-west-europe-infra",
"subscription_facet_attributes": {
"autoheal": true,
"hypervisor": false,
"id": 33593,
"last_checkin": "2020-12-23 13:19:42 UTC",
"purpose_addons": [],
"purpose_role": "",
"purpose_usage": "",
"registered_at": "2020-12-23 13:19:35 UTC",
"registered_through": "satcap.rest.example.com"
},
"subscription_global_status": 1,
"subscription_status": 1,
"subscription_status_label": "Partially entitled"
}],
"search": "name=invwharn108.test.exampl.com",
"sort": {
"by": null,
"order": null
},
"subtotal": 1,
"total": 13121
},
"msg": "OK (unknown bytes)",
"x_request_id": "e02dd9b3-b2f8-4c39-8bf5-1ce1897d9548"
}
}
Which gives the recap:
PLAY [localhost] *************************************************************************************************
TASK [debug] *****************************************************************************************************
ok: [localhost] => {
"msg": {
"architecture_name": "x86_64",
"content_source_name": "satcap.rest.example.com",
"content_source_url": "https://satcap.rest.example.com:9090",
"content_view_name": "ccv-azure-infra-rhel7",
"egistered_through": "satcap.rest.example.com",
"id": 33485,
"ip": "192.168.88.88",
"lifecycle_environment_name": "Prod",
"name": "invwharn108.test.exampl.com"
}
}
PLAY RECAP *******************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

How to filter JSON data in Ansible?

I'm querying an API with Ansible command's command. The API returns a JSON object with network information.
I'd like to get a server's private ip based on its public ip. I know this is possible with JSON query filter but I can't figure out how.
The code:
- name: Get RPN topology
command: 'curl -X GET -H "Authorization: Bearer {{ onlineApiToken }}" "https://api.online.net/api/v1/rpn/group"'
register: RPN
delegate_to: 127.0.0.1
This is what RPN.stdout output looks like:
TASK [debug] ****************************************************************************************
ok: [ps1] => {
"changed": false,
"msg": [
{
"id": 7406,
"members": [
{
"id": 0000,
"ip": "x.x.x.x",
"owner": "buzut",
"private_ip": "10.91.154.39",
"speed": 100,
"status": "active"
},
{
"id": 1111,
"ip": "y.y.y.y",
"owner": "buzut",
"private_ip": "10.91.120.148",
"speed": 100,
"status": "active"
},
{
"id": 2222,
"ip": "z.z.z.z",
"owner": "buzut",
"private_ip": "10.91.165.215",
"speed": 1000,
"status": "active"
}
],
"name": "MySQL",
"owner": "buzut",
"shared": false,
"status": "updating"
}
]
}
The question: how do I get a server's private ip, based on its public one?
You probably want to take some time to read through the JMESPath Examples to familiarize yourself with the syntax supported by the json_query filter. The following gives you the result you want:
- hosts: localhost
gather_facts: false
vars:
RPN:
stdout: >-
[
{
"id": 7406,
"members": [
{
"id": 0,
"ip": "x.x.x.x",
"owner": "buzut",
"private_ip": "10.91.154.39",
"speed": 100,
"status": "active"
},
{
"id": 1111,
"ip": "y.y.y.y",
"owner": "buzut",
"private_ip": "10.91.120.148",
"speed": 100,
"status": "active"
},
{
"id": 2222,
"ip": "z.z.z.z",
"owner": "buzut",
"private_ip": "10.91.165.215",
"speed": 1000,
"status": "active"
}
],
"name": "MySQL",
"owner": "buzut",
"shared": false,
"status": "updating"
}
]
tasks:
- name: lookup server based on public ip
debug:
var: item
with_items: "{{RPN.stdout|from_json|json_query(public_to_private_ip)}}"
vars:
public_to_private_ip: >-
[].members[?ip=='{{ public_ip }}'].private_ip
If I call this like:
ansible-playbook playbook.yml -e public_ip=y.y.y.y
I get:
TASK [lookup server based on public ip] ****************************************
ok: [localhost] => (item=10.91.120.148) => {
"item": "10.91.120.148"
}
And if I call it as:
ansible-playbook playbook.yml -e public_ip=z.z.z.z
I get:
TASK [lookup server based on public ip] ****************************************
ok: [localhost] => (item=10.91.165.215) => {
"item": "10.91.165.215"
}
---
- name: play1
hosts: localhost
gather_facts: false
vars:
public_ip: "x.x.x.x"
tasks:
- name: Print data
debug: var=RPN
- name: Lookup value in json var
debug: var=item
with_items: "{{RPN|json_query(public_to_private_ip)}}"
vars:
public_to_private_ip: "members[?ip=='{{ public_ip }}'].private_ip"
Gives you:
PLAY [play1] *******************************************************************
TASK [Print data] **************************************************************
ok: [localhost] => {
"RPN": {
"id": 7406,
"members": [
{
"id": "0000",
"ip": "x.x.x.x",
"owner": "buzut",
"private_ip": "10.91.154.39",
"speed": 100,
"status": "active"
},
{
"id": 1111,
"ip": "y.y.y.y",
"owner": "buzut",
"private_ip": "10.91.120.148",
"speed": 100,
"status": "active"
},
{
"id": 2222,
"ip": "z.z.z.z",
"owner": "buzut",
"private_ip": "10.91.165.215",
"speed": 1000,
"status": "active"
}
],
"name": "MySQL",
"owner": "buzut",
"shared": false,
"status": "updating"
}
}
TASK [Lookup value in json var] ************************************************
ok: [localhost] => (item=10.91.154.39) => {
"item": "10.91.154.39"
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
Where members[?ip=='{{ public_ip }}'].private_ip does the magic.

Ansible: EC2 provisioning and Iterations

I am trying to start a bunch of EC2 instances, then install something on them based on the IP given by AWS. With only one EC2, I can add the host and proceed without any issue,but when I chain them using with_dict, I can't achieve it anymore...
The following runs as I want, but I can't understand how to deal with the registered variable ec2_infos I got from the provisioning...
- name: Create Test EC2 instances
ec2:
group: default
image: ami-40d28157
instance_type: '{{item.value.type}}'
instance_tags:
Name: "{{ tag+'-'+item.value.name }}"
key_name: privatekey
region: us-west-1
vpc_subnet_id: subnet-REDACTD
wait: yes
with_dict: '{{ec2_stack}}'
register: ec2_infos
With a dictionary like
ec2_stack:
serv1:
type: t2.micro
name: server1
serv2:
type: t2.small
name: server2
ec2_infos is structures like:
"ec2_infos": {
"changed": true,
"msg": "All items completed",
"results": [
{
"_ansible_item_result": true,
"_ansible_no_log": false,
"_ansible_parsed": true,
"changed": true,
"instance_ids": [
"i-0fewq09812ddq6"
],
"instances": [
{
"ami_launch_index": "0",
"architecture": "x86_64",
"block_device_mapping": {
"/dev/sda1": {
"delete_on_termination": true,
"status": "attached",
"volume_id": "vol-0987654"
}
},
"dns_name": "",
"ebs_optimized": false,
"groups": {
"sg-qdwdww": "default"
},
"hypervisor": "xen",
"id": "i-083665656521dwq6",
"image_id": "ami-40d28157",
"launch_time": "2016-11-24T20:38:53.000Z",
"placement": "us-west-1d",
"private_ip": "x.x.x.x",
"public_dns_name": "",
"public_ip": null,
"ramdisk": null,
"region": "us-east-1",
"root_device_name": "/dev/sda1",
"root_device_type": "ebs",
"state": "running",
"state_code": 16,
"tags": {
"Name": "server1",
"Team": "blah"
},
"tenancy": "default","tenancy": "default",
"virtualization_type": "hvm"
}
],
"invocation": {
"module_args": {
"assign_public_ip": false,
"exact_count": null,
"group": [
"default"
],
"group_id": null,
"id": null,
"image": "ami-40d28157",
"instance_ids": null,
"instance_initiated_shutdown_behavior": null,
"instance_profile_name": null,
"instance_tags": {
"Name": "server1",
"Team": "blah"
},
"instance_type": "t2.micro",
"kernel": null,
"volumes": null,
"vpc_subnet_id": "subnet-abcdfed",
"wait": true,
"wait_timeout": "300",
"zone": null
},
"module_name": "ec2"
},
"item": {
"key": "serv1",
"value": {
"name": "server1",
"type": "t2.micro"
}
},
"tagged_instances": []
},
{
"_ansible_item_result": true,
"_ansible_no_log": false,
"_ansible_parsed": true,
"changed": true,
"instance_ids": [
"i-0971278624334fd"
],
"instances": [
{
"ami_launch_index": "0",
"architecture": "x86_64",
"block_device_mapping": {
"/dev/sda1": {
"delete_on_termination": true,
"status": "attached",
"volume_id": "vol-9999999"
}
},
"dns_name": "",
"ebs_optimized": false,
"groups": {
"sg-redactd": "default"
},
"launch_time": "2016-11-24T20:39:21.000Z",
"private_ip": "y.y.y.y",
"public_dns_name": "",
"public_ip": null,
"ramdisk": null,
"state": "running",
"state_code": 16,
"tags": {
"Name": "serv2",
"Team": "blah"
},
"tenancy": "default",
"virtualization_type": "hvm"
}
],
"invocation": {
"module_args": {
"assign_public_ip": false,
"wait_timeout": "300",
"zone": null
},
"module_name": "ec2"
},
"item": {
"key": "server2",
"value": {
"name": "serv2",
"type": "t2.small"
}
},
"tagged_instances": []
}
]
}
I tried with_items and with_subelements in different ways, but I can't manage to get every IPs of the new EC2. I don't even need to sort them just extract them from the instances part and feed them to add_host so I can proceed.
Anybody knows a clean way to do so, or would be kind enough to explain to me how to deal with a registered variable after a loop properly ?
Answer from the comments:
ec2_infos.results | map(attribute='instances') | sum(start=[]) | map(attribute='private_ip') | list

How to format a JSON string as a table using jq?

Just started out with Bash scripting and stumbled upon jq to work with JSON.
I need to transform a JSON string like below to a table for output in the terminal.
[{
"name": "George",
"id": 12,
"email": "george#domain.example"
}, {
"name": "Jack",
"id": 18,
"email": "jack#domain.example"
}, {
"name": "Joe",
"id": 19,
"email": "joe#domain.example"
}]
What I want to display in the terminal:
ID Name
=================
12 George
18 Jack
19 Joe
Notice how I don't want to display the email property for each row, so the jq command should involve some filtering. The following gives me a plain list of names and id's:
list=$(echo "$data" | jq -r '.[] | .name, .id')
printf "$list"
The problem with that is, I cannot display it like a table. I know jq has some formatting options, but not nearly as good as the options I have when using printf. I think I want to get these values in an array which I can then loop through myself to do the formatting...? The things I tried give me varying results, but never what I really want.
Can someone point me in the right direction?
Using the #tsv filter has much to recommend it, mainly because it handles numerous "edge cases" in a standard way:
.[] | [.id, .name] | #tsv
Adding the headers can be done like so:
jq -r '["ID","NAME"], ["--","------"], (.[] | [.id, .name]) | #tsv'
The result:
ID NAME
-- ------
12 George
18 Jack
19 Joe
As pointed out by #Tobia, you might want to format the table for viewing by using column to post-process the result produced by jq. If you are using a bash-like shell then column -ts $'\t' should be quite portable.
length*"-"
To automate the production of the line of dashes:
jq -r '(["ID","NAME"] | (., map(length*"-"))), (.[] | [.id, .name]) | #tsv'
Why not something like:
echo '[{
"name": "George",
"id": 12,
"email": "george#domain.example"
}, {
"name": "Jack",
"id": 18,
"email": "jack#domain.example"
}, {
"name": "Joe",
"id": 19,
"email": "joe#domain.example"
}]' | jq -r '.[] | "\(.id)\t\(.name)"'
Output
12 George
18 Jack
19 Joe
Edit 1 : For fine grained formatting use tools like awk
echo '[{
"name": "George",
"id": 12,
"email": "george#domain.example"
}, {
"name": "Jack",
"id": 18,
"email": "jack#domain.example"
}, {
"name": "Joe",
"id": 19,
"email": "joe#domain.example"
}]' | jq -r '.[] | [.id, .name] | #csv' | awk -v FS="," 'BEGIN{print "ID\tName";print "============"}{printf "%s\t%s%s",$1,$2,ORS}'
ID Name
============
12 "George"
18 "Jack"
19 "Joe"
Edit 2 : In reply to
There's no way I can get a variable containing an array straight
from jq?
Why not?
A bit involved example( in fact modified from yours ) where email is changed to an array demonstrates this
echo '[{
"name": "George",
"id": 20,
"email": [ "george#domain1.example" , "george#domain2.example" ]
}, {
"name": "Jack",
"id": 18,
"email": [ "jack#domain3.example" , "jack#domain5.example" ]
}, {
"name": "Joe",
"id": 19,
"email": [ "joe#domain.example" ]
}]' | jq -r '.[] | .email'
Output
[
"george#domain1.example",
"george#domain2.example"
]
[
"jack#domain3.example",
"jack#domain5.example"
]
[
"joe#domain.example"
]
Defining headers by hand is suboptimal! Omitting headers is also suboptimal.
TL;DR
data
[{ "name": "George", "id": 12, "email": "george#domain.example" },
{ "name": "Jack", "id": 18, "email": "jack#domain.example" },
{ "name": "Joe", "id": 19, "email": "joe#domain.example" }]
script
[.[]| with_entries( .key |= ascii_downcase ) ]
| (.[0] |keys_unsorted | #tsv)
, (.[] |map(.) |#tsv)
how to run
$ < data jq -rf script | column -t
name id email
George 12 george#domain.example
Jack 18 jack#domain.example
Joe 19 joe#domain.example
I found this question while summarizng some data from amazon web services. The problem I was working on, in case you want another example:
$ aws ec2 describe-spot-instance-requests | tee /tmp/ins |
jq --raw-output '
# extract instances as a flat list.
[.SpotInstanceRequests | .[]
# remove unwanted data
| {
State,
statusCode: .Status.Code,
type: .LaunchSpecification.InstanceType,
blockPrice: .ActualBlockHourlyPrice,
created: .CreateTime,
SpotInstanceRequestId}
]
# lowercase keys
# (for predictable sorting, optional)
| [.[]| with_entries( .key |= ascii_downcase ) ]
| (.[0] |keys_unsorted | #tsv) # print headers
, (.[]|.|map(.) |#tsv) # print table
' | column -t
Output:
state statuscode type blockprice created spotinstancerequestid
closed instance-terminated-by-user t3.nano 0.002000 2019-02-24T15:21:36.000Z sir-r5bh7skq
cancelled bad-parameters t3.nano 0.002000 2019-02-24T14:51:47.000Z sir-1k9s5h3m
closed instance-terminated-by-user t3.nano 0.002000 2019-02-24T14:55:26.000Z sir-43x16b6n
cancelled bad-parameters t3.nano 0.002000 2019-02-24T14:29:23.000Z sir-2jsh5brn
active fulfilled t3.nano 0.002000 2019-02-24T15:37:26.000Z sir-z1e9591m
cancelled bad-parameters t3.nano 0.002000 2019-02-24T14:33:42.000Z sir-n7c15y5p
Input:
$ cat /tmp/ins
{
"SpotInstanceRequests": [
{
"Status": {
"Message": "2019-02-24T15:29:38+0000 : 2019-02-24T15:29:38+0000 : Spot Instance terminated due to user-initiated termination.",
"Code": "instance-terminated-by-user",
"UpdateTime": "2019-02-24T15:31:03.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T15:21:36.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"InstanceId": "i-0414083bef5e91d94",
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-r5bh7skq",
"State": "closed",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "public",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T15:21:36.000Z",
"SpotPrice": "0.008000"
},
{
"Status": {
"Message": "Your Spot request failed due to bad parameters.",
"Code": "bad-parameters",
"UpdateTime": "2019-02-24T14:51:48.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T14:51:47.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"Fault": {
"Message": "Invalid device name /dev/sda",
"Code": "InvalidBlockDeviceMapping"
},
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-1k9s5h3m",
"State": "cancelled",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "public",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T14:51:47.000Z",
"SpotPrice": "0.011600"
},
{
"Status": {
"Message": "2019-02-24T15:02:17+0000 : 2019-02-24T15:02:17+0000 : Spot Instance terminated due to user-initiated termination.",
"Code": "instance-terminated-by-user",
"UpdateTime": "2019-02-24T15:03:34.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T14:55:26.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"InstanceId": "i-010442ac3cc85ec08",
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-43x16b6n",
"State": "closed",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "public",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T14:55:26.000Z",
"SpotPrice": "0.011600"
},
{
"Status": {
"Message": "Your Spot request failed due to bad parameters.",
"Code": "bad-parameters",
"UpdateTime": "2019-02-24T14:29:24.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T14:29:23.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"Fault": {
"Message": "Addressing type must be 'public'",
"Code": "InvalidParameterCombination"
},
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-2jsh5brn",
"State": "cancelled",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T14:29:23.000Z",
"SpotPrice": "0.011600"
},
{
"Status": {
"Message": "Your spot request is fulfilled.",
"Code": "fulfilled",
"UpdateTime": "2019-02-24T15:37:28.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T15:37:26.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"InstanceId": "i-0a29e9de6d59d433f",
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-z1e9591m",
"State": "active",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "public",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T15:37:26.000Z",
"SpotPrice": "0.008000"
},
{
"Status": {
"Message": "Your Spot request failed due to bad parameters.",
"Code": "bad-parameters",
"UpdateTime": "2019-02-24T14:33:43.000Z"
},
"ActualBlockHourlyPrice": "0.002000",
"ValidUntil": "2019-03-03T14:33:42.000Z",
"InstanceInterruptionBehavior": "terminate",
"Tags": [],
"Fault": {
"Message": "Invalid device name /dev/sda",
"Code": "InvalidBlockDeviceMapping"
},
"BlockDurationMinutes": 60,
"SpotInstanceRequestId": "sir-n7c15y5p",
"State": "cancelled",
"ProductDescription": "Linux/UNIX",
"LaunchedAvailabilityZone": "eu-north-1a",
"LaunchSpecification": {
"Placement": {
"Tenancy": "default",
"AvailabilityZone": "eu-north-1a"
},
"ImageId": "ami-6d27a913",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda",
"VirtualName": "root",
"NoDevice": "",
"Ebs": {
"Encrypted": false,
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"EbsOptimized": false,
"SecurityGroups": [
{
"GroupName": "default"
}
],
"Monitoring": {
"Enabled": false
},
"InstanceType": "t3.nano",
"AddressingType": "public",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "eth-zero",
"NetworkInterfaceId": "",
"DeleteOnTermination": true,
"SubnetId": "subnet-420ffc2b",
"AssociatePublicIpAddress": true
}
]
},
"Type": "one-time",
"CreateTime": "2019-02-24T14:33:42.000Z",
"SpotPrice": "0.011600"
}
]
}
The problem with the answers above is they only work if the fields are all about the same width.
To avoid this issue, the Linux column command could be used:
// input.json
[
{
"name": "George",
"id": "a very very long field",
"email": "george#domain.example"
},
{
"name": "Jack",
"id": 18,
"email": "jack#domain.example"
},
{
"name": "Joe",
"id": 19,
"email": "joe#domain.example"
}
]
Then:
▶ jq -r '.[] | [.id, .name] | #tsv' input.json | column -ts $'\t'
a very very long field George
18 Jack
19 Joe
I made a mix with all responses to get all this behaviours
create header table
handle long fields
create a function to reuse
function bash
function jsonArrayToTable(){
jq -r '(.[0] | ([keys[] | .] |(., map(length*"-")))), (.[] | ([keys[] as $k | .[$k]])) | #tsv' | column -t -s $'\t'
}
Sample use
echo '[{"key1":"V1.1", "key2":"V2.1"}, {"keyA":"V1.2", "key2":"V2.2"}]' | jsonArrayToTable
output
key1 key2
---- ----
V1.1 V2.1
V2.2 V1.2
If you want to generate an HTML table instead of a table for terminal output:
echo '[{
"name": "George",
"id": 12,
"email": "george#domain.example"
}, {
"name": "Jack",
"id": 18,
"email": "jack#domain.example"
}, {
"name": "Joe",
"id": 19,
"email": "joe#domain.example"
}]' | jq -r 'map("<tr><td>" + .name + "</td><td>" + (.id | tostring) + "</td></tr>") | ["<table>"] + . + ["</table>"] | .[]'
Output:
<table>
<tr><td>George</td><td>12</td></tr>
<tr><td>Jack</td><td>18</td></tr>
<tr><td>Joe</td><td>19</td></tr>
</table>
If the values don't contain spaces, this might be helpful:
read -r -a data <<<'name1 value1 name2 value2'
echo "name value"
echo "=========="
for ((i=0; i<${#data[#]}; i+=2)); do
echo ${data[$i]} ${data[$((i+1))]}
done
Output
name value
==========
name1 value1
name2 value2
More simple implement:
jq -r '(.[0]|keys_unsorted|(.,map(length*"-"))),.[]|map(.)|#tsv'|column -ts $'\t'
you can add the following jq function into ~/.jq:
def pretty_table:
(.[0]|keys_unsorted|(.,map(length*"-"))),.[]|map(.)|#tsv
;
and then run:
cat apps.json | jq -r pretty_table | column -ts $'\t'