Filter a JSON document in Ansible - json

I have a JSON reply from a GitHub repository with a list of possible downloads for a certain release (the assets array in the document).
I want to get the browser download URL when the name of an asset ends with x64.AppImage.
In Ansible, the filters are built apon jmespath and using its terminal tool, I can query the url with the following expression:
assets[?ends_with(name, 'x64.AppImage')].browser_download_url
With the following playbook, the JSON document is queried and stored in the json_reply variable.
---
- hosts: local
tasks:
- name: Get list of Rambox releases
uri:
url: "https://api.github.com/repos/saenzramiro/rambox/releases/latest"
body_format: json
register: json_reply
- name: Filter reply
debug: URL -> "{{ item }}"
with_items:
- "{{ json_reply.json | json_query(json_filter) }}"
vars:
- json_filter: assets[?ends_with(name, 'x64.AppImage')].browser_download_url
However, executing this gives the following error:
fatal: [localhost]: FAILED! => {
"msg": "JMESPathError in json_query filter plugin:\nIn function ends_with(), invalid type for value: latest-mac.json, expected one of: ['string'], received: \"unknown\""
}
Where latest-mac.json is the first object in the assets array.
How can I make Ansible to iterate over all the assets array and apply my filter?
PS:
If instead of querying if the name ends with a word I specify it directly, the filter works:
assets[?name == 'Rambox-0.5.13-x64.AppImage')].browser_download_url
JSON example:
{
"url": "https://api.github.com/repos/saenzramiro/rambox/releases/8001922",
"prerelease": false,
"created_at": "2017-10-04T21:14:15Z",
"published_at": "2017-10-05T01:10:55Z",
"assets": [
{
"url": "https://api.github.com/repos/saenzramiro/rambox/releases/assets/4985942",
"id": 4985942,
"name": "latest-mac.json",
"uploader": {
"login": "saenzramiro",
"id": 2694669
},
"browser_download_url": "https://github.com/saenzramiro/rambox/releases/download/0.5.13/latest-mac.json"
},
{
"url": "https://api.github.com/repos/saenzramiro/rambox/releases/assets/4985640",
"id": 4985640,
"name": "Rambox-0.5.13-x64.AppImage",
"uploader": {
"login": "saenzramiro",
"id": 2694669
},
"browser_download_url": "https://github.com/saenzramiro/rambox/releases/download/0.5.13/Rambox-0.5.13-x64.AppImage"
}
],
"tarball_url": "https://api.github.com/repos/saenzramiro/rambox/tarball/0.5.13"
}

The problem of type errors in JMESPath filters is discussed in issue 27299.
You can use this patched json_query.py filter plugin.
Or apply double conversion to your object as a workaround: | to_json | from_json |.
This will convert object to JSON (thus plain strings) and back, so json_query will treat strings as supported type.

Loop through each asset
Print the browser URL of the item if it ends with x64.AppImage
Solution not using JMESPath:
- name: Filter reply
debug: var=item.browser_download_url
with_items: "{{ json_reply.json.assets }}"
when: item.browser_download_url | regex_search('x64.AppImage$')

As #helloV said, you can accomplish this using Ansible loops, although there's no reason to involve a regular expression match. You can use the same test you're already using:
- name: Filter reply
debug:
var: item.browser_download_url
with_items: "{{ json_reply.json.assets }}"
when: item.name.endswith('x64.AppImage')
The root problem would appear to be an Ansible bug. The error comes from the following check in the jmespath library:
if actual_typename not in allowed_types:
raise exceptions.JMESPathTypeError(
function_name, current,
self._convert_to_jmespath_type(actual_typename), types)
At the point this code is called, the data type of values in your json response is AnsibleUnsafeText, where as allowed_types is [str, unicode]. I think the transformation of values from native types to the AnsibleUnsafeText type probably is some sort of standard Ansible module behavior being imposed by the uri module. We can work around it by using curl instead, like this:
- name: Get list of Rambox releases
command: >
curl -s "https://api.github.com/repos/saenzramiro/rambox/releases/latest"
register: json_reply
And then:
- name: Filter reply
debug:
var: item.browser_download_url
with_items: >
{{ json_reply.stdout|from_json|json_query('assets[?ends_with(name, `x64.AppImage`)]') }}

Related

Ansible using variables properly for uri request

Something is wrong with the way I am substituting values and constructing a json template in ansible to be sent via the uri module to a webserver. Can anyone assist?
It seems like ansible is converting all the double brackets in the json to single brackets when it is running the replace command
Full Example:
I have a variable file where I am storing secrets:
ie secrets.yml
---
username: "admin"
password: "admin"
I am reading these in and replace the values in a json template:
ie template.json
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}
the playbook has tasks structured thus
- name: load template
set_fact:
loginTemplate: "{{ lookup('file', 'template.json') }}"
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) }}"
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate }}"
body_template: json
If I put the values in the json and just submit without the extra set_fact step to fill in the values it works.
Try converting the request body to json content using to_json filter in set_fact.
- name: replace values with actuals
set_fact:
filledTemplate: "{{ loginTemplate |
replace('***USERNAME_FIELD***', username) |
replace('***PASSWORD_FIELD***', password) | to_json }}"
or in uri module,
- name: submit request
uri:
url: "http://blah.com/api"
method: POST
body: "{{ filledTemplate | to_json }}"
body_template: json
Also, are you missing : in the json file like?
{
"username": "***USERNAME_FIELD***",
"password": "***PASSWORD_FIELD***"
}

Ansible: Invalid JSON when using --extra-vars

Hi community,
I have been struggling with an issue in ansible issue for days now.
Everything is executed wihtin a Jenkins pipeline.
The ansible command looks like:
sh """
ansible-playbook ${env.WORKSPACE}/cost-optimization/ansible/manage_dynamo_db.yml \
--extra-vars '{"projectNameDeployConfig":${projectNameDeployConfig},"numberOfReplicas":${numberOfReplicas},"dynamodbtask":${dynamodbtask}}'
"""
And the playbooks is:
playbook.yml
---
- hosts: localhost
vars:
numberOfReplicas: "{{numberOfReplicas}}"
dynamodbtask: "{{dynamodbtask}}"
namespace: "{{projectNameDeployConfig}}"
status: "{{status}}"
- tasks:
- name: "Get replica number for the pods"
command: aws dynamodb put-item --table-name pods_replicas
register: getResult
when: dynamodbtask == "get"
- name: "Update replica number for specified pods"
command: |
aws dynamodb put-item
--table-name pods_replicas
--item '{"ProjectNameDeployConfig":{"S":{{namespace}}},"NumberReplicas":{"N":{{numberOfReplicas}}}}'
register: updatePayload
when: dynamodbtask == "put" and getResult is skipped
However, there is always the following error:
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["aws", "dynamodb", "put-item", "--table-name",
"pods_replicas", "--item", "{\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":
{\"N\":0}}"], "delta": "0:00:01.702107", "end": "2020-02-09 16:58:26.055579",
"msg": "non-zero return code", "rc": 255, "start": "2020-02-09 16:58:24.353472", "stderr": "\nError parsing parameter '--item': Invalid JSON: No JSON object could be decoded\nJSON received: {\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":{\"N\":0}}", "stderr_lines": ["", "Error parsing parameter '--item': Invalid JSON: No JSON object could be decoded", "JSON received: {\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":{\"N\":0}}"], "stdout": "", "stdout_lines": []}
There are two answers to your question: the simple one and the correct one
The simple one is that had you actually fed the JSON into jq, or python -m json.tool, you would have observed that namespace is unquoted:
"{\"ProjectNameDeployConfig\":{\"S\": LERN-PolicyCenterV10 },\"NumberReplicas\": {\"N\":0}}"
where I added a huge amount of space, but didn't otherwise alter the quotes
The correct answer is that you should never use jinja2 to try and assemble structured text when there are filters that do so for you.
What you actually want is to use the to_json filter:
- name: "Update replica number for specified pods"
command: |
aws dynamodb put-item
--table-name pods_replicas
--item {{ dynamodb_item | to_json | quote }}
vars:
dynamodb_item:
"ProjectNameDeployConfig":
"S": '{{ projectNameDeployConfig }}'
"NumberReplicas":
"N": 0
register: updatePayload
when: dynamodbtask == "put" and getResult is skipped
although you'll notice that I changed your variable name because namespace is the name of a type in jinja2, so you can either call it ns or I just used the interpolation value from your vars: block at the top of the playbook, as it doesn't appear that it changed from then

extracting a variable from json output then debug and register the outout with ansible

Hi I have a problem of getting one of the variables extracted from a json output after doing a curl to be parsed and registered back to ansible
Playbook:
- name: debug stdout
debug:
msg: "{{ result.stdout | from_json }}"
register: dataresult
- name: debug fact
debug:
msg: "{{ dataresult.data.start_time_string }}"
output :
TASK [backup_api : debug stdout]
***********************************************
task path: /home/ansible/cm-dha/roles/backup_api/tasks/main.yml:36
ok: [127.0.0.1] => {
"msg": {
"data": [
{
"backup_id": 40362,
"certified": null,
"instance_id": 148,
"start_time": 1506985211,
"start_time_string": "10/03/2017 03:00:11 am"
}
],
"timestamp": 1507022232
}
}
error:
fatal: [127.0.0.1]: FAILED! => {
"failed": true,
"msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'data'\n\nThe error appears to have been in '/home/ansible/cm-dha/roles/backup_api/tasks/main.yml': line 48, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: debug fact\n ^ here\n"
The error is happening when trying to extract the value start_time_string
so how to do it probably as I tried too many things like using with_items, with_dict , simulating the data[] output to debug and even doing a json query but without success
so any help here?
Don't use debug to assign facts, use set_fact instead:
- name: debug stdout
set_fact:
dataresult: "{{ result.stdout | from_json }}"
- name: debug fact
debug:
msg: "{{ dataresult.data[0].start_time_string }}"
Thanks to accepted answer, I've made it as below for my case. Leaving it here, maybe it helps someone.
- name: Connectors
shell: curl -X GET http://myserver:8083/connectors
register: out6
- set_fact:
dataresult: "{{ out6.stdout | from_json }}"
- debug: var=dataresult

ansible parse json array reply from api

I am trying to parse a json response from an API. The response in a browser looks like:
[{url: "abc.com/xyz"}]
I request it from ansible:
- name: Get url
uri:
url: my-url...
method: GET
force: yes
return_content: yes
#HEADER_Content-Type: "application/json"
register: json_response
I get a reply from ansible that looks like this (with debug):
- name: print reply
debug:
var: json_response
verbosity: 1
which gives:
ok: [server] => {
"json_response": {
... //removed for readability
"content": "({:url \"https://the-file-I-want\"})"
}
So it seems like some parsing happened already (note the colons :).
Accessing the content seem to work (with debug json_response['content']):
ok: [server] => {
"json_response['content']": "({:url \"https://the-file-I-want\"})"
}
But I cannot seems to access the json response url. If I try to take the first element of the array, I get "(" so it seems it is still a string.
- name: print reply2
debug:
var: json_response['content'][0]
verbosity: 1
from_json does not seem to work: fatal: [server]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined....
How do I parse a json reply like this one?
I created a json file response.json with the following contents:
{
content: ({:url \"https://the-file-I-want\"})
}
Then, in my playbook I loaded the file and to get the url you need, I created a custom jinja filter since Jinja2 does not have any filter for finding sub-string or regexp.
My custom filter named filter.py(you can name it anything) is in a dir called filter_plugins in the same directory as my playbook. My filter.py file is as follows:
import re
class FilterModule(object):
''' Custom filters are loaded by FilterModule objects '''
def filters(self):
return {'urlsubstr': self.urlsubstr}
def urlsubstr(self,content):
url = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', content)
return url[0]
After creating the custom filter, I got the url like this:
- hosts: localhost
vars:
json_response: "{{ lookup('file', 'response.json') | from_json }}"
tasks:
- debug: msg="{{ json_response.content | urlsubstr }}"
with_dict: "{{ json_response }}"
This is the output of running my playbook:
TASK [setup] *******************************************************************
ok: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => (item={'value': u'({:url "https://the-file-I-want"})', 'key': u'content'}) => {
"item": {
"key": "content",
"value": "({:url \"https://the-file-I-want\"})"
},
"msg": "https://the-file-I-want"
}
Hope this helps.
To make the response json use the to_json filter, then navigate to the url key. That should give the value you're looking for: https://.....
Here's the documentation:
http://docs.ansible.com/ansible/playbooks_filters.html#filters-for-formatting-data
The original response works on any case but it seems overkill unless there's a problem with the conversion to JSON, like in yours.
Nevertheless I think it might help me do something else I intend to do, which is why I was looking in here.

how it I parse each row of data from the google sheets api with ansible's with_items?

I am using the Google Sheets V4 Values collection and I am having trouble figuring out how to get each row to parse in to an {{ item }}
My Ansible ymal looks like.
tasks:
- name: debug url
debug:
msg: "{{ g_url }}"
- name: GET data from google spead sheet api
uri:
url: "{{ g_url }}"
return_content: yes
dest: /tmp/o_gd_form.json
register: google_data
- name: whats the dump?
debug:
var: "{{ item.0 |to_json }}"
with_items: "{{ google_data.json.values() }}" # this is the line that needs to be fixed
And the the responding json looks like:
{
"range": "Sheet1!A1:D5",
"majorDimension": "ROWS",
"values": [
["Item", "Cost", "Stocked", "Ship Date"],
["Wheel", "$20.50", "4", "3/1/2016"],
["Door", "$15", "2", "3/15/2016"],
["Engine", "$100", "1", "30/20/2016"],
["Totals", "$135.5", "7", "3/20/2016"]
],
}
I can't seem figure out how to write the the with_items to return an {{ item }} of json like ["Engine", "$100", "1", "30/20/2016"].
any help or do I need to split this task out to some middleware parser?
The Google sheets api documentation is at:
https://developers.google.com/sheets/samples/reading
To get what you want, use:
- debug: msg="Name={{ item.0 }} Cost={{ item.1 }}"
with_list: "{{ google_data.json['values'] }}"
There are two problems in your code:
values is a special keyword, so you can't access json.values, but should use json['values'] instead.
with_items flattens the list of lists, so you end up with long list of strings. You should use with_list to iterate over outer list (rows) and get inner lists (column values) as items.