ansible - parsing json w jmesquery - json

I'm querying Splunk via the uri module w a certain search:
- name: splunk query
uri:
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=json
method: POST
user: ...
password: ...
validate_certs: false
body_format: form-urlencoded
return_content: true
headers:
Content-Type: application/json
body:
- [ search, "{{ splunk_search }}" ]
vars:
- splunk_search: '| ...'
register: me
Splunk returns a 200, w content:
TASK [debug] ********************************************************************************************************************************************************************************************
ok: [host1] => {
"msg": "{\n \"title\": \"clientMessageId\",\n \"app\": \"whatever\"\n}\n{\n \"title\": \"a_title\",\n \"app\": \"some_app\"\n}\n{\n \"title\": \"another_title\",\n \"app\": \"my_app\"\n}\n{\n \"title\": \"title_for_app\",\n \"app\": \"another_app\"\n}"
}
However, I can't properly parse the output, I've tried:
- name: query result from content
debug:
msg: "{{ me.content | json_query('title') }}"
In this case, Ansible will return an empty string. I assume something is wrong with the formatting of the output.
Q: How can I configure Ansible to properly parse the json output, so that a new list is created based on the output.
Example:
my_list:
- title: clientMessageId
app: whatever
- title: a_title
app: some_app
...

What I did was change the url output_mode in which Splunk returns the content from json to csv.
url: https://localhost:8089/servicesNS/admin/search/search/jobs/export?output_mode=csv
Then, I wrote the output to a file and read it using the ansible read_csv module.
- name: write received content to csv file on controller
copy:
content: "{{ me.content }}"
dest: "./file.csv"
mode: '0775'
delegate_to: localhost
- name: read csv file
read_csv:
path: ./file.csv
delegate_to: localhost
register: read_csv
- name: set fact to create list
set_fact:
the_output: "{{ read_csv.list }}"

Related

Extract part of JSON in ansible playbook

I want to extract a part from a json extra vars input and use this as a variable in further commands.
The extra vars being parsed towards ansible is:
{
"problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2",
}
I want to extract the part -5484403941961857966_1631165040000V2 and store it into a variable.
- name: get pid from URL
set_fact:
pidproblem: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(.+)', '\\1') }}"
- name: show pid
debug:
var: pidproblem[0]
- name: update problem with output
when: state == "OPEN"
uri:
url: https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/"{{ pidproblem[0] }}"/comments
method: POST
headers:
Content-Type: application/json; charset=utf-8
Authorization: Api-Token xxxxx
body_format: json
body: "{\"message\":\"TEST\",\"context\":\"TEST\"}"
Could the issue reside in the fact that the id is subsituded as "6551567569324750926_1631192580000V2" instead of 6551567569324750926_1631192580000V2?
"url": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/\"6551567569324750926_1631192580000V2\"/comments"
There is a urlsplit filter which can split a URL into known segments. We can use this to break down the URL and get the last fragment, i.e.
"{{ problemUrl | urlsplit('fragment') }}"
Gives...
problems/problemdetails;pid=-5484403941961857966_1631165040000V2
Now this gives us a more "manageable" string. We can do a regex_search (with groups) on this, to get the pid, like:
- name: get pid from URL
set_fact:
pid: "{{ problemUrl | urlsplit('fragment') | regex_search('pid=(-.+)', '\\1') }}"
- name: show pid
debug:
var: pid[0]
- name: update problem with output
uri:
url: "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/api/v2/problems/{{ pid[0] }}/comments"
# other params
Not super-reliable as we don't know how your url can change, but you could use some regex filter to extract the pid value:
- hosts: localhost
vars:
problemUrl: '{ "problemUrl": "https://xxxxx.xxxxxxxxx-xxxxx.xxxx/e/58b59a93-xxxx-xxxx-xxxx-91bb5ca1f41c/#problems/problemdetails;pid=-5484403941961857966_1631165040000V2;other=false" }'
tasks:
- name: set_fact some paramater
set_fact:
pid: "{{ (problemUrl | from_json).problemUrl | regex_replace('.*pid=(?P<pid>[^;]*).*', '\\g<pid>') }}"
- name: "update"
debug:
msg: "{{ pid }}"

How to POST Json file with arguments using Ansible

All
I am using Ansible to POST data on a website. I have taken a very simple example of the JSON content
{
"test2":"{{test}}"
}
Below is snippet for ansible task responsible to POST data:
vars:
test: "somerandomtest"
tasks:
- name: Test
register: json_output
uri:
url: "http://localhost:3000"
validate_certs: no
follow_redirects: none
headers:
Authorization: Bearer 12344
Content-Type: application/json
method: POST
body_format: json
body: " {{ lookup('file','test.json') | from_json }} "
#body: '{"test2":" {{test}} "}'
timeout: 600
But on the web service I see the below data, where "test" is not replaced with "somerandomtest".
{'test2': '{{test}}'}
Not sure what I am doing wrong, or maybe I am using lookup the wrong way. If I replace the body with the content everything is fine, but the real JSON I work with is 400 lines, and I want to keep it separate.
Can anyone please let me know what can I do to change the JSON data using Ansible?
The file lookup doesn't evaluate file content for Jinja template markers (which would lead to unexpected behavior if a document just happened to contain {{).
If you want to read in a template, use the template lookup instead:
- hosts: localhost
gather_facts: false
tasks:
vars:
test: "somerandomtest"
tasks:
- name: Test
register: json_output
uri:
url: "https://httpbin.org/post"
validate_certs: no
follow_redirects: none
headers:
Content-Type: application/json
method: POST
body_format: json
body: " {{ lookup('template','test.json') }} "
return_content: true
timeout: 600
- debug:
var: json_output
This shows that the content sent to the remote server is:
{'test2': 'somerandomtest'}
Put the hash into a dictionary and format it to_json. For example
- hosts: localhost
vars:
test: somerandomtest
body:
test2: "{{ test }}"
tasks:
- debug:
msg: "{{ body|to_json }}"
gives the JSON
msg: '{"test2": "somerandomtest"}'
The template gives the same result
shell> cat test.json
{
"test2": "{{ test }}"
}
- debug:
msg: "{{ lookup('template','test.json')|to_json }}"

Multiline json string in ansible task

In an ansible task I am trying to break a long JSON string in a HTTP POST request into multiple lines based on:
In YAML, how do I break a string over multiple lines?
I have tried:
- name: "Test POST request"
uri:
url: "{{ HOST }}/api/"
method: POST
return_content: yes
body: >-
"{\"id\":\"{{ app_id }}\",
\"name\":\"prefix-{{ name }}\",
\"type\":\"ds\",
\"typeLogoUrl\":\"\",
\"access\":\"all\",
\"url\":\"{{ HOST_URL }}",
\"password\":\"\",
\"user\":\"\",
\"database\":\"\",
\"jsonData\":{\"a\":\"{{ a_var }}\",\"b\":true,\"c\":\"{{ c_var }}\"},
\"secureJsonFields\":{}}"
body_format: json
user: "{{ user }}"
password: "{{ password }}"
force_basic_auth: yes
headers:
Content-Type: application/json
But when I run it I get errors, summarized below:
[{\"classification\":\"DeserializationError\",\"message\":\"invalid character '\\\\n' in string literal\"}
Any suggestion on how to break this down into multiple without above error?
You specified body_format: json, so you can write your body in yaml
body:
id: "{{ org_id }}"
name: "prefix-{{ namespace }}"
type: datasource
typeLogoUrl: ""
access: proxy
url: "{{ HOST_URL }}"
password: ""
user:""
database: ""
jsonData:
a: "{{ a_var }}"
b: true
c: "{{ c_var }}"
secureJsonFields: ""

Ansible: creating variables from JSON response

I am struggling on understanding how to get the response from one uri call to pass to another. I am creating a playbook for my storage system. It requires a SessionKey for all REST calls passed in a custom header "X-HP3PAR-WSAPI-SessionKey"
The initial authentication REST call works succeeds and I see the SessionKey. I am unable to get the SessionKey to be passed to the subsequent call.
Any suggestions?
---
- name: Connect to 3par
hosts: localhost
become: no
vars:
auth_3par_user: "3paruser"
auth_3par_password: "3parpass"
ip_address_3par: 1.1.1.1
rest_api_url_3par: "https://{{ ip_address_3par }}:8080/api/v1"
tasks:
- name: check if 3par WSAPI is running
uri:
url: "{{ rest_api_url_3par }}/credentials"
force_basic_auth: yes
method: POST
HEADER_Content-Type: application/json
body_format: json
body: "{ 'user': '{{ auth_3par_user }}', 'password': '{{ auth_3par_password }}' }"
status_code: 201
return_content: yes
validate_certs: no
register: output
- name: Parsing key
debug:
msg: "{{ output.json.key }}"
- name: GET 3par volumes
uri:
url: "{{ rest_api_url_3par }}/volumes"
method: GET
HEADER_Content-Type: "application/json"
headers:
"X-HP3PAR-WSAPI-SessionKey": "{{ output.json.key }}"
status_code: 200
return_content: yes
validate_certs: no
register: volume_output
- name: Parsing Volumes GET
debug:
var={{ volume_output.stdout | from_json }}
TASK [Parsing key] *************************************************************
task path: /home/chris/ansible/ansible_3par/playbooks/connect_3par.yml:26
ok: [localhost] => {
"msg": "0-f150f34f975693b3c95efa84b13bd605-00cc315a"
}
TASK [check if 3par WSAPI is running] ******************************************
task path: /home/chris/ansible/ansible_3par/playbooks/connect_3par.yml:30
ESTABLISH LOCAL CONNECTION FOR USER: chris
127.0.0.1 EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1513213268.76-220322452758647 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1513213268.76-220322452758647 )" )
127.0.0.1 PUT /tmp/tmpgOtWjh TO /home/chris/.ansible/tmp/ansible-tmp-1513213268.76-220322452758647/uri
127.0.0.1 EXEC LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/chris/.ansible/tmp/ansible-tmp-1513213268.76-220322452758647/uri; rm -rf "/home/chris/.ansible/tmp/ansible-tmp-1513213268.76-220322452758647/" > /dev/null 2>&1
fatal: [localhost]: FAILED! => {"changed": false, "connection": "close", "content": "{\"code\":6,\"desc\":\"invalid session key\"}",
Ansible version 2.0.0.2
You are mixing old and new syntax for headers in uri module.
And as you use an antiquated Ansible 2.0 version, headers parameter is completely ignored.
Upgrade to the current Ansible version.
For reference:
if you use Ansible < 2.1 define all headers using HEADER_*,
if you use Ansible >= 2.1 define all headers under headers.
Posting the corrected, functional code for reference after fixing headers and updating Ansible to latest version.
---
- name: Connect to 3par
hosts: localhost
become: no
vars:
auth_3par_user: "3paruser"
auth_3par_password: "3parpass"
ip_address_3par: 1.1.1.1
rest_api_url_3par: "https://{{ ip_address_3par }}:8080/api/v1"
tasks:
- name: check if 3par WSAPI is running
uri:
url: "{{ rest_api_url_3par }}/credentials"
method: POST
headers:
Content-Type: "application/json"
body_format: json
body: "{ 'user': '{{ auth_3par_user }}', 'password': '{{ auth_3par_password }}' }"
status_code: 201
return_content: yes
validate_certs: no
register: output
- name: Parsing key
debug:
msg: "{{ output.json.key }}"
- name: GET 3par volumes
uri:
url: "{{ rest_api_url_3par }}/volumes"
method: GET
headers:
Content-Type: "application/json"
X-HP3PAR-WSAPI-SessionKey: "{{ output.json.key }}"
Accept: "application/json"
status_code: 200
return_content: yes
validate_certs: no
register: volume_output
- name: Parsing Volumes GET
debug:
msg: "{{ volume_output }}"enter code here

Ansible: access json field and use it as variable

I have to send some PUT requests to a server, at the URL: https://XXXX.com/id, and passing as the body some json files (item1.json, item2.json ...).
- name: invoke service
uri:
url: "https://XXXX.com/{{ item.id }}"
method: PUT
return_content: yes
body_format: json
headers:
Content-Type: "application/json"
X-Auth-Token: "XXXXXX"
body: "{{ lookup('file', item) }}"
with_items:
- item1.json
- item2.json
- item3.json
The id parameter of the url is within the respective json file. The structure of the json files is the following:
{
"address": "163.111.111.111",
"id": "ajsaljlsaaj",
"server": "dnnwqkwnlqkwnldkwqldn"
}
The code I have written seems not to work, and I get 'ansible.vars.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'id'.
So how can that field been accessed?
The issue is that in the following line:
url: "https://XXXX.com/{{ item.id }}"
The value of item is the JSON file name as defined in with_items, not the content of the JSON file.
The quickest fix is to open and parse the JSON file the same way you do in your body: declaration:
- name: invoke service
uri:
url: "https://XXXX.com/{{ ( lookup('file', item)|from_json ).id }}"
method: PUT
return_content: yes
body_format: json
headers:
Content-Type: "application/json"
X-Auth-Token: "XXXXXX"
body: "{{ lookup('file', item) }}"
with_items:
- item1.json
- item2.json
- item3.json
One nicer solution would be to use the with_file: directive instead of with_items.
with_file would automatically open and read the file content, so no need to call lookup anymore:
- name: Provision the Docker Swarm managers
hosts: localhost
tags: provision
gather_facts: False
become: True
tasks:
- name: invoke service
uri:
url: "https://XXXX.com/{{ (item|from_json).id }}"
method: PUT
return_content: yes
body_format: json
headers:
Content-Type: "application/json"
X-Auth-Token: "XXXXXX"
body: "{{ item }}"
with_file:
- item1.json
- item2.json
- item3.json