Processing JSON from REST call in Ansible [duplicate] - json

I'm currently using Ansible 1.7.2. I have the following test playbook:
---
- hosts: localhost
tasks:
- name: set fact 1
set_fact: foo="[ 'zero' ]"
- name: set fact 2
set_fact: foo="{{ foo }} + [ 'one' ]"
- name: set fact 3
set_fact: foo="{{ foo }} + [ 'two', 'three' ]"
- name: set fact 4
set_fact: foo="{{ foo }} + [ '{{ item }}' ]"
with_items:
- four
- five
- six
- debug: var=foo
The first task sets a fact that's a list with one item in it. The subsequent tasks append to that list with more values. The first three tasks work as expected, but the last one doesn't. Here's the output when I run this:
PLAY [localhost] **************************************************************
GATHERING FACTS ***************************************************************
ok: [localhost]
TASK: [set fact 1] ************************************************************
ok: [localhost]
TASK: [set fact 2] ************************************************************
ok: [localhost]
TASK: [set fact 3] ************************************************************
ok: [localhost]
TASK: [set fact 4] ************************************************************
ok: [localhost] => (item=four)
ok: [localhost] => (item=five)
ok: [localhost] => (item=six)
TASK: [debug var=foo] *********************************************************
ok: [localhost] => {
"foo": [
"zero",
"one",
"two",
"three",
"six"
]
}
PLAY RECAP ********************************************************************
localhost : ok=6 changed=0 unreachable=0 failed=0
Given the with_items in task 4 and the fact that the output shows the task properly iterated over the items in that list, I would have expected the result to contain all the numbers zero through six. But that last task seems to only be evaluating set_fact with the last item in the list. Is this possibly a bug in Ansible?
Edit: I also just tested this on ansible 1.8 and the output was identical.

There is a workaround which may help. You may "register" results for each set_fact iteration and then map that results to list:
---
- hosts: localhost
tasks:
- name: set fact
set_fact: foo_item="{{ item }}"
with_items:
- four
- five
- six
register: foo_result
- name: make a list
set_fact: foo="{{ foo_result.results | map(attribute='ansible_facts.foo_item') | list }}"
- debug: var=foo
Output:
< TASK: debug var=foo >
---------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
ok: [localhost] => {
"var": {
"foo": [
"four",
"five",
"six"
]
}
}

As mentioned in other people's comments, the top solution given here was not working for me in Ansible 2.2, particularly when also using with_items.
It appears that OP's intended approach does work now with a slight change to the quoting of item.
- set_fact: something="{{ something + [ item ] }}"
with_items:
- one
- two
- three
And a longer example where I've handled the initial case of the list being undefined and added an optional when because that was also causing me grief:
- set_fact: something="{{ something|default([]) + [ item ] }}"
with_items:
- one
- two
- three
when: item.name in allowed_things.item_list

I was hunting around for an answer to this question. I found this helpful. The pattern wasn't apparent in the documentation for with_items.
https://github.com/ansible/ansible/issues/39389
- hosts: localhost
connection: local
gather_facts: no
tasks:
- name: set_fact
set_fact:
foo: "{{ foo }} + [ '{{ item }}' ]"
with_items:
- "one"
- "two"
- "three"
vars:
foo: []
- name: Print the var
debug:
var: foo

Jinja 2.6 does not have the map function. So an alternate way of doing this would be:
set_fact: foo="{% for i in bar_result.results %}{{ i.ansible_facts.foo_item }}{%endfor%}"

Below works for me:
- name: set fact
set_fact:
foo_item: "{{foo_item | default([]) + [item]}}"
loop:
- four
- five
- six

Updated 2018-06-08: My previous answer was a bit of hack so I have come back and looked at this again. This is a cleaner Jinja2 approach.
- name: Set fact 4
set_fact:
foo: "{% for i in foo_result.results %}{% do foo.append(i) %}{% endfor %}{{ foo }}"
I am adding this answer as current best answer for Ansible 2.2+ does not completely cover the original question. Thanks to Russ Huguley for your answer this got me headed in the right direction but it left me with a concatenated string not a list. This solution gets a list but becomes even more hacky. I hope this gets resolved in a cleaner manner.
- name: build foo_string
set_fact:
foo_string: "{% for i in foo_result.results %}{{ i.ansible_facts.foo_item }}{% if not loop.last %},{% endif %}{%endfor%}"
- name: set fact foo
set_fact:
foo: "{{ foo_string.split(',') }}"

Looks like this behavior is how Ansible currently works, although there is a lot of interest in fixing it to work as desired. There's currently a pull request with the desired functionality so hopefully this will get incorporated into Ansible eventually.

Related

Syntax to check if var

I have a playbook to get all disks letter configured on my server and I need a task to verify if extra var letter is on the list.
For example I need to check if "F" is on the json data below.
Could you please help me on the best best syntax?
Thanks
{
"disks_drives_letter": [
[
"C"
],
[
"D"
],
[
"E"
],
[]
]
}
You can use setup module to get your host information like disks. For more information about the setup module https://docs.ansible.com/ansible/latest/collections/ansible/builtin/setup_module.html
Example of playbook:
- hosts: localhost
tasks:
gather_facts: false
vars:
my_disk_drives: ['sda', 'sdb']
tasks:
- name: Collect host hardware information
setup:
gather_subset:
- hardware
- name: Output if disk exist
debug:
msg: "{{ item }} exists"
loop: "{{ my_disk_drives }}"
when: item in hostvars[inventory_hostname].ansible_devices.keys() | list
- name: Output if disks does not exist
debug:
msg: "{{ item }} does not exist"
loop: "{{ my_disk_drives }}"
when: not item in hostvars[inventory_hostname].ansible_devices.keys() | list
Output:
TASK [Output if disk exist]
ok: [localhost] => (item=sda) => {
"msg": "sda exists"
}
skipping: [localhost] => (item=sdb)
TASK [Output if disks does not exist]
skipping: [localhost] => (item=sda)
ok: [localhost] => (item=sdb) => {
"msg": "sdb does not exist"
}
Use filters intersect and difference, and declare the lists
my_disks_exist: "{{ ansible_devices.keys()|intersect(my_disks) }}"
my_disks_not_exist: "{{ my_disks|difference(my_disks_exist) }}"
Example of a complete playbook for testing
- hosts: localhost
vars:
my_disks: [sda, sdb, sdc]
my_disks_exist: "{{ ansible_devices.keys()|intersect(my_disks) }}"
my_disks_not_exist: "{{ my_disks|difference(my_disks_exist) }}"
tasks:
- setup:
gather_subset: devices
- debug:
var: ansible_devices.keys()
- debug:
var: my_disks_exist
- debug:
var: my_disks_not_exist
| flatten help me thanks #vladimir-botka
- name: Get all disks letter from the disks infos
set_fact:
disks_drives_letters: "{{ win_disk_facts | json_query(query) | flatten }}"
- name: Check if disk_letter is used on server fail:
msg: "The disk letter already exist on the VM" when: '"{{ drive_letter }}" in "{{ disks_drives_letters}}"'

JMESPath Query in Ansible

--- EDIT ---
Partial-Solution: Messed around with the JMESPath syntax and was able to successfully get a match for the first test case (without the optional variable) using:
jmesquery: "{{ datacenter }}{{ subcategory }}.{{ refine_hosts }}.[*][].[*][][]"
I am writing an Ansible Playbook that takes a list of hosts from a network server, parses the JSON list, and finds hostnames that matches the user's input when they deploy the playbook as a Jenkin's Job through it's API.
The issue I am encountering is that I am unable to successfully query the JSON host list. Currently, I am only trying to run the following test case:
datacenter: a
subcategory: bc
refine_hosts: QA
However, the final version of this playbook should be able to take in values for datacenter, subcategory, and refine_hosts with an optional input value of host_type. An example test case including the optional input value would be the following:
datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
In my playbook, I am using JMESPath within the following task:
- name: Build HOSTS list
set_fact:
hosts_list: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: '%%datacenter%%-%%subcategory%%.%%refine_hosts%%.[*][*][][]'
The JSON host list is structured in the following manner (I am unable to edit the structure of the host list, but it will always follow the following structure nonetheless):
{
"a-bc":{
"all":{
"webServer":[
],
"archive":[
"someHostAlias-123.privateDomain.com"
],
"central":[
"someHostAlias-456.privateDomain.com"
]
},
"QA":{
"xyz":{
"INBOUND_HTTP":[
"someHostAlias-789.privateDomain.com"
],
"WEBSITE":[
"someHostAlias-1011.privateDomain.com"
]
}
}
}
}
I have been using the following websites for this issue:
JMESPath Tutorial
Ansible JMESPath Documentation
JSONPath Expression Tester
StackOverflow: How to Use Variable in JMESPath Expression
Gitter: JMESPath/chat
I apologize if the query seems obvious, this is my first attempt at an Ansible Playbook. All help/feedback is greatly appreciated.
One of the issue of your query is that you are confusing [*] — a list projection — that selects all the elements of a list with .* — an object projection — that selects all the properties of a dictionary.
So, one solution in JMESPath, would be to do:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
Given the playbook:
- hosts: localhost
gather_facts: no
tasks:
- debug:
msg: "{{ jsondata | json_query(jmesquery) }}"
loop: "{{ fake_user_input }}"
loop_control:
label: "{{ jmesquery }}"
vars:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
datacenter: "{{ item.datacenter }}"
subcategory: "{{ item.subcategory }}"
refine_hosts: "{{ item.refine_hosts }}"
host_type: "{{ item.host_type | default('') }}"
fake_user_input:
- datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
- datacenter: a
subcategory: bc
refine_hosts: QA
jsondata:
a-bc:
all:
webServer: []
archive:
- someHostAlias-123.privateDomain.com
central:
- someHostAlias-456.privateDomain.com
QA:
xyz:
INBOUND_HTTP:
- someHostAlias-789.privateDomain.com
WEBSITE:
- someHostAlias-1011.privateDomain.com
This yields:
ok: [localhost] => (item="a-bc".QA.*. WEBSITE[] | []) =>
msg:
- someHostAlias-1011.privateDomain.com
ok: [localhost] => (item="a-bc".QA.*. *[] | []) =>
msg:
- someHostAlias-789.privateDomain.com
- someHostAlias-1011.privateDomain.com

I'm trying to get ansible output formatted like item1 - item2 instead of two separate lists of items

I have a functioning ansible script that connects to AWS and prints out tags from instances in a specified state.
The problem I'm having is the print out is two separate lists e.g.
What I get is:
name1
name2
name3
description1
description2
description3
What I would like is:
name1 - description1
name2 - description2
name3 - description3
I've tried pushing this all into a dictionary, but got lost in the attempt. There must be an easier way.
Here's my code:
- name: print stopped systems
hosts: all
become: false
tasks:
- name: Gather ec2_metadata_facts (use -vv to show all)
action: ec2_metadata_facts
- name: pull instance info with ec2_instance_info
ec2_instance_info:
region: "{{ lookup('env','AWS_DEFAULT_REGION') }}"
aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
aws_secret_key: "{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
filters:
instance-state-name: [ "shutting-down", "stopping", "stopped" ]
register: ec2_info
- name: print Instance Info
debug:
msg:
- "{{ ec2_info | json_query(name_tag) }}"
- "{{ ec2_info | json_query(description_tag) }}"
vars:
name_tag: "instances[*].tags.Name"
description_tag: "instances[*].tags.Description"
The output of the debug statement looks like this:
TASK [print Instance Info] **********************************************************************************************************************
ok: [tools-server-01] =>
msg:
- - server-01
- server-02
- - Description for server one
- Description for server two
Thanks #Rickkwa
I've got this mostly working.
here's my current code
- name: Print stopped systems
hosts: all
become: false
tasks:
- name: Gather ec2_metadata_facts (use -vv to show all)
action: ec2_metadata_facts
- name: pull instance info with ec2_instance_info
ec2_instance_info:
region: "{{ lookup('env','AWS_DEFAULT_REGION') }}"
aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
aws_secret_key: "{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
filters:
instance-state-name: [ "shutting-down", "stopping", "stopped" ]
register: ec2_info
- name: Create Name list
no_log: true
set_fact:
name_l: "{{ec2_info | json_query(name_tag)}}"
vars:
name_tag: "instances[*].tags.Name"
- name: Create Description list
no_log: true
set_fact:
desc_l: "{{ ec2_info | json_query(description_tag) }}"
vars:
description_tag: "instances[*].tags.Description"
- name: print together
debug:
msg: "{{item.0}} --- {{item.1}}"
loop: "{{ name_l|zip(desc_l)|list}}"
Unfortunately, the last debug statement includes some 'garbage" I'd like to remove...anyone have a suggestion to get rid of the stuff following the "u"s?
here's the output :'
TASK [print together] ***************************************************************************************************************************
ok: [dev-bsd-01] => (item=[u'server-01', u'server one description']) =>
msg: server-01 --- server one description
ok: [dev-bsd-01] => (item=[u'server-02', u'server two description']) =>
msg: server-02 --- server two description

Parsing CSV with Ansible

I need to parse a CSV with Ansible. The issue I'm facing is that I don't have fixed columns, they are variables.
For example: I need to extract network informations and to use them for configure vmware guests networks. In this case I can have a CSV with these:
(other columns) Nic1_Vlan Nic1_IP Nic1_MASK Nic1_Scope (other columns)
Otherwise I can have this:
(other columns) Nic1_Vlan Nic1_IP Nic1_MASK Nic1_Scope Nic2_Vlan Nic2_IP Nic2_MASK Nic2_Scope (other columns )
I can have 1 nic only or 3 in one shot.
Do you have any hint about this scenario? Would you use jinja2 or do you think is better to use a json/yaml variable files?
Thank you for any reply!!
Regards
For example the play below
- hosts: localhost
vars:
rec_len: 4
delimeter: ' '
tasks:
- set_fact:
nics: "{{ (lookup('file', 'data.csv')|trim).split(delimeter) }}"
- set_fact:
len: "{{ (nics|length/rec_len)|int }}"
- set_fact:
nics_list: "{{ nics_list|default([]) + [[
nics[(item*rec_len)|int],
nics[(item*rec_len+1)|int],
nics[(item*rec_len+2)|int],
nics[(item*rec_len+3)|int] ]] }}"
loop: "{{ range(0, len|int)|list }}"
- debug:
msg: "{{ item }}"
loop: "{{ nics_list }}"
gives
ok: [localhost] => (item=[u'Nic1_Vlan', u'Nic1_IP', u'Nic1_MASK', u'Nic1_Scope']) => {
"msg": [
"Nic1_Vlan",
"Nic1_IP",
"Nic1_MASK",
"Nic1_Scope"
]
}
ok: [localhost] => (item=[u'Nic2_Vlan', u'Nic2_IP', u'Nic2_MASK', u'Nic2_Scope']) => {
"msg": [
"Nic2_Vlan",
"Nic2_IP",
"Nic2_MASK",
"Nic2_Scope"
]
}
Is this what you're looking for?

Ansible parse json and read result into different variables

I've set up a task which queries the github api meta endpoint and returns the following
{
"verifiable_password_authentication": true,
"github_services_sha": "f9e3a6b98d76d9964a6613d581164039b8d54d89",
"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
],
"git": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20",
"13.229.188.59/32",
"13.250.177.223/32",
"18.194.104.89/32",
"18.195.85.27/32",
"35.159.8.160/32",
"52.74.223.119/32"
],
"pages": [
"192.30.252.153/32",
"192.30.252.154/32",
"185.199.108.153/32",
"185.199.109.153/32",
"185.199.110.153/32",
"185.199.111.153/32"
],
"importer": [
"54.87.5.173",
"54.166.52.62",
"23.20.92.3"
]
}
What I need to do is get the 3 hook IPs and read them each into their own variable.
I've tried a couple of solutions i've found around but nothing is seeming to work for me.
I've got as far as drilling down into the json so i'm being returned only the 3 IPs, but how do I get them out and into variables individually?
i gave it a shot using j2 syntax in the variable name part, and - TIL - looks like the jinja2 syntax is allowed in that part as well!
please see playbook to process the hooks list variable and assign to variables variable_1, variable_2, variable_3 and so on:
- hosts: localhost
gather_facts: false
vars:
counter: 1
hooks:
- 192.30.252.0/22
- 185.199.108.0/22
- 140.82.112.0/20
tasks:
- name: populate vars
set_fact:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ hooks }}"
- name: print vars
debug:
msg: "variable_1: {{variable_1}}, variable_2: {{variable_2}}, variable_3: {{variable_3}}"
and the output:
[root#optima-ansible ILIAS]# ansible-playbook 50257063.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [populate vars] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=192.30.252.0/22)
ok: [localhost] => (item=185.199.108.0/22)
ok: [localhost] => (item=140.82.112.0/20)
TASK [print vars] **********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "variable_1: 192.30.252.0/22, variable_2: 185.199.108.0/22, variable_3: 140.82.112.0/20"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[root#optima-ansible ILIAS]#
hope it helps
UPDATE:
something weird i noticed - also TIL - is that if you reverse the lines:
variable_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
to:
counter: "{{ counter | int + 1 }}"
variable_{{counter}}: "{{ item }}"
you still end up with the same variable names, _1 to _3, while i would expect to get _2 to _4.
I guess ansible loops behave differently than expected from other programming languages.
---
- name: Query Github Meta API and get Hook Ips
hosts: local
connection: local
vars:
counter: 1
tasks:
- name: Query API
uri:
url: https://api.github.com/meta
return_content: yes
register: response
- name: Populate Hook Variables
set_fact:
webhook_ip_{{counter}}: "{{ item }}"
counter: "{{ counter | int + 1 }}"
with_items:
- "{{ response['json']['hooks'] }}"
- name: print vars
debug:
msg: "Variable_1: {{ webhook_ip_1 }}, Variable_2: {{ webhook_ip_2 }}, Variable_3: {{ webhook_ip_3 }}"
Works with GitHub Webhook IPs in a loop
- name: get request to github
uri:
url: "https://api.github.com/meta"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
#X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: GitHub webhook IPs
debug:
msg: "{{ item }}"
with_items: "{{ (json_response.content | from_json).hooks }}"