Prettify JSON file in ansible - json

Say I have a JSON file from a template
- name: consul config file
template: >
src={{ consul_config_template }}
dest={{ consul_config_file }}
owner={{ consul_user }}
group={{ consul_group }}
mode=0755
How can I prettify the JSON file in order to remove extra spaces and newlines? Is there any ansible module I can call after template?


You can do this with one shot:
- copy:
content: "{{ item | to_nice_json }}"
dest: "{{ consul_config_file }}"
owner: "{{ consul_user }}"
group: "{{ consul_group }}"
mode: 0755
with_template: "{{ consul_config_template }}"

Related

How can I efficiently loop through and combine fields of a lookup/csv file?

I want to loop through and combine fields of a lookup/csv file into a list.
Below is a sample CSV file and read into a list (hosts_list)
id,hostname,host_ip,country_code,country_name
ID01,myhost1,10.2.3.2,US,United States
ID02,myhost2,10.2.3.3,US,United States
ID03,myhost3,10.2.3.4,UK,United Kingdom
ID04,myhost3,10.2.3.4,US,United Kingdom
Expected output is
['US-myhost1', 'US-myhost2', 'US-myhost3']
I've done it in two methods
### Method1 - Looping the list. Works good on small subset, but highly inefficient for large fields
- name: "Read as a list"
set_fact:
my_clubbed_list: "{{ my_clubbed_list|default([])+ [ my_clubbed_field ]}}"
loop: "{{ hosts_list.list }}"
vars:
- my_clubbed_field: "{{item.country_code}}-{{item.hostname}}"
when: item.country_code == 'US'
### Method2 - Using Jinja loop. Works fast but bit ugly
- debug:
msg: '{{ clubbed }}'
vars:
clubbed: |
"
{%- for result in hosts_list.list -%}
{% if 'US' == result.country_code %}
'{{ result.country_code }}-{{ result.hostname }}',
{% endif %}
{% endfor %}
"
Is there a better/efficient way to do this?

In a nutshell:
---
- hosts: localhost
gather_facts: false
vars:
hosts_query: >-
[][country_code,hostname].join('-', #)
hosts_list: "{{ hosts_csv.list | d([]) | json_query(hosts_query) }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: files/hosts.csv
register: hosts_csv
- name: Show calculated hosts list
ansible.builtin.debug:
var: hosts_list
If you don't have jmespath/json_query available, you can eventually replace the vars section above with:
vars:
hosts_codes: "{{ hosts_csv.list | d([]) | map(attribute='country_code') }}"
hosts_names: "{{ hosts_csv.list | d([]) | map(attribute='hostname') }}"
hosts_list: "{{ hosts_codes | zip(hosts_names) | map('join', '-') }}"
With both scenario and using your above CSV content we get:
TASK [Get CSV content] *****************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Show calculated hosts list] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"hosts_list": [
"US-myhost1",
"US-myhost2",
"UK-myhost3",
"US-myhost3"
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Use the Jinja template. For example,
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
gives
clubbed_dir:
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
Example of a complete playbook for testing
- hosts: localhost
vars:
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: "{{ playbook_dir }}/files/hosts.csv"
register: hosts_csv
- debug:
var: clubbed_dir|to_yaml
- debug:
var: clubbed_dir.US|to_yaml
gives
PLAY [localhost] *****************************************************************************
TASK [Get CSV content] ***********************************************************************
ok: [localhost]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir|to_yaml: |-
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir.US|to_yaml: |-
[US-myhost1, US-myhost2, US-myhost3]
PLAY RECAP ***********************************************************************************
localhost: ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Looping through user list to create AD users in ansible

I have a user list in json that is looped through in ansible playbook. I want the playbook to iterate through the list and create users in active directory, but skip over the already existing users in AD. Currently, it will give me an error if the user already exists in the AD. How do I loop through it and ignore the users already in AD?
This is what I have so far:
- name: Create user in AD
community.windows.win_domain_user:
name: '{{ item.firstname }} {{ item.surname }}'
firstname: '{{ item.firstname }}'
surname: '{{ item.surname }}'
upn: '{{ item.upn }}'
password: '{{ pass }}'
user_cannot_change_password: false
state: present
path: OU=Normal,OU=Accounts,DC=prod,DC=osur,DC=org
groups:
- dev_users
attributes:
displayName: '{{ item.display_name }}'
domain_username: '{{ domain_user }}'
domain_password: "{{ ansible_password }}"
domain_server: '{{ domain }}'
with_items: "{{ Users.users }}"
register: create_user_output
ignore_errors: yes

compare 2 files and get 3rd file

I have 2 csv files as
CSV1:
ABC, NIC.Slot.1-1-1:, A6:C3:4F:8E:D5:30
DEF, NIC.Slot.1-2-1:, B2:D5:8C:5A:43:60
CSV2:
XYZ, ethernet15:3, ABC_DPDK2, b2:d5:8c:5a:43:60
UVW, ethernet15:2, ABC_DPDK1, a6:c3:4f:8e:d5:30
I want another CSV file which will compare the MAC of both the files and update rest in output csv file
Below is the desired output
CSV3:
ABC, UVW, NIC.Slot.1-1-1:, A6:C3:4F:8E:D5:30, ethernet15:2, ABC_DPDK1
DEF, XYZ, NIC.Slot.1-2-1:, B2:D5:8C:5A:43:60, ethernet15:3, ABC_DPDK2
is there anyway i can do it via ansible

Given the files
shell> cat net1.csv
ABC, NIC.Slot.1-1-1:, A6:C3:4F:8E:D5:30
DEF, NIC.Slot.1-2-1:, B2:D5:8C:5A:43:60
shell> cat net2.csv
XYZ, ethernet15:3, ABC_DPDK2, b2:d5:8c:5a:43:60
UVW, ethernet15:2, ABC_DPDK1, a6:c3:4f:8e:d5:30
Put the declarations below as appropriate
fields: [name1, name2, nic, MAC, dev, label]
csv_files:
- {file: net1.csv, fields: [name1, nic, mac]}
- {file: net2.csv, fields: [name2, dev, label, mac]}
_list: "{{ csv.results|map(attribute='list')|flatten }}"
_MAC: "{{ _list|map(attribute='mac')|map('trim')|map('upper')|
map('community.general.dict_kv', 'MAC')|list }}"
_groups: "{{ _list|zip(_MAC)|map('combine')|groupby('MAC') }}"
_values: "{{ _groups|map(attribute=1)|map('combine')|list }}"
Read the files, for example
- community.general.read_csv:
path: "{{ playbook_dir }}/{{ item.file }}"
fieldnames: "{{ item.fields }}"
skipinitialspace: true
register: csv
loop: "{{ csv_files }}"
gives
_list:
- mac: A6:C3:4F:8E:D5:30
name1: ABC
nic: 'NIC.Slot.1-1-1:'
- mac: B2:D5:8C:5A:43:60
name1: DEF
nic: 'NIC.Slot.1-2-1:'
- dev: ethernet15:3
label: ABC_DPDK2
mac: 'b2:d5:8c:5a:43:60 '
name2: XYZ
- dev: ethernet15:2
label: ABC_DPDK1
mac: a6:c3:4f:8e:d5:30
name2: UVW
_values:
- MAC: A6:C3:4F:8E:D5:30
dev: ethernet15:2
label: ABC_DPDK1
mac: a6:c3:4f:8e:d5:30
name1: ABC
name2: UVW
nic: 'NIC.Slot.1-1-1:'
- MAC: B2:D5:8C:5A:43:60
dev: ethernet15:3
label: ABC_DPDK2
mac: 'b2:d5:8c:5a:43:60 '
name1: DEF
name2: XYZ
nic: 'NIC.Slot.1-2-1:'
Create the CSV file
- ansible.builtin.copy:
dest: net3.csv
content: |-
{% for line in _values %}
{{ fields|map('extract', line)|join(', ') }}
{% endfor %}
gives
shell> cat ~/net3.csv
ABC, UVW, NIC.Slot.1-1-1:, A6:C3:4F:8E:D5:30, ethernet15:2, ABC_DPDK1
DEF, XYZ, NIC.Slot.1-2-1:, B2:D5:8C:5A:43:60, ethernet15:3, ABC_DPDK2
Example of a complete playbook
- hosts: localhost
vars:
fields: [name1, name2, nic, MAC, dev, label]
csv_files:
- {file: net1.csv, fields: [name1, nic, mac]}
- {file: net2.csv, fields: [name2, dev, label, mac]}
_list: "{{ csv.results|map(attribute='list')|flatten }}"
_MAC: "{{ _list|map(attribute='mac')|map('trim')|map('upper')|
map('community.general.dict_kv', 'MAC')|list }}"
_groups: "{{ _list|zip(_MAC)|map('combine')|groupby('MAC') }}"
_values: "{{ _groups|map(attribute=1)|map('combine')|list }}"
tasks:
- community.general.read_csv:
path: "{{ playbook_dir }}/{{ item.file }}"
fieldnames: "{{ item.fields }}"
skipinitialspace: true
register: csv
loop: "{{ csv_files }}"
- ansible.builtin.copy:
dest: net3.csv
content: |-
{% for line in _values %}
{{ fields|map('extract', line)|join(', ') }}
{% endfor %}

Ansible : multy loop into json file

I have as a source a json file that contains a list of NetworkFlow keys, and from which i would like to extract information to create security rules, using a double loop in ansible.
Below an example from my json file :
{
"Name":"Some_name",
"NetworkFlow":[
{
"GroupName":"Test1",
"Type":"Ingress",
"Env":"prod",
"Server":[
"192.168.1.1",
"192.168.1.2"
],
"Service":[
{
"Protocol":"TCP",
"Port":"443,22,53"
},
{
"Protocol":"UDP",
"Port":"21"
}
]
},
{
"GroupName":"Test2",
"Type":"Egress",
"Env":"dev",
"Server":[
"192.168.1.3",
"192.168.1.4"
],
"Service":[
{
"Protocol":"UDP",
"Port":"9996,9997"
}
]
}
]
}
so firstly i have to loop for each NetworkFlow section, and inside each one, i have to loop in the list of servers and also in the list of services (protocols and ports) to get a simular parsing like the below:
#rule= Server,Protocol,Port,Type,Env,GroupName
msg: 192.168.1.1,TCP,443,Ingress,prod,Test1
msg: 192.168.1.1,TCP,22,Ingress,prod,Test1
msg: 192.168.1.1,TCP,53,Ingress,prod,Test1
msg: 192.168.1.1,UDP,21,Ingress,prod,Test1
msg: 192.168.1.2,TCP,443,Ingress,prod,Test1
msg: 192.168.1.2,TCP,22,Ingress,prod,Test1
msg: 192.168.1.2,TCP,53,Ingress,prod,Test1
msg: 192.168.1.2,UDP,21,Ingress,prod,Test1
msg: 192.168.1.3,UDP,9996,Egress,dev,Test2
msg: 192.168.1.3,UDP,9997,Egress,dev,Test2
msg: 192.168.1.4,UDP,9996,Egress,dev,Test2
msg: 192.168.1.4,UDP,9997,Egress,dev,Test2
Here Below my playbook tasks :
my main task :
---
- name: Include JSON file
include_vars:
file: test.json
- include_tasks: rules.yml
loop: "{{ NetworkFlow }}"
loop_control:
loop_var: oi
my rule task :
---
- set_fact:
Services: "{{ Services|from_yaml }}"
vars:
Services: |
{% for service in oi.Service %}
{% for port in service.Port.split(',') %}
- Protocol: {{ service.Protocol }}
Port: {{ port }}
{% endfor %}
{% endfor %}
- debug:
msg: "{{ i.0 }},{{ i.1.Protocol }},{{ i.1.Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}"
with_nested:
- "{{ oi.Server }}"
- "{{ Services }}"
loop_control:
loop_var: i
For info, my oi.Service.Port can have a list of port separated by a comma !
I tried with a loop inside a with_nested and it work for the first key Test1, but i didn't get the correct parsing for the second NetworkFlow key Test2
TASK [test : set_fact] *****************************************************************************************************************************************
ok: [localhost]
TASK [test : debug] ********************************************************************************************************************************************
"msg": "192.168.1.1,TCP,443,Ingress,prod,Test1"
"msg": "192.168.1.1,TCP,22,Ingress,prod,Test1"
"msg": "192.168.1.1,TCP,53,Ingress,prod,Test1"
"msg": "192.168.1.1,UDP,21,Ingress,prod,Test1"
"msg": "192.168.1.2,TCP,443,Ingress,prod,Test1"
"msg": "192.168.1.2,TCP,22,Ingress,prod,Test1"
"msg": "192.168.1.2,TCP,53,Ingress,prod,Test1"
"msg": "192.168.1.2,UDP,21,Ingress,prod,Test1"
}
TASK [test : set_fact] *****************************************************************************************************************************************
ok: [localhost]
TASK [test : debug] ********************************************************************************************************************************************
"msg": "192.168.1.3,TCP,443,Egress,dev,Test2"
"msg": "192.168.1.3,TCP,22,Egress,dev,Test2"
"msg": "192.168.1.3,TCP,53,Egress,dev,Test2"
"msg": "192.168.1.3,UDP,21,Egress,dev,Test2"
"msg": "192.168.1.4,TCP,443,Egress,dev,Test2"
"msg": "192.168.1.4,TCP,22,Egress,dev,Test2"
"msg": "192.168.1.4,TCP,53,Egress,dev,Test2"
"msg": "192.168.1.4,UDP,21,Egress,dev,Test2"
Have anyone idea for how to deal with that please?

The task below creates the list Services including the servers
- set_fact:
Services: "{{ Services|default([]) + Service|from_yaml }}"
vars:
Service: |
{% for Port in item.1.Port.split(',') %}
- {{ item.0 }},{{ item.1.Protocol }},{{ Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}
{% endfor %}
with_nested:
- "{{ oi.Server }}"
- "{{ oi.Service }}"
Services:
- 192.168.1.1,TCP,443,Ingress,prod,Test1
- 192.168.1.1,TCP,22,Ingress,prod,Test1
- 192.168.1.1,TCP,53,Ingress,prod,Test1
- 192.168.1.1,UDP,21,Ingress,prod,Test1
- 192.168.1.2,TCP,443,Ingress,prod,Test1
- 192.168.1.2,TCP,22,Ingress,prod,Test1
- 192.168.1.2,TCP,53,Ingress,prod,Test1
- 192.168.1.2,UDP,21,Ingress,prod,Test1
- 192.168.1.3,UDP,9996,Egress,dev,Test2
- 192.168.1.3,UDP,9997,Egress,dev,Test2
- 192.168.1.4,UDP,9996,Egress,dev,Test2
- 192.168.1.4,UDP,9997,Egress,dev,Test2

How to do CSV lookups in ansible where the key comes from a variable?

So ansible has the possibility of looking up things from a CSV file, the example on their web page is:
- debug: msg="The atomic number of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=,') }}"
- debug: msg="The atomic mass of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=, col=2') }}"
Now, my CSV file contains a mapping of hostnames to a number, like this:
HOST,ID
foo,0
bar,1
Now, when I adapt this to:
- debug: msg="My ID is {{ lookup('csvfile', '{{ inventory_hostname }} file=my.csv delimiter=,') }}"
I get the error:
Failed to template msg="My ID is {{ lookup('csvfile', '{{ inventory_hostname }} file=my.csv delimiter=,') }}": need more than 1 value to unpack
How do I do this right?

use the string formatting
- debug: msg="My ID is {{ lookup('csvfile', '{} file=my.csv delimiter=,'.format(inventory_hostname)) }}"