How can I efficiently loop through and combine fields of a lookup/csv file? - csv

Question

I want to loop through and combine fields of a lookup/csv file into a list.
Below is a sample CSV file and read into a list (hosts_list)
id,hostname,host_ip,country_code,country_name
ID01,myhost1,10.2.3.2,US,United States
ID02,myhost2,10.2.3.3,US,United States
ID03,myhost3,10.2.3.4,UK,United Kingdom
ID04,myhost3,10.2.3.4,US,United Kingdom
Expected output is
['US-myhost1', 'US-myhost2', 'US-myhost3']
I've done it in two methods
### Method1 - Looping the list. Works good on small subset, but highly inefficient for large fields
- name: "Read as a list"
set_fact:
my_clubbed_list: "{{ my_clubbed_list|default([])+ [ my_clubbed_field ]}}"
loop: "{{ hosts_list.list }}"
vars:
- my_clubbed_field: "{{item.country_code}}-{{item.hostname}}"
when: item.country_code == 'US'
### Method2 - Using Jinja loop. Works fast but bit ugly
- debug:
msg: '{{ clubbed }}'
vars:
clubbed: |
"
{%- for result in hosts_list.list -%}
{% if 'US' == result.country_code %}
'{{ result.country_code }}-{{ result.hostname }}',
{% endif %}
{% endfor %}
"
Is there a better/efficient way to do this?

Answer 1

In a nutshell:
---
- hosts: localhost
gather_facts: false
vars:
hosts_query: >-
[][country_code,hostname].join('-', #)
hosts_list: "{{ hosts_csv.list | d([]) | json_query(hosts_query) }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: files/hosts.csv
register: hosts_csv
- name: Show calculated hosts list
ansible.builtin.debug:
var: hosts_list
If you don't have jmespath/json_query available, you can eventually replace the vars section above with:
vars:
hosts_codes: "{{ hosts_csv.list | d([]) | map(attribute='country_code') }}"
hosts_names: "{{ hosts_csv.list | d([]) | map(attribute='hostname') }}"
hosts_list: "{{ hosts_codes | zip(hosts_names) | map('join', '-') }}"
With both scenario and using your above CSV content we get:
TASK [Get CSV content] *****************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Show calculated hosts list] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"hosts_list": [
"US-myhost1",
"US-myhost2",
"UK-myhost3",
"US-myhost3"
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Answer 2

Use the Jinja template. For example,
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
gives
clubbed_dir:
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
Example of a complete playbook for testing
- hosts: localhost
vars:
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: "{{ playbook_dir }}/files/hosts.csv"
register: hosts_csv
- debug:
var: clubbed_dir|to_yaml
- debug:
var: clubbed_dir.US|to_yaml
gives
PLAY [localhost] *****************************************************************************
TASK [Get CSV content] ***********************************************************************
ok: [localhost]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir|to_yaml: |-
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir.US|to_yaml: |-
[US-myhost1, US-myhost2, US-myhost3]
PLAY RECAP ***********************************************************************************
localhost: ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0