How to do CSV lookups in ansible where the key comes from a variable? - jinja2

So ansible has the possibility of looking up things from a CSV file, the example on their web page is:
- debug: msg="The atomic number of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=,') }}"
- debug: msg="The atomic mass of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=, col=2') }}"
Now, my CSV file contains a mapping of hostnames to a number, like this:
HOST,ID
foo,0
bar,1
Now, when I adapt this to:
- debug: msg="My ID is {{ lookup('csvfile', '{{ inventory_hostname }} file=my.csv delimiter=,') }}"
I get the error:
Failed to template msg="My ID is {{ lookup('csvfile', '{{ inventory_hostname }} file=my.csv delimiter=,') }}": need more than 1 value to unpack
How do I do this right?


use the string formatting
- debug: msg="My ID is {{ lookup('csvfile', '{} file=my.csv delimiter=,'.format(inventory_hostname)) }}"

Related

How can I efficiently loop through and combine fields of a lookup/csv file?

I want to loop through and combine fields of a lookup/csv file into a list.
Below is a sample CSV file and read into a list (hosts_list)
id,hostname,host_ip,country_code,country_name
ID01,myhost1,10.2.3.2,US,United States
ID02,myhost2,10.2.3.3,US,United States
ID03,myhost3,10.2.3.4,UK,United Kingdom
ID04,myhost3,10.2.3.4,US,United Kingdom
Expected output is
['US-myhost1', 'US-myhost2', 'US-myhost3']
I've done it in two methods
### Method1 - Looping the list. Works good on small subset, but highly inefficient for large fields
- name: "Read as a list"
set_fact:
my_clubbed_list: "{{ my_clubbed_list|default([])+ [ my_clubbed_field ]}}"
loop: "{{ hosts_list.list }}"
vars:
- my_clubbed_field: "{{item.country_code}}-{{item.hostname}}"
when: item.country_code == 'US'
### Method2 - Using Jinja loop. Works fast but bit ugly
- debug:
msg: '{{ clubbed }}'
vars:
clubbed: |
"
{%- for result in hosts_list.list -%}
{% if 'US' == result.country_code %}
'{{ result.country_code }}-{{ result.hostname }}',
{% endif %}
{% endfor %}
"
Is there a better/efficient way to do this?

In a nutshell:
---
- hosts: localhost
gather_facts: false
vars:
hosts_query: >-
[][country_code,hostname].join('-', #)
hosts_list: "{{ hosts_csv.list | d([]) | json_query(hosts_query) }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: files/hosts.csv
register: hosts_csv
- name: Show calculated hosts list
ansible.builtin.debug:
var: hosts_list
If you don't have jmespath/json_query available, you can eventually replace the vars section above with:
vars:
hosts_codes: "{{ hosts_csv.list | d([]) | map(attribute='country_code') }}"
hosts_names: "{{ hosts_csv.list | d([]) | map(attribute='hostname') }}"
hosts_list: "{{ hosts_codes | zip(hosts_names) | map('join', '-') }}"
With both scenario and using your above CSV content we get:
TASK [Get CSV content] *****************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Show calculated hosts list] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"hosts_list": [
"US-myhost1",
"US-myhost2",
"UK-myhost3",
"US-myhost3"
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Use the Jinja template. For example,
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
gives
clubbed_dir:
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
Example of a complete playbook for testing
- hosts: localhost
vars:
clubbed_str: |
{% for i in hosts_csv.list|groupby('country_code') %}
{{ i.0 }}:
{% for h in i.1|map(attribute='hostname') %}
- {{ i.0 }}-{{ h }}
{% endfor %}
{% endfor %}
clubbed_dir: "{{ clubbed_str|from_yaml }}"
tasks:
- name: Get CSV content
ansible.builtin.read_csv:
path: "{{ playbook_dir }}/files/hosts.csv"
register: hosts_csv
- debug:
var: clubbed_dir|to_yaml
- debug:
var: clubbed_dir.US|to_yaml
gives
PLAY [localhost] *****************************************************************************
TASK [Get CSV content] ***********************************************************************
ok: [localhost]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir|to_yaml: |-
UK: [UK-myhost3]
US: [US-myhost1, US-myhost2, US-myhost3]
TASK [debug] *********************************************************************************
ok: [localhost] =>
clubbed_dir.US|to_yaml: |-
[US-myhost1, US-myhost2, US-myhost3]
PLAY RECAP ***********************************************************************************
localhost: ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

DBT using macros within external schema yml file

We define external redshift spectrum table using an yml file. Like the one below.
- name: visitor
description: 'visitor data'
external:
location: "s3://{{ env_var('ENV') }}-temp-raw-s3/{{ s3_prefix({{ env_var('ENV') }}) }}/Visitor"
file_format: parquet
partitions:
- name: year
data_type: VARCHAR(10)
vals: # macro w/ keyword args to generate list of values
macro: dbt.dates_in_range
args:
start_date_str: '{{ var("START_DT") }}'
end_date_str: '{{ var("START_DT") }}'
in_fmt: '%Y-%m-%d'
out_fmt: '%Y'
I would like to change the prefix of the s3 bucket based on the environment variable.
Here is the macro I have tried to change the prefix. I am calling that macro in in the yml file above in line 4 (location).
{% macro s3_prefix(env_var) %}
{% if env_var == "prod" %}
prefix_string = "prefix_prod"
{% else %}
prefix_string = "prefix_non_prod"
{% endif %}
{{ return(prefix_string) }}
{% endmacro %}
I am getting some compilation errors. Is there an easier way to achieve this? Any help would be really appreciated.

Accessing items in nested dicts and lists in Jinja2

Data:
primaries:
ca:
- 10.51.60.45
- 10.51.60.46
ny:
- 10.52.60.45
- 10.52.60.46
az:
- 10.53.60.45
- 10.53.60.46
I want a flattened list of all IP's(or a for loop which can iterate through just the IP's), but the cities ca and ny and az could be anything.

Ansible's extract filter, which extracts the value of a key from a container, makes this very simple.
{{ primaries | map('extract', primaries) | flatten }}
You can also directly use the dictionary's values() method, which is slightly less flexible (the extract approach allows you to filter the keys beforehand, which you can't do here.)
{{ primaries.values() | flatten }}

You just need to iterate through the keys of the dictionary.
{% for region, ips in primaries.items() %}
{% for ip in ips %}
{{ ip }}
{% endfor %}
{% endfor %}
Read the Jinja docs on for.

Using Ansible, you can get a flattened list of ips using the json_query filter:
List of ip addresses:
{% for addr in primaries|json_query('*[][]') %}
- {{ addr }}
{% endfor %}
This results in:
List of ip addresses:
- 10.51.60.45
- 10.51.60.46
- 10.52.60.45
- 10.52.60.46
- 10.53.60.45
- 10.53.60.46
Here's a runnable example:
- hosts: localhost
gather_facts: false
vars:
primaries:
ca:
- 10.51.60.45
- 10.51.60.46
ny:
- 10.52.60.45
- 10.52.60.46
az:
- 10.53.60.45
- 10.53.60.46
tasks:
- copy:
dest: addresses.txt
content: |
List of ip addresses:
{% for addr in primaries|json_query('*[][]') %}
- {{ addr }}
{% endfor %}
The json_query filter uses the JMESPath query language.

Prettify JSON file in ansible

Say I have a JSON file from a template
- name: consul config file
template: >
src={{ consul_config_template }}
dest={{ consul_config_file }}
owner={{ consul_user }}
group={{ consul_group }}
mode=0755
How can I prettify the JSON file in order to remove extra spaces and newlines? Is there any ansible module I can call after template?

You can do this with one shot:
- copy:
content: "{{ item | to_nice_json }}"
dest: "{{ consul_config_file }}"
owner: "{{ consul_user }}"
group: "{{ consul_group }}"
mode: 0755
with_template: "{{ consul_config_template }}"

Ansible: Get all the IP addresses of a group

Let's imagine an inventory file like this:
node-01 ansible_ssh_host=192.168.100.101
node-02 ansible_ssh_host=192.168.100.102
node-03 ansible_ssh_host=192.168.100.103
node-04 ansible_ssh_host=192.168.100.104
node-05 ansible_ssh_host=192.168.100.105
[mainnodes]
node-[01:04]
In my playbook I now want to create some variables containing the IP addresses of the group mainnodes:
vars:
main_nodes_ips: "192.168.100.101,192.168.100.102,192.168.100.103,192.168.100.104"
main_nodes_ips_with_port: "192.168.100.101:3000,192.168.100.102:3000,192.168.100.103:3000,192.168.100.104:3000"
This is what I got so far:
vars:
main_nodes_ips: "{{groups['mainnodes']|join(',')}}"
main_nodes_ips_with_port: "{{groups['mainnodes']|join(':3000,')}}"
but that would use the host names instead of the IP addresses.
Any ideas how this could be done?
Update:
looking at the docs for a while, I think this would allow me to loop through all the ip adresses:
{% for host in groups['mainnodes'] %}
{{hostvars[host]['ansible_ssh_host']}}
{% endfor %}
But I just can't figure out how to create an array that holds all these IPs. So that I can use the |join() command on them.
Update2:
I just thought I had figured it out... but it turns out that you cannot use the {% %} syntax in the playbook... or can I?
Well in the vars section it didn't. :/
vars:
{% set main_nodes_ip_arr=[] %}
{% for host in groups['mesos-slave'] %}
{% if main_nodes_ip_arr.insert(loop.index,hostvars[host]['ansible_ssh_host']) %} {% endif %}
{% endfor %}
main_nodes_ips: "{{main_nodes_ip_arr|join(',')}}"
main_nodes_ips_with_port: "{{main_nodes_ip_arr|join(':3000,')}}"

I find the magic map extract here.
main_nodes_ips: "{{ groups['mainnodes'] | map('extract', hostvars, ['ansible_host']) | join(',') }}"
main_nodes_ips_with_port: "{{ groups['mainnodes'] | map('extract', hostvars, ['ansible_host']) | join(':3000,') }}:3000"
An alternative(idea comes from here):
main_nodes_ips: "{{ groups['mainnodes'] | map('extract', hostvars, ['ansible_eth0', 'ipv4', 'address']) | join(',') }}"
(Suppose the interface is eth0)

i came across this problem a while back and this is what i came up with (not optimal, but it works)
---
# playbook.yml
- hosts: localhost
connection: local
tasks:
- name: create deploy template
template:
src: iplist.txt
dest: /tmp/iplist.txt
- include_vars: /tmp/iplist.txt
- debug: var=ip
and the template file is
ip:
{% for h in groups['webservers'] %}
- {{ hostvars[h].ansible_ssh_host }}
{% endfor %}

This do the trick for me. Not relying on the interface name
- main_nodes_ips: "{{ groups['mainnodes'] | map('extract', hostvars, ['ansible_default_ipv4', 'address']) | join(',') }}"

- name: Create List of nodes to be added into Cluster
set_fact: nodelist={%for host in groups['mygroup']%}"{{hostvars[host].ansible_eth0.ipv4.address}}"{% if not loop.last %},{% endif %}{% endfor %}
- debug: msg=[{{nodelist}}]
- name: Set Cluster node list in config file
lineinfile:
path: "/etc/myfonfig.cfg"
line: "hosts: [{{ nodelist }}]"
as results you will have the following line in config file:
hosts: ["192.168.126.38","192.168.126.39","192.168.126.40"]

I got it to work on my own now. I'm not too happy about the solution, but it will do:
main_nodes_ips: "{% set IP_ARR=[] %}{% for host in groups['mainnodes'] %}{% if IP_ARR.insert(loop.index,hostvars[host]['ansible_ssh_host']) %}{% endif %}{% endfor %}{{IP_ARR|join(',')}}"
main_nodes_ips_with_port: "{% set IP_ARR=[] %}{% for host in groups['mainnodes'] %}{% if IP_ARR.insert(loop.index,hostvars[host]['ansible_ssh_host']) %}{% endif %}{% endfor %}{{IP_ARR|join(':3000,')}

I've done this by using ansible facts in a playbook.
This playbook takes ansible_all_ipv4_addresses list and ansible_nodename (which is actually fully qualified domain name), iterates through all hosts and saves the data in localpath_to_save_ips file on your localhost. You can change localpath_to_save_ips to the absolute path on your localhost.
---
- hosts: all
become: yes
gather_facts: yes
tasks:
- name: get ip
local_action: shell echo {{ ansible_all_ipv4_addresses }} {{ ansible_nodename }} >> localpath_to_save_ips

I found the "only way" to acceess other group's ip's, when any of the following is true:
some members are not bootstrapped by ansible yet
using serial
group is not part of playbook
Is as follows:
{% set ips=[] %}{% for host in groups['othergroup'] %}{% if ips.append(lookup('dig', host)) %}{% endif %}{% endfor %}{{ ips }}
Requires dnspython on the machine running ansible, install via
sudo apt-get install python-dnspython
If anyone knows a better way given the conditions, I'd love to get rid of this abomination.

this is what I did in order to not be relied on eth0 (thanks to ADV-IT's answer):
- name: gathering facts
hosts: mainnodes
gather_facts: true
- hosts: mainnodes
tasks:
- name: Create List of nodes
set_fact: nodelist={%for host in groups['mainnodes']%}"{{hostvars[host]['ansible_env'].SSH_CONNECTION.split(' ')[2]}}"{% if not loop.last %},{% endif %}{% endfor %}

I ran into a similar problem getting the IP address of a node in another group.
Using a construct like:
the_ip: "{{ hostvars[groups['master'][0]]['ansible_default_ipv4'].address }}"
works only when running the group master, which was not part of my playbook (I was running on localhost).
I have overcome the problem by adding an extra play to playbook, like:
- hosts: master
gather_facts: yes
become: no
vars:
- the_master_ip: "{{ hostvars[groups['master'][0]]['ansible_default_ipv4'].address }}"
tasks:
- debug: var=the_master_ip
- set_fact: the_ip={{ the_master_ip }}
After which I can use the the_ip in the next play of the playbook.
This may also solve the abomination mentioned by #Petroldrake ?

##Just fetch Ip's using -ansible_default_ipv4.address- & redirect to a local file & then use it
name: gathering_facts
hosts: hosts
gather_facts: true
tasks:
name: Rediret to the file
shell: echo "{{ansible_default_ipv4.address}}" >>ipss.txt
delegate_to: localhost